Brabazon started life as Hercules Brabazon Sharpe, the second son of Hercules Sharpe. He was brought up in Paris and early on displayed a talent in the arts. After the Sharpes' return to England, Hercules was educated at Harrow, where he painted watercolor landscapes, and Trinity College, Cambridge, where he read mathematics. In 1847 his older brother died and the Brabazon estates (and name) in Ireland were passed to him.