The Earl succeeded his father in 1859. From 1837-59, he was a Member of Parliament for Southampton, Bath, and Forfarshire, successively. On 23 March 1839 he married Juliana Cavendish Philips, eldest daughter of Sir George R. Philips, 2nd Bt. of Warwickshire. They had three children: Robert (3rd Earl), George (4th Earl), and Julia (married 4th Lord Abercromby).