The Earl succeeded his father, Charles Agar, in 1809. He married Diana [d. 1841], daughter of the 11th Earl of Pembroke, on 17 May 1816. They had a daughter and three sons, of whom son James Charles Herbert Welbore Ellis succeeded his father as 3rd Earl. The 2cd Earl's art collection was sold in 1806 to Lord Grosvenor.