Henry Vernon was the son of George, 1st Baron Vernon, and his third wife, Martha Harcourt (granddaughter of the Lord Chancellor, Viscount Harcourt, and sister of the 1st Earl of Harcourt). He succeeded his half-brother, George, 2nd Baron Vernon, in 1813. On 14 February 1779, he married Elizabeth Rebecca Anne [d. 1793], daughter of Sir Charles Sedley, 2nd Bt. of Southfleet. They had a son and two daughters. In November of 1795, he married secondly to Alice Lucy [d. 1827], daughter of Sir John Whiteford, Bt.; they had two sons. His eldest son, George Charles, succeeded Henry as 4th Baron Vernon.