The eldest son of Henry, 3rd Baron Vernon, by his first wife, Elizabeth Rebecca Anne (daughter of Sir Charles Sedley, 2nd Bt. of Southfleet), George Charles Vernon succeeded his father as baron in 1829. On 5 August, 1802, he married Frances Maria [d. 1837], only daughter of Admiral Right Hon. Sir John Borlase-Warren, Bt. (she assumed by sign-manual in 1826 the name of Warren only.) They had a son, George John, who succeeded Henry as 5th Baron Vernon.