The son of James, 5th Duke of Hamilton, by his first wife, Anne (daughter of the 4th Earl of Dundonald), Sir James succeeded his father as Duke in 1742/43. A Knight of the Order of the Thistle, on 14 February 1752 he married Elizabeth [d. 1790], 2nd daughter of John Gunning and the Hon. Bridget Bourke, of Castle Coote, county Roscommon, Ireland. James and Elizabeth had three children--James George, Douglas, and Elizabeth--the sons James and Douglas succeeded their father as 7th and 8th Dukes, respectively. The 6th Duke's widow remarried in 1759 to the 5th Duke of Argyll, and was created Baroness Hamilton on 20 May 1776.