Samuel Green Sargent was a descendant of William Sargent, who settled in Malden, Massachusetts after coming to America from Northampton, England in 1638. Samuel was born in Chelsea, Massachusetts. On 18 May 1786, he married Martha ("Patty") Hills [1763-1812] of Malden. Sadly, their first two children, Patty and Samuel, died in infancy. Before 1795, the couple moved to Savannah, Georgia. There, three of their children were born: Patty Hills [1795-1831, whose given name was really Martha, like her mother's]; Maria Green [1796-1815], and Eliza Lynde [1798-1827]. The family moved back to the Boston area by 11 April 1800, as records show that their son Samuel Seaver was born on that day in Charlestown, Massachusetts. Another child, Lucinda, died before her first birthday in 1806. Samuel Sargent worked as a merchant both in Darby Ward, Savannah, and at Spear's Wharf on Boston Harbor. A family portrait of 1800 by an unknown American "naive" painter depicts the tasteful, upper-middle class interior of the Sargent home and serves as evidence of Sargent's mercantile success. After the death of his wife Martha ("Patty"), Sargent married Mary Hills [1780-1830]. The picture (now in the NGA) was passed down in the family to great-granddaughter Martha Gertrude Farner Cork of Waukesha, Wisconsin, who in turn gave it to her granddaughter, Gertrude Louise Cork Swezy of Coxsackie, New York by mid-1941.