John Paine Jr. was the son of John Paine; the latter was raised by his aunt and uncle, Captain and Mrs. (née Anna Paine) Samuel Chandler, who had no children of their own. John Paine had inherited Chandler's 300-acre farm, as well as two portraits--one of the Captain, and one of Anna Paine Chandler--painted by the Captain's brother, Winthrop Chandler, around 1780. John Paine, Jr. inherited the pictures (now in the NGA), which he in turn passed to his son, John Merrick Paine; they continued to descend in the Paine family.