George Worthen Whistler was the son of Mary Ducatel and George William Whistler, who were married in 1848. Mary Ducatel was descended from Robert Barry, nephew of Captain James Barry (d. 7 January 1808) and his wife Joanna Gould Barry (d. 18 October 1811). The Barrys were imigrants of Irish descent who settled in Baltimore in 1793. They came from Lisbon, Portugal, and James served as the Portugese counsel general in the US. The James Barrys had two daughters, Ann (d. 17 July 1808) and Mary (d. 17 November 1805), both of whom were unmarried. The Barry family was painted by American artist Gilbert Stuart; the portraits of the Barry daughters are now in the NGA (1954.9.3 and 1954.9.4). George William Whistler was the older step-brother of painter James McNeill Whistler.