Thomas Duncan Smith was the son of William Rudolph Smith [d. 1868] and his wife Eliza Anthony [1789-1821], and the great-grandson of Joseph Anthony [1738-1798]. Anthony, from Rhode Island, settled in Philadelphia in 1782, where he ran a shipping firm. Anthony's sister, Elizabeth, was the mother of the artist Gilbert Stuart, whom he patronized. Anthony commissioned Stuart to make portraits of his family; a portrait of Anthony by Stuart is now in the NGA (1942.8.11).