He painted his first portrait at age 13 and at age 17 obtained a sitting by George Washington. This began a lifetime quest of trying to produce the most fitting image of Washington and have it accepted as the President's official likeness. During the 1840s and 50s, he made over seventy replicas which provided him with a continuous income.

In 1797, he and his brother Raphaelle opened a museum of art and natural history in Baltimore, but it was not successful. He then assisted his father in unearthing and assembling the first complete skeleton of a mastodon ever found. Traveling to Paris and London, he decided to paint historical scenes but got negative responses, so he reestablished himself as a successful portrait painter, especially of George Washington and Thomas Jefferson. In New York, he served as President of the American Academy of Art and spent the remainder of his life in Philadelphia.