Built in 1775-76 as a "hospital for the sick," by 1793 this structure had been designated as the residence for the Mount Vernon gardener. The gardener played an important role on the estate, and George Washington had high expectations for his performance. In a letter written in 1797, Washington stated that, "the man ought to be a good Kitchen Gardener, to have some knowledge of a Green house and hot house, and how to raise things in hot beds." In return, George Washington promised that "he would be furnished with a good apartment, convenient to his work, to reside in." In 1799, the gardener was a young bachelor named William Spence, who had been recruited from Scotland.
The Gardener's House was restored to its 18th-century appearance in 2006.