Built in 1775-76 as a "hospital for the sick," by 1793 this structure had been designated as the residence for the Mount Vernon gardener.  In 1799, the gardener was a young bachelor named William Spence, who had been recruited from Scotland.  The gardener played an important role on the estate, and Washington had high expectations for his performance.  In a letter written in 1797, Washington stated that, "the man ought to be a good Kitchen Gardener, to have some knowledge of a Green house and hot house, and how to raise things in hot beds."  In return, Washington promised that, "he would be furnished with a good apartment, convenient to his work, to reside in."

The Gardener's House was restored to its eighteenth century appearance in 2006.