James N. Williamson was a Confederate veteran who married E. M. Holt’s daughter Mary Elizabeth Holt and entered into the Holt family business. Some accounts give construction dates of 1869 and 1871 for this house, but newspaper articles suggest that it was built ca. 1873 by Denny and remodeled a few years later. Lyndon Swain was reportedly the architect of the 1878 remodeling.