The Milton House, built in Milton in 1844 by Joseph Goodrich, is said to have been the first grout (or concrete) building in the United States and served as a hotel and country store. During the Civil War it was a station on the Underground Railroad, sheltering runaway slaves from the South. It is on the National Register of Historic Sites.