David Madison Smith had two sons by his first wife David Harrison and Joseph Warren Smith. After she died he remarried to Catherine Julia Watson and had 6 more children Laura, Neal, Charles, John Madison, Catherine Ella, and Emma Bell.
In 1852 David Madison moved his family to Swainsboro, GA from Edgefield District, SC.
In 1858 David Madison Smith was appointed sheriff due to the death of Reubin Boatright.
During the Civil War, David Madison was a soldier in Company C, 38th Regt. Georgia Volunteer Infantry. This regiment served in the brigade commanded, in turn, by Generals Alexander Lawton, John Brown Gordon, and John H. Lowe. Only a handful remained to surrender at Appomattox, April 1865. David Madison recived a servere head wound and was eventually sent home, disabled.
In 1881, he died and was buried in a field serveral hundred yards from the Courthouse in the heart of Swainsboro. The site today is occupied by Swainsboro Primitive Baptist Church. In later years, he was joined by Catherine, and sons, Charles and Neal. The grave site was accidentally covered in 1930 when an annex was built onto the church.
A Confederate Memorial marker was replaced on September 22, 1979 to honor David Madison Smith. The site is southwest corner of the Swainsboro, GA Privitive Baptist Churchyard, at West Main and Roberts. True to the Army of Virginia, it is on the "high ground", faced with shrubs.