Son of Henry and Mary Elizabeth Bland Lee of "Lee Hall" in Westmoreland Co., Virginia. A member of the influential Lee family which had been prominent in Virginia affairs since the seventeenth century. Married Lucy Ludwell Grymes in 1753. Founder of the "Leesylvania" Plantation in Prince William Co., Virginia; today the site of the Leesylvania State Park.
Father of Governor and Maj. Gen. Henry "Light-Horse Harry" Lee, Attorney-General Charles Lee, Representative Richard Bland Lee, Mary, Theodoric, Edmund, Lucy and Anne Lee.