Son of James Wesley Ellis and Elizabeth Freeman Ellis. When he was 10 years old, he and his family moved to the Atlanta Prison Farm property. His father was the foreman of the prison farm and his grandfather was the superintendent. Growing up, he worked on a farm, had a milk and paper route, did construction, ran a concessions business at Grant Park, and ran a laundry route. By the time he completed West Georgia College and Georgia State University in 1957 with a BBA in general management, he had worked as a prison guard and foreman, sold insurance, IBM Machines and real estate, and held various other jobs to work his way through college. After returning from serving two years in the U.S. Army, he held a variety of jobs until he found his place in the automobile business with General Motors in 1960. In 1970, he and his wife, Billie, had an opportunity to establish their own automobile dealership. They took the biggest financial plunge of their lives, sold several of their assets, used all of their savings, cashed in Billie's retirement fund from over 15 years at Georgia Tech, and borrowed the balance to begin Jim Ellis Volkswagen in 1971. What began as a significant business risk turned into the Jim Ellis Automotive Group. He was a member of Johns Creek Baptist Church for over 15 years. He was preceded in death by his parents; sons, William Gregory Ellis and James "Jimmy" Edward Ellis; and sisters, Ann Hayes and Betty Turner. Surviving are his wife of 69 years, Billie Sammons Ellis; daughters, Karen Ellis Black and Kristi Ellis Cohron; grandchildren, Stacey Ellis Hodges and husband, Gregory; Brooke Ellis Gatlin and husband, James, James Wesley Ellis, III and wife, Monica, Taylor Elizabeth Cohron, Trey Ellis Cohron; Tara Keltner and husband, Jason, Sheyanne Navarro and husband, Scott, and Caroline Cohron; daughter-in-law, Glynnell Ellis; great-grandchildren, Ivie Ellis Gatlin, Tyler Gregory Gatlin, Ellen Grace Hodges, Andrew Davis Hodges, Asher Jacob Ellis, James Ryland Ellis, and Camden Grant Ellis; and sister-in-law, Patricia Johnson.