Guy was a jazz trumpeter. He had a promising career as a progressive bop musician working alongside prominent acts until a drug addiction ruined his career.

Guy began his professional music career performing in New York City and joined Fats Waller's backup band in the late 1930s. In 1938, Guy succeeded Dizzy Gillespie in Teddy Hill's orchestra and patterned a style that followed his role model, Roy Eldridge. Despite his range, speed, and potential, Guy never managed to surpass Eldridge's abilities. He became a key soloist in Coleman Hawkins' short-lived big band in 1940.

In 1941 and 1942, Guy was a regular performer as a member of the after-hours band at Minton's Playhouse, a Harlem jazz club, alongside Nick Fenton, Kenny Clarke, and Thelonious Monk in jam sessions with early bop music. During this period, Guy was involved in recordings by Jerry Newman and appeared on songs by Charlie Christian, Hot Lips Page, Roy Eldridge, and Don Byas. Guy began incorporating Gillespie's influences into his playing, with his performance on Monk's song "Epistrophy" (1942), arguably the highlight of Guy's recording career. Much of his appearances as an instrumentalist are marked by his enthusiasm and tempo, though on occasions he would over exert himself and sound erratic.

Guy struggled with a heroin addiction throughout the majority of his brief career. In 1945 and 1946, Guy was involved with singer Billie Holiday both professionally and intimately. Holiday's drug addictions were a problem on the set. She earned more than $1,000 a week from club ventures, but spent most of it on heroin. Joe Guy, her lover and reputed husband, traveled to Hollywood while Holiday was filming and supplied her with drugs. He was banned from the set by Holiday's manager, Joe Glaser.

When Guy and Holiday were arrested for drug possession, the two cut ties. Guy moved to his hometown, Birmingham, Alabama, and faded into obscurity. He performed at the Woodland Club with local musician Frank Adams and advised others about the dangers of addiction.