He was called "Barkin' Joe" Kelly when he was a star guard for the University of Southern California. But for those who knew him best, his coaching behavior was not that of a barker. Kelly was raised in Wisconsin where he played on the unbeaten state champion Madison High School basketball team. He caught the eye of USC, who offered him a scholarship, where he eventually became an All-Coast selection. His biggest success came from coaching where he developed winning teams at Fresno High School and Fresno City College. That winning tradition was fostered during his playing days where he went sixty-three consecutive games without tasting a defeat, beginning in high school and into his sophomore year for the Trojans. Kelly even caused a change in basketball rules when he sat on the ball and read a newspaper while Stanford stubbornly stayed in a zone defense. Collegiate rules were changed so that you had to keep the ball moving even in a stall. Kelly remembered that, during one game in his twelve-year career at Fresno City College, his under-manned team was playing Allan Hancock College with future USC All-American and pro star John Rudometkin. John Toomasian, who succeeded Kelly as the Rams coach, was his assistant for five years and remembers that game well. "Joe told me John, there is no way we can beat these guys.' But I told him if we held the ball and split the floor, it is possible we could," Toomasian said. "I knew a lot about the spread, so we tried it and it almost worked. We missed a couple of layups or it would've." The final score was 10-6 and everyone all over the country wanted to know how it happened, especially against a team like Hancock.
Kelly began his coaching career at Tehachapi High School and then continued for two years at the San Diego Naval Training Center. When he was discharged, he went to Fresno High in 1942 and in ten seasons with the Warriors was 133-56, winning four league championships and the Valley title in 1951. He was a solid fundamentalist and instilled confidence in his players. When All-City guard Larry Huebner went to UCLA and turned out for the Bruin freshman team, the coaches asked him where he learned his fundamentals, "Joe Kelly taught me" was the short answer that Huebner gave them. Kelly finished out his career with twelve seasons at Fresno City College where his Rams won two state championships in 1955 and 1963 and six conference crowns. His teams also finished as runners-up for the state title in 1956,1964, and 1965. His overall record of 275-102 equated to a .729 average.
He was inducted into the Community College Men's Basketball Coaches Hall of Fame. "Joe was a good coach because his head was always into the game, Toomasian said. "He was my mentor and he gave me the job, so eventually I became head coach. Joe was pretty laid back on the bench and didn't shout at his players and never used profanity...He never used more than six or seven players at Fresno City until the game was decided one way or the other. His teams were very difficult to beat more than once in the same season." Kelly also assisted Erwin Ginsburg with his football teams. During some of his coaching years, he also officiated various high school sports.