Born in Athens, Alabama, Andy Nelson was a four-sport athlete at Athens High School playing football, basketball, track and baseball, serving as co-captain of the 1951 team and winning letters for each of his four years he completed.

In college, Nelson played quarterback and defensive back for then Memphis State University. A four-year letterman (1952 -1956), he gained All-American recognition in his senior season and graduated with a BS in Health and Physical Education.

Drafted in the eleventh round by the Baltimore Colts in 1957 as a quarterback and defensive back, Nelson played seven years with the Colts as a strong side safety earning All-Pro recognition several years.

The Colts made it to two world Championships in 1958 and '59 with Andy Nelson defending. Nelson also made a great showing when the Colts took on the New York Giants in 1958 - the first overtime (sudden death) game in NFL history, later dubbed "the greatest game ever played". Nelson was selected for the All-Time Baltimore Colts team for the 50th Anniversary in 2003 after playing seven years with the Colts.
After enjoying an eight year career in the NFL including one year for the Giants in 1964, Nelson spent two years as a player and defensive coach with the Harrisburg Caps of the Atlantic Coast Football League, the Colts farm team, where he led the league in pass defense before going on to be named the head football coach.

Enjoying the experience, Nelson spent the next five years coaching where he won League Championships three of the five years including as the Norfolk Neptunes Defensive Coordinator (1970 Championship), and the Chambersburg Cardinals of the Seaboard League (1972-73). Nelson was also defensive coordinator for Philadelphia Bell in the World Football League from 1974 - 75.

Nelson is a current resident of Maryland where he owns and operates Andy Nelson's Barbecue with help from some of his seven children. The century-old tradition of serving "serious barbecue" was passed down from his father.

Nelson was also voted into the Limestone County Hall of Fame (2003) in Alabama and the Memphis State Hall of Fame (1976).