The 77-year-old has tantalised, delighted and disappointed listeners on Radio 2 and then Radio 5 Live but his voice is no longer strong enough to broadcast after surgery to treat throat cancer.
The presenter - popularly known by his initials as JAG - joined the BBC in 1972 and went on to become one of the most recognisable voices on radio.
Edinburgh-born Gordon, whose delivery often made it possible for fans to predict the fortune of their team simply from the inflection, has often admitted he knows little about football other than the results.
Just last year he said he had no plans ever to step down from his role. "I keep telling the BBC they will have to carry me out on a stretcher. I'll die with the microphone in my hand," he said in an interview.
But the BBC said today that although his surgery was successful, "sadly his voice is now not strong enough to broadcast".
Richard Burgess, head of BBC Radio Sport, said: "This is desperately sad news for everyone at BBC Sport and we know our sadness will be shared by many millions of listeners.
"A voice which is, of course, recognised around the globe through the BBC World Service and a voice which embodies authority, clarity and charm.
"For so many of us, James has been a mainstay in our lives - a reassuring and reliable presence every week. He is a broadcasting legend."
- Top 10 retirement tips - special 16-page guide
- How to get the best deal out of different retirement options
Mark Pougatch, presenter of 5 Live Sport, said: "Even people who don't really even like football knew who James was, even if they didn't realise it.
"Such was James's unique style of reading the classifieds, his wonderful inflections and stresses, that even non-believers of the sport knew the result after the home team's score.
"Nobody else will be able to say 'Wolverhampton Wanderers' with quite such mellifluous tones.
"But enough of this 'James'. To those of us lucky enough to work at BBC Radio Sport, he is JAG. And JAG is an institution, a legend and a gentleman all rolled into one."
Gordon, who contracted polio as a child and had to wear leg supports until his late teens, officially retired from the BBC many years ago but has continued as a freelance.
He joined the BBC as an announcer and newsreader after a career in music publishing. He began reading the results in 1974.
Gordon, who is married with a son and two grandchildren, today spoke of his sadness at leaving the job which has been his "life" - and of his pride at enlivening the broadcasts.
He said: "It's with great sorrow that I now have to give up the most exciting part of my career, The Classified Football Results. They have been my life. Such fun getting it right. The most important thing, though, has been making it exciting for the listener.
"I want to thank my producer of some 30 years, Audrey Adams, whose love of sport and determination to get it right has made all the difference.
"How proud I am that I have served the very best of BBC radio sport, albeit in a small way with the classifieds, and I know that the great team will continue to present the best in radio sport coverage."
Gordon worked with the likes of Bert Kaempfert and James Last before auditioning for the BBC when head of presentation Jimmy Kingsbury had wanted to find an announcer with a slight Scottish accent.
Reflecting on his early days, he said: "I was terrified at first but I put my heart and soul into it and have loved it ever since."
Gordon, who was diagnosed with polio at the age of just six months, had played the piano and clarinet and believes it was his musical training which led to his distinctive style.
"I had studied music early on and perhaps I thought 'Hmmm, if I get a rhythm to it, it'll make it easier to read' and that is how it came about really," he recalled in an interview.
He added: "I'd be furious if I mucked it up but I like to think that I don't muck it up. I concentrate so much. You could put a bomb next to me and I probably wouldn't hear it. I just carry on."
Many people have assumed over the years that he delivered a famous fictional result which once formed part of a Two Ronnies sketch. In an interview about his career, Gordon said he had never said the line during the results but still held out the hope he may do so.
"Of course one day I would love to read out on the air Forar Athletic five, East Fife four - I'd love to do that," he said.
Could you save £824 a month for retirement?
Where can you live in luxury on a state pension?
Postcode lottery could lose you 20% of your income