Around 1.7 million pensioners had to file a tax return last year, according to new research.
Former pensions minister Sir Steve Webb said retirement "does not mean freedom from the misery of the annual tax return", with more than 250,000 of those submitting tax returns aged 80 or over.
Pensioners also make up a much higher proportion of taxpayers now compared to the early 1990s, further research by Royal London suggests.
Analysis of HMRC data shows that around one in five taxpayers this year will be over 65, compared to around one in nine in the early 1990s.
Sir Steve, a former Liberal Democrat MP who is now Royal London's director of policy, said: "It is clear that even retirement does not mean freedom from the misery of the annual tax return.
"It is shocking that over 250,000 people aged over 80 are still being asked to deal with this paperwork each year.
"The pledge to do away with tax returns altogether cannot come too soon for this group."
More than 10 million people submitted a tax return in 2015/16, according to a freedom of information request by Royal London, an increase of around 750,000 on five years earlier.
HMRC's goal is to make individual tax assessment largely digital by 2020.
A spokesman for HMRC said: "The vast majority of pensioners do not need to fill in tax returns, and we are taking thousands of them out of self-assessment every year.
"We offer a wide range of support for those who need to complete a tax return through our website, helplines and home visits."
HMRC figures suggest that 22% of pensioners actually pay tax and only a small percentage of those need to submit annual returns.