All 281 walk-in tax offices in the UK are to close and a new service will open, in a move unions warned put 1,300 jobs at risk.
HM Revenue and Customs said only a small minority of its 40 million customers used the centres, and demand halved from five million visitors in 2005-06 to fewer than two million in 2013.
Some centres are now open just one day a week, while only one in 10 needed a face-to-face appointment, said HMRC, adding there will be no compulsory redundancies.
Staff can work in the new service, take up other jobs in the organisation or apply for voluntary redundancy.
The Public and Commercial Services union said the closures, from May, will mean that 2.5 million pensioners, vulnerable workers and tax credit claimants will lose a vital service.
HMRC said that following a successful seven-month trial in the North East of England, the new service will provide expert advisers to resolve issues on the phone, or at a range of "convenient locations".
Ruth Owen, HMRC's director general for personal tax, said: "HMRC is dedicated to providing help to customers when they need it. The pilot showed that this is a valuable service for those who cannot get
the help they need elsewhere.
"Our specialised phone service will help customers whose affairs can be resolved over the telephone, and our face-to-face help will be available to those who need it, visiting them at a place convenient to them."
The new service will save customers around £17 million a year in lost time and travel costs, and will save taxpayers over £27 million a year, said HMRC.
PCS general secretary Mark Serwotka said: "HMRC has failed to make the case for closing these offices that provide a lifeline for vulnerable taxpayers.
"These closures seriously undermine the Government's claim it wants to ensure people pay their taxes, and it makes no economic sense to continue cutting jobs from the very department that collects the revenue that funds the public services we all rely on."