The Local Government Association claims taxpayers would be more than £20 billion per year better off if HM Revenue and Customs was able to match the tax collection rates achieved by councils.
Tax which goes uncollected by HMRC is costing the equivalent of £1,370 for every household in England and Wales, it claims.
The LGA called for a commitment to at least halve the uncollected tax deficit by the end of this Parliament.
Cllr Sharon Taylor, chairwoman of the LGA's finance panel and Labour leader of Stevenage Borough Council in Hertfordshire, said: "If central government's collection rates were as good as those of councils, the public purse would be more than £20 billion better off.
"That's almost double the size of the total spending cuts announced in the Chancellor's spending round.
"When the taxman's balance sheet falls short, it takes a toll on public services across the board.
"Local government is the most efficient part of the public sector. Council tax has one of the highest collection rates of any tax. We cannot afford for local services like road maintenance, libraries and social care to continue paying the price of those who evade and avoid their tax liability. We all need government to do better in tackling this.
"Government has made positive noises on tackling unpaid tax. We now want to see a clear timetable which sets out how government will significantly improve their tax collection record."
An HMRC spokesman said: "The LGA have fundamentally misunderstood the nature of the tax gap which is not a measure of tax unpaid.
In fact HMRC already collects 99% of all taxes and duties which are collectible from a very wide customer base of 60 million taxpayers, increasing our tax take by £1.4 billion to £475.6 billion in the last year alone - on average we collect £1.3 billion a day for the UK."