A council house building renaissance is needed to tackle the deepening "crisis" surrounding affordable homes, according to the Local Government Association (LGA).
The LGA argued councils should be handed stronger powers to spark a housebuilding revival amid the wider economic uncertainty following the vote to leave the EU.
Projections for the LGA covering England found at least 3.98 million working age people will still need access to affordable housing options by 2024.
The projections found that around 5.4 million people of working age will need access to affordable housing by 2024 if qualification levels do not increase.
The LGA said the last time the country was building more than the 250,000 houses that it is estimated the nation needs was in 1977/78 - when councils built 44% of new homes.
It said that in 2013/14, councils were only able to build 1% of new homes.
The LGA, which represents more than 370 councils in England and Wales, is urging ministers to "take urgent steps so councils can resume their historic role as a major builder of new homes and help tackle the nation's deepening housing crisis".
To "spark a revitalisation of council house building", the LGA is calling on Government to allow councils to borrow to invest in housing in the same way that they are able to borrow to invest in other projects and keep 100% of the receipts from properties sold through Right to Buy to build new homes. The LGA argued Right to Buy receipts should be combined with other funding.
A separate report from the Centre for Economics and Business Research predicts the average UK home will still cost around £40,000 more in five years' time, despite the "tremors" from the Brexit vote.
The expected increases mean the average UK house price could rise from £194,000 in 2016 to £234,000 in 2021.
Councillor Peter Box, LGA housing spokesman, said: "A renaissance in house building by councils must be at the heart of this.
"The private sector clearly has an important role to play but the reality is that it cannot build the homes we need on its own, and will likely be further restricted by uncertainties in the months and years ahead.
"Councils want to get on with the job of building the new homes that people in their areas desperately need.
"If we are to stand any chance of solving our housing crisis, councils must be able to replace sold homes and reinvest in building more of the genuine affordable homes our communities desperately need now more than ever."