Nearly one in three homes green-lit to be built in England during the past five years have not been completed, Shelter has found.
More than 320,000 homes have not been built despite having been granted residential planning permission, according to analysis by the housing charity, which said ordinary working families are bearing the brunt of the issue of "phantom homes".
An estimated 68% of homes with planning permission have been completed over the past five years, Shelter calculated.
It said the problem is particularly acute in London, where around one in two homes with permission have not been built.
But the Home Builders Federation (HBF) said housing supply has shown significant growth and planning delays mean permissions can take years to process to the point where construction can start.
Shelter included a one-year time lag between a home being given the go-ahead and the build being completed when making its calculations. It said if it had not allowed for a one-year time gap, its findings for the number of homes not built would have been higher.
Shelter used figures from the Department for Communities and Local Government (DCLG) among other data to make its calculations.
The charity claimed the country's housebuilding system encourages developers to sit on land and drip out new homes, to keep prices high.
It said the Government needs to "get tough" and hand powers to councils to tax those who do not build fast enough, as well as taking forward policies outlined in the recent housing white paper, such as granting planning permission to developers based on their track record.
Anne Baxendale, head of communications, policy and campaigns at Shelter, said: "House-builders are trickling out a handful of poor-quality homes at a snail's pace, meaning there are simply not enough affordable homes and ordinary working families are bearing the brunt."
David O'Leary, policy director at the HBF, said: "Housing supply is up by more than 50% in just three years, with the overwhelming contribution coming from national house builders.
"While headline planning permission data is growing at unprecedented rates, a reflection of builders' intention to build more in the coming years, the majority of this land is not at a stage at which it can yet be built on.
"Delays in the planning system mean permissions can take years to process to the point where construction can start, especially on very large sites with complex infrastructure requirements.
"The cost and risk involved in securing planning permission has hampered the ability of small firms to grow, with large companies dedicating significant resource to navigating the process."
He said many "of these so-called 'phantom homes'" will be plots on sites where construction is under way.
Mr O'Leary said: "Oversimplified and ideologically driven analysis distracts from the efforts of builders large and small, public and private, to tackle the housing crisis."
A spokesman for the DCLG said: "The Government has been clear that we want to tackle barriers to stalled developments, so we are investing £2.3 billion to deliver the infrastructure needed to support new homes.
"We know the build out of sites remains slow, that's why in our housing white paper we've set out a number measures to speed up delivery including a new housing delivery test to ensure new homes get built on time and give councils a range of tools to make this happen."