Private landlords received £9.3bn in housing benefit last year - almost double the amount they did a decade ago, according to a new report.
The increased number of private renters receiving the benefit and the higher costs of private rentals have contributed to the rise, the National Housing Federation (NHF) said.
David Orr, chief executive of the NHF, said it was "madness" that the money was being spent on the private sector and not being invested in affordable homes.
But the Government said it was taking steps to manage the bill.
In 2006, £4.6bn was paid out in housing benefit to private landlords, but since 2008 there has been a 42% rise in the number of private renters receiving housing benefit, according to the NHF analysis.
The organisation said the Government could save £1.5bn a year if all those housed in the private rented sector lived in affordable housing.
"It is madness to spend £9 billion of taxpayers' money lining the pockets of private landlords, rather than investing in affordable homes," Orr said.
"Housing associations want to build the homes the nation needs. By loosening restrictions on existing funding, the Government can free up housing associations to build more affordable housing at better value to the taxpayer and directly address the housing crisis."