Emergency admissions to hospitals cost the NHS £13.7 billion in 2015/16, according to a National Audit Office (NAO) report.
Around a quarter of the 5.8 million admissions to A&E departments in 2016/17 were considered avoidable by NHS England, and the report warned that while progress has been made to manage their impact on hospitals, the "challenge of managing emergency admissions is far from being under control".
NAO head Sir Amyas Morse said overloaded A&E units were a "constant point of stress for patients and the NHS".
The report estimates the real terms cost of emergency admissions has increased by 2.2% since 2013/14, while emergency admissions increased by 7% over the same period.
Overall emergency admissions grew by a quarter (24%) from 2007/08 to 2016/17, and the report warned they pose a "serious challenge" to both the service and financial position of the NHS.
While more people are being admitted, it found the time they spend in hospital is getting shorter, with the majority (79%) of the growth in emergency admissions since 2013/14 caused by people who did not stay overnight.
The rise in emergency admissions of people aged 65 and over accounted for more than a tenth (12%) over the past four years.
The impact of the increase can be seen in the fall in the average number of available beds in hospitals, with a 5.8% drop in the number of available general and acute beds between 2010/11 and 2016/17.
Bed occupancy has been increasing generally since at least 2010/11, reaching a seasonal peak of 91.4% in the fourth quarter of 2016/17.
The report also highlighted the issue of emergency readmissions, where patients end up being readmitted to hospital after being discharged.
It said Healthwatch England estimated emergency readmissions to have risen by 22.8% between 2012/13 and 2016/17, raising questions about the appropriateness of some decisions to discharge patients along with the support provided to them once they have left.
It noted that while the rate of growth in emergency admissions has slowed slightly in 2016/17, there is only limited evidence to show that this is due to initiatives such as the Better Care Fund, which seeks to join up health and care services, and NHS England's urgent and emergency care programme.
The report also warned that NHS England does not have good enough data on emergency admissions, as it said it does not always accurately record the causes, severity, source of referral and diagnosis of patients.
It said an assessment by the Royal College of Emergency Medicine and NHS Digital found there were only records on where people come from for 5% of attendances, such as a road traffic accident.
Sir Amyas said: "It is a problem for all of us that A&Es remain overloaded and a constant point of stress for patients and the NHS.
"A lot of effort is being made by NHS England.
"At the centre of this is increased 'daycase' treatment but the decision to stop methodical measurement of emergency readmissions a few years ago makes it difficult to understand whether daycase interventions achieve enduring results."