Up to 30,000 passport applications have been hit by delays, David Cameron has said.
The Prime Minister insisted that hundreds of extra staff have been drafted in and new offices are being opened in Liverpool next week in to help tackle the backlog.
Ed Miliband called on the premier to "get a grip" as he warned that thousands of people face cancelled holidays as a result of the problems.
But Mr Cameron accused the Labour leader of trying to "frighten people".
He said: "We have 300,000 extra applications than is normal at this time of year. We have increased massively the staff. The level of applications outside the normal three-week limit is less than 10% of that 300,000."
Unions have claimed more than 500,000 applications are waiting to be dealt with and say delays in the service are down to the loss of 300 jobs and the closure of 20 passport offices over the past five years.
Photographs taken by anonymous members of the passport agency's staff have emerged that are claimed to show boxes of passport applications stacked up and waiting to be dealt with in the Liverpool office.
Earlier today Home Secretary Theresa May said the Passport Office has extended its hours to open from 7am until midnight, while staff are working longer hours and more days of the week to tackle the surge in applications.
Some 250 staff have been redeployed to frontline passport services who were prioritising all outstanding applications, which would allow for an extra 25,000 application examinations every week, MPs were told.
An extra 650 workers have been drafted in to work on customer helplines.
Mr Miliband said applicants were being forced to pay extra to try to fast-track their applications.
"To add insult to injury people are being told if they want their applications processed in the three-week target they have to pay £55 extra."
The Labour leader accused Mrs May of being too busy battling against Education Secretary Michael Gove to focus on the passport problems.
He said: "The truth of the picture of this Government is we have the Home Secretary fighting with the Education Secretary but not paying attention to the business of Government."
"The truth is that is tens of thousands of people are finding that their holidays are being cancelled because they are not actually getting a passport," he added.
"He says they have increased the resources at the passport agency - that is not the case. There are greater responsibilities for the passport agency since 2010 and fewer resources."
Labour said that numbers of staff at the Passport Office had been cut sharply since 2010/11, when Home Office accounts recorded 4,017 workers.
Since then, the accounts show there were 3,134 staff in 2011/12 and 3,284 in 2012/13, while minister James Brokenshire told the House of Commons yesterday that, on March 31 this year, the Passport Office had 3,444 full-time equivalent staff.
A senior Labour source said ministers were being "misleading at best" if they claimed an increase in staff numbers by using a baseline of 2011/12.
The source added: "During that same time the workload has increased dramatically because of the Passport Office taking over from the Foreign Office the job of issuing passports overseas. One of the big reasons for the backlog is that they have had a huge increase in responsibility and a decrease in staff."
Some 85 Labour MPs have reported a total of more than 500 constituents coming to them with worries about delayed passport applications.
Labour is calling on Mrs May to make an urgent statement to the House on the problems and for ministers and the Passport Office chief executive to provide reassurance that action is being taken to ensure that the three-week processing deadline is met.
The party also wants a report to the House on measures to ensure there is no repeat of the current crisis.