More than three quarters (76%) of drivers say potholes should be filled within a week, a survey has found.
The AA poll of more than 18,000 motorists also found that almost a fifth (18%) think 24 hours is the maximum time it should take for repairs to be completed.
Councils are expected to fill the most dangerous potholes within a few hours, but it can take several months to fix defects deemed less significant.
Last month, a coroner urged Surrey County Council to reconsider how it rates the severity of potholes after a cyclist was killed.
Ralph Brazier, 52, was thrown from his bike in Weybridge on March 1 last year after riding over a pothole which had previously been given an ineffective repair and was assessed by the council as only requiring further attention within five working days.
AA president Edmund King called for the "worsening condition of our road network" to be addressed quickly to prevent deaths of vulnerable road users such as cyclists and motorcyclists, and reduce compensation payouts for vehicle damage.
He said: "It can't be right that one council could fill a lower-risk pothole within three weeks, while a neighbouring council needs four weeks.
"Councils should have a back-stop deadline, but with emphasis on carrying out the repairs quicker.
"With the General Election just around the corner, it would be an ideal time for all parties to tell the public how they intend to repair and improve our roads.
"Whoever forms the government on June 9, they will need to act fast or we will continue to have moon-like craters on our roads into the next parliament."
A recent study by the Asphalt Industry Alliance found that local authorities need more than £12 billion of funding to bring the road network up to scratch.
The gap between the amount councils say they received in the last year and what they require to keep roads in reasonable order is almost £730 million.
A spokesman for the Local Government Association, which represents more than 370 councils in England and Wales, said: "Councils take the maintenance of their roads very seriously, fixing a pothole every 19 seconds.
"This amounts to 1.75 million potholes per year, at an average of over 10,400 per local authority."