Half of those over 50 struggle to see their GP on the same day they fall ill, according to a poll of more than 9,000 people.
One in three said they also find it hard to book an appointment to see their GP in advance, with those aged 50 to 59 struggling the most to book an appointment either in advance or on the same day.
The poll, by Saga Health Insurance, suggests Londoners find it the hardest to get an appointment on the same day, with 60% saying they often struggle, while half of people in Scotland polled said the same.
Kevin McMullan, senior manager of health insurance at Saga, said: "The last thing people want to do when they feel under the weather is battle to see their GP.
"We need to be encouraging these people to visit their GP rather than put more barriers in the way so that they can get better as soon as possible and carry on with their normal everyday life."
The poll comes after a survey of almost a million patients on Thursday found most people have no interest in their GP surgery opening at weekends.
The research showed that 81% of patients do not find current GP opening times inconvenient.
Just 15% said weekend opening would make it easier for them to see a doctor, while only 2% said they would be able to attend Sunday appointments.
When those patients who said they were interested in weekend opening were asked to rate their preferences, 74% preferred a Saturday opening.
The research team from the University of East Anglia (UEA) and the University of Oxford, used data from the national 2014 General Practice Patient Survey.
An official review released by NHS England of seven-day services has also found "very low" demand for Sunday appointments, although evening opening during the week is popular with patients.