Hospitals are failing to raise concerns about incompetent locum doctors, a new report warns.
Some hospitals take no action when they see poor practice - even though some stand-in doctors are demanding as much as £155 per hour for covering shifts.
A new report into five-yearly checks on doctors for the General Medical Council (GMC) found some locums were falling through the gaps when it came to having their fitness to practise assessed.
It also found some patients were too scared to offer feedback on their family doctor, while most medics only sought patient opinions one day every five years.
The study warned that information on locum doctors was not always shared between the hospitals where they worked, while some hospitals were "unwilling to provide frank feedback" when a doctor was not up to par.
Sometimes hospitals told agencies - which make millions of pounds from supplying temporary doctors - "Please don't send this doctor again" but did not give full reasons why.
While monitoring of locum doctors was better than it used to be, there were still "weak" parts of the system, the report went on.
There are 86 locum agencies across England, covering 8,517 locum doctors.
In his report, Sir Keith Pearson, chairman of the GMC's revalidation advisory board and chairman of Health Education England, said he was concerned that some doctors, including some locums, were "falling outside the most exacting standards" of the five-year checks for doctors, known as revalidation.
He said there was "no obvious mechanism" for identifying low levels of concern about a doctor's competency and that not all doctors were overseen by an officer with responsibility for their appraisal.
Figures also showed that agencies deferred 36% of checks on stand-in locums, more than twice the 16% for other types of organisations.
The report said it was "not yet fully understood" why this was the case, but could include failings by agencies, problems with the performance of locums or difficulties in getting their paperwork together.
Sir Keith said: "If you have a locum who is not as supported as they would be in another environment ... it leaves open the possibility that there could be a patient safety issue."
Data from NHS Improvement in December showed that some stand-in doctors were earning as much as £155 an hour for covering shifts in hospitals with serious staff shortages.
The Government has introduced an agency cap on how much should be paid to locums, although some hospitals were forced to compete for staff and pay above these rates.
The new report also warned that some patients were put off providing feedback on their doctor because they feared offering negative comments would affect their future care.
While doctors felt patient feedback was one of the most important aspects of their five-year appraisal, patients feared the "official" appearance of the questionnaire, it said.
Doctors also often only collected feedback one day every five years to meet the requirements of revalidation, collecting around 40 or 50 responses each time.
Sir Keith said the collection of 50 pieces of patient opinion over five years "does not seem to be an awful lot".
He also confirmed it was the doctor who collected the feedback from patients, although those carrying out the appraisal were there to analyse it.
The GMC welcomed the report.
A statement said: "Sir Keith has delivered a clear and incisive report and we are grateful to him for the quality of his review.
"We are reassured to hear that revalidation is becoming embedded locally and beginning to impact on clinical practice, professional behaviour and patient safety.
"But revalidation is still a new process and we acknowledge the difficulties and challenges that Sir Keith identifies."
It said there were clear areas for action, including ensuring patients were more engaged in providing feedback to their doctors.
It added: "We share Sir Keith's view that oversight of, and support for, locum doctors needs to be strengthened - particularly for those in secondary care - and are working with NHS England and the Department of Health already to address the issues the report identifies."
It pointed to the "significant variation in the resources and quality" of locum agencies which were not always "fulfilling their obligations to doctors and that relevant information is not always transferred when locum doctors move between locations".
Health minister Philip Dunne said: "While this report acknowledges that making sure doctors are up to date is vital to providing high quality patient care, we know there is more to do to ensure that revalidation is as streamlined and effective as possible for doctors.
"Particularly, the report identifies a crucial issue with locums being revalidated for the roles they are undertaking, so we'll want to see further progress in future."
Dr Mark Porter, chairman of the British Medical Association (BMA), said: "The revalidation process helps strengthen patient trust by ensuring doctors have a system of regular appraisals, and this must apply to all doctors, including locums.
"Ideally the use of locums or agency staff would be minimised by having better long-term investment and staffing levels but in the meantime, locum agencies should all do what the best do, which is to organise and provide continued professional development, appraisal and revalidation for these doctors.
"The entire purpose of revalidation could otherwise be weakened by those locums falling through the cracks."