Broadmoor high security hospital has been rated inadequate by a watchdog with concerns raised about patients being physically restrained too often.
Inspectors from the Care Quality Commission said they did not see convincing evidence that seclusion and restraint were only being used in cases when it was deemed absolutely necessary.
The withdrawal of extra pay for some staff working at Broadmoor, which houses some of Britain's most notorious male criminals including Yorkshire Ripper Peter Sutcliffe, has also led to problems recruiting people to work there, it said.
Issues over staffing levels do not pose a threat to security at the West London Mental Health NHS Trust unit, but they raise concerns around cancelled activities and patients spending too much time in their rooms, the report said.
The trust, which covers a wide range of services including community-based and specialist provision for more than 33,000 people, received a "requires improvement" rating in relation to being safe, effective and well-led, following the five-day inspection in June.
Recommendations include ensuring seclusion facilities are in good condition, making sure staffing levels are sufficient to guarantee patient and staff safety, and better staff engagement to make sure employees feel their input is always valued.
Inspectors rated the trust as "good" for being caring and responsive, and said staff at Broadmoor in particular were seen to show "real concern for patients on an individual basis and a desire to see them progress towards recovery".
Trust chief executive Steve Shrubb welcomed the positive rating for staff but recognised the need to cut down on the frequency of patients being restrained, and said conditions for staff must be improved.
He said: "We need to speed up the work the leadership team is undertaking to create the conditions for our staff to be the best they can be. This includes increasing staffing, improving staff engagement and morale and reducing the use of restrictive practices including restraint and seclusion."
Dr Paul Lelliott, the deputy chief inspector of hospitals and lead for mental health, said staff shortages in the trust are having a "noticeable impact on the quality of some services".
"In the forensic services, this is compounded by low morale. As well as affecting the quality of care, there is the risk that staff might not show the openness, transparency and honesty that are essential to provide safe care.
"We were concerned at the apparent overuse of physical restraint, and the failure to keep proper records. Staff must use restraint only as a last resort, and minimise the use of restraint in the prone (face-down) position. They must record the use of all types of restrictive intervention. They must also make the necessary physical health observations to ensure the safety of patients who have been given an injection to manage disturbed or distressed behaviour."
He added: "West London Mental Health NHS Trust is a large organisation with many parts. Despite the problems described above, our inspectors visited other services where staff were positive about the work of the trust and where care was delivered by hard-working, caring and compassionate staff."