Soaring numbers of prisoners are being returned to custody after being released, creating a "revolving door" system which needs to be overhauled, a campaign group has claimed.
Offenders who are freed to carry on serving their sentence under supervision in the community can be recalled if they fail to comply with the conditions of their licence.
Figures show that in June 1995 only around 150 members of the prison population were recalled offenders. As of June this year, the number was 6,600.
In the period between January and March this year, 5,185 offenders were recalled for breaching the conditions of their licence - an increase of 22% compared with the same period last year.
Under the Offender Rehabilitation Act 2014, licence supervision was expanded so anyone sentenced to more than a day in jail receives at least 12 months' supervision on release.
Failing to report to a probation officer can amount to a breach, while criminals can also be returned to prison if there is any deterioration in behaviour which leads officials to conclude there is an increased risk of committing further offences.
In a submission to the Office of the United Nations High Commissioner for Human Rights, the Howard League for Penal Reform argued that the possibility of recall to prison as a consequence of a breach of licence should be removed.
There is a "lack of due process" in the system, the group claimed, adding that reasons for recall are "broad and often vague".
It said new crimes could be dealt with separately through the usual channels, while other community penalties could be imposed for breaches of licence that do not amount to a criminal offence.
Andrew Neilson, director of campaigns at the Howard League, said: "Why are we sending men and women back to prison when they have not committed new crimes?
"Far from transforming rehabilitation, privatising the probation service and making more people subject to licence conditions has sped up the revolving door, returning people to prison and putting more pressure on a system that fails everyone.
"Removing the possibility of recall to custody would be a more sensible way to help people who are struggling to comply with their licence conditions."
The Howard League also described prisons as overcrowded and warned there has been a decline in safety behind bars since 2012.
The Ministry of Justice said recent figures on recalls are "not comparable" with statistics from the 1990s.
A spokeswoman for the department said: "There have been significant changes to the way we manage offenders which mean comparing them is misleading.
"Public protection is our priority and offenders on licence must comply with a strict set of conditions. If these conditions are breached they face going back to prison.
"Safety in prisons is fundamental to the proper functioning of our justice system and a vital part of our reform plans."