Short prison sentences of less than 12 months should be a "last resort", Justice Secretary David Gauke said.
Prison terms of less than a year do little to rehabilitate offenders, with two-thirds of inmates going on to re-offend after release from a short-term sentence.
Mr Gauke told The Times he would like the overall prison population to come down.
"Twenty-five years ago the population was 44,000. Today it's 84,000," he said.
"I would like it to fall."
He said efforts to cut the number of people incarcerated would depend on "how successfully we can build confidence in non-custodial sentences and how effective we can be in reducing re-offending".
Mr Gauke said the rise had been driven by "longer and tougher" sentences for serious crimes.
But he acknowledged concerns about the role of shorter terms: "There is an issue about public protection, but I think we need to look at the efficacy of short sentences."
He said: "The evidence shows that when the person has been inside for less than 12 months the re-offending rate is about 66%, but the re-offending rate for those who get a non-custodial sentence is lower.
"Short sentences should be a last resort."
He also hinted at a new approach to dealing with older prisoners, with the newspaper reporting that could mean secure old people's homes.
"We've got 1,600 prisoners over the age of 70," he said.
"We haven't developed precisely what the right answer is, but clearly there are different needs, and they're not a danger to society in large part."