People using the pension freedoms have seen as much as 10% of their retirement pot swallowed up by providers' charges, research from Citizens Advice has found.
The charity also found seven in 10 (70%) people accessing their retirement pot under the freedoms are not shopping around for different products, meaning they potentially risk ending up with a poor value deal that does not meet their needs.
Fears that switching provider may incur larger fees than staying put appear to be putting many people off shopping around, the findings suggest. One in six (15%) people stuck with their provider because they did not want to be hit by exit charges.
Citizens Advice found that people have been faced with "high" and "confusing" exit fees and other charges to get at their savings.
It estimates from its findings that 160,000 people have paid to access their pensions since the freedoms were launched in April 2015.
Two-fifths (41%) of people using the pension freedoms told Citizens Advice they paid at least one type of charge.
Those facing fees reported an average cost of £1,577 in total - and "most worryingly", those with smaller pots faced higher average charges - Citizens Advice found. People who have pensions of £20,000 or less who have faced fees were paying an average of £1,966.
Citizens Advice said: "For some consumers this can mean they have lost 10% of their retirement savings to charges levied by providers."
Some people are also facing long waits to access their money, Citizens Advice found. And more than half of consumers received their first payment within a month of making a request, around one in six (16%) had to wait over two months.
The charity made its findings from a survey more than 500 people who had accessed their pension since the launch of the freedoms, which give over-55s more flexibility over how they access their cash rather than being required to buy a guaranteed income called an annuity.
Some consumers are finding that their providers are not offering the full range of freedoms, Citizens Advice said, with more than two-fifths (44%) of those who switched provider saying they did so to access the product they wanted.
The Financial Conduct Authority (FCA) recently set out plans to clamp down on early exit charges, which may be levied when someone transfers or takes their pension benefits after the age of 55 but before their selected retirement age.
The FCA has proposed that exit charges should be capped at 1% of the value of the pot.
But Citizens Advice believes the proposed cap is too high and a standard £50 exit charge to cover admin costs should be put in place.
Citizens Advice helps deliver the Government-backed Pension Wise guidance for people using the retirement freedoms. Citizens Advice said its report reflects its views as a consumer champion and not those of Pension Wise.
Gillian Guy, chief executive of Citizens Advice, said: "Picking a pension product is one of the biggest financial decisions people will ever make, so it's worrying that so many aren't shopping around.
"More and more consumers are choosing drawdown products but our research shows they aren't checking whether they're getting the best deal.
"The Government and industry needs to work together to make it easier for consumers to compare drawdown products and choose the one which best meets their needs.
"The threat of excessive charges can also put people off making the right pension choices for them."
Former pensions minister Steve Webb, who is now policy director at Royal London, said: "As this valuable report shows, consumers will not get the full benefit of the new freedoms unless there is much more shopping around.
"Many consumers could benefit from impartial financial advice which could make a real difference to the value for money that they get from their retirement savings.
"Much more needs to be done to increase awareness of the value of impartial advice and guidance at retirement."
Yvonne Braun, director of policy, long term savings and protection for the Association of British Insurers (ABI), said it is important that people consider the choices available to them under the pension freedoms, which have given savers a "far wider range of options than ever before".
She continued: "Pension providers want customers to have all the information they need to make the right decision for them, and are committed to promoting the Government's valuable Pension Wise guidance service.
"Our latest industry figures, which measure what customers actually do, show that 58% of savers switched provider when buying a drawdown product and 38% switched when getting an annuity. A higher number of customers are likely to have shopped around but then found they were already with the best provider for them, so won't have switched. We also estimate around half of customers who stayed have guaranteed annuity rates."