Citizens Advice said consumers who challenge the fees - of up to £625 - were seeing the charges passed to debt collectors.
The charity said some people were finding themselves locked into unsuitable broadband contracts or hit by arbitrary cancellation fees when they switched to get a better service.
One woman was hit by a cancellation fee, even though it was not in the terms and conditions of her contract, when she tried to switch provider after her broadband speed became so bad that she was paying repeatedly to use an internet cafe.
Who is the UK's worst broadband provider?
Another man in his 70s changed provider after his service stopped working altogether following months of problems, but was sent a letter telling him he owed more than £200 for early cancellation.
The demand was then handed over to a debt collection agency.
Those moving house were sometimes hit by early cancellation fees, despite the fact that they could not transfer the service to their new address.
Other broadband-related complaints to Citizens Advice over the last year included "snail's pace" connection speeds, persistent faults and bad customer service.
More than half of problems reported to the consumer organisation were for sub-standard service, it said.
Article continues below
Citizens Advice said consumers phoned in more than 3,300 internet and broadband problems to its bureaux in England and Wales, and more than 4,500 issues were reported to its website across England, Scotland and Wales in the year to June.
The top three issues in England and Wales concerned the cancellation of and exit from contracts, making up 23% of problems, complaints and redress (18%) and costs, billing or payment (15%).
Citizens Advice is calling on internet service providers to scrap cancellation fees for customers who have had persistent problems with their service to ensure that consumers are not forced to remain in unsatisfactory contracts.
It also said providers need to improve their customer service and be "a lot more careful" when handing cancellation fees over to debt collectors.
Citizens Advice chief executive Gillian Guy said: "People are finding themselves held captive by bad broadband services.
"Some consumers who have stood up to problem suppliers have found themselves being punished for switching when they've been hit with a cancellation fee that is then passed over to a debt collection agency.
"Internet service providers must not shackle customers seeking a better service with unreasonable fees that can turn into shock debt. All internet users need to be able to easily have a way out of inadequate contracts and broadband speeds that only give them daily frustration."
Citizens Advice Scotland chief executive Margaret Lynch said: "We want to see a fairer country where people as citizens and consumers are empowered and have their rights respected.
"Arbitrary fees locking people into contracts that don't work for them in are a step in the wrong direction."
More on AOL Money
Get free broadband for two years
Landline free broadband launched