Research from Which? found two-thirds of people (67%) believed firms used high-rate numbers to discourage people from calling them.
Three quarters (75%) said they were reluctant to contact customer services if they had to call a high-rate number, while three in five (63%) would be put off making complaint due to a costly call charge, t he poll of 2,070 people found.
Which? has now launched a campaign to call for all companies to provide a basic, local rate number for customer service enquiries and complaints by telephone.
Which? executive director Richard Lloyd said: "It's outrageous that consumers are faced with a high phone bill just to ask a question or make a complaint.
"It's no wonder that people think companies do this deliberately to deter them from complaining.
"We want an end to all costly calls for customer service and complaints, and new rules so that all companies have to provide a local rate number. There should be no exceptions."
Under recent changes to the EU Consumer Rights Directive (CRD), calls to customer helplines must be charged at no more than the basic rate.
But financial services and public bodies are not included in the legislation and the Government is currently consulting on whether to include travel, timeshare and package travel in the ban, Which? said.
The Which? Costly Calls campaign has called for t he Financial Conduct Authority to clarify existing rules to stop financial services companies from using costly numbers for complaints lines.
The Government should extend the CRD ban to the travel industry and public bodies should also stop using costly numbers, the consumer group said.
Many high street banks use 0845 numbers for their complaint lines including HSBC, Natwest, RBS, Santander, First Direct and Halifax, Which? said.
Public bodies including the Student Loans Company, the Environment Agency Flood Line and the Redundancy payments service helpline also use 0845 numbers, it added.
The National Audit Office revealed that callers to higher rate public service helplines paid £56 million in call charges in 2012-13.
In its research, Which? also found two thirds (66%) of people had chosen a different method to complain, such as emailing, to avoid calling a high-rate number.
Four in five people (80%) believed companies who have high rate phone numbers do not value them as a customer.
Almost half of people (49%) have regretted calling a high-rate number in the past year because of the cost, while three in five people (58%) said they could not afford to call a high-rate number from their mobile.
Four in five (79%) did not think it was reasonable for public bodies to have high-rate numbers, according to the poll.