Borrowers needing as little as £100 could be charged up to £156 more by their bank than it would cost at a payday lender, Which? has found.
The consumer group is urging a crackdown on the "punitive" charges that can be incurred when someone goes into an unauthorised overdraft.
Which? compared the cost of borrowing £100 for 30 days using an unauthorised overdraft.
It found some banks charge more than seven times the maximum £24 charge on a payday loan.
Customers who need £100 could end up paying as much as £180 in fees at their bank if they borrowed the money across two monthly billing periods, according to the findings.
Which? found that NatWest could charge £180 to borrow for 30 days over two billing periods, a difference of £156 compared with a payday loan costing £24.
Customers using Lloyds or Santander could be charged £160 to borrow across two billing periods, £136 more than a £24 payday loan, Which? found.
NatWest said in a statement: "We encourage all of our customers to contact us if they are going to enter unarranged overdraft, regardless of the amount or the length of time.
"We offer a number of alternative solutions, such as putting an arranged overdraft in place, where the costs are considerably less.
"Customers are not charged if their unarranged borrowing is £10 or less, and our act now alert service alerts the customer of upcoming unarranged borrowing to allow them time to transfer money to avoid unarranged overdraft fees."
Lloyds and Santander said they have a range of tools in place to help customers manage their money.
Payday loan charges were capped in 2015, as part of moves to prevent people getting trapped in a debt spiral.
The Financial Conduct Authority (FCA) recently announced plans to put the high-cost credit sector under the spotlight, including overdrafts as well as payday lenders, pawnbrokers and firms offering rent-to-own credit on household goods.
The Competition and Markets Authority (CMA) previously found poor price transparency surrounding overdrafts generally, particularly when a customer slips into an unarranged overdraft.
Vickie Sheriff, Which? director of campaigns and communications, said: "It's not right that people with a financial shortfall can be charged so much more by the big high street banks than they would by a payday loan company, especially if the money is borrowed over two monthly charging periods.
"If banks can continue to set their own charges, then consumers will continue to be hit by exorbitant fees.
"The Financial Conduct Authority must use its current review to cap these high charges and ensure consumers cannot be charged more for unarranged overdrafts than arranged overdrafts."
Mike O'Connor, chief executive at StepChange Debt Charity, said the research shows some of the serious problems that overdrafts can cause and highlights the need for urgent action.
He said: "Every day we help people who have regularly exceeded their overdraft limit and been hit with extra fees and charges.
"This has made getting back on track the next month an even more difficult challenge."
A spokeswoman for the British Bankers' Association (BBA) said: "Banks have invested heavily in digital technology to alert customers when their account might be slipping into the red and help them to avoid unnecessary fees and charges.
"We would always encourage customers who think they might need to borrow money to speak to their bank to pre-arrange an overdraft facility, so they can be certain that payments will be made and keep borrowing costs down."