BT-owned Plusnet has been fined £880,000 for continuing to charge more than 1,000 former landline and broadband customers, Ofcom said.
The regulator found that the telecoms company broke a fundamental billing rule by continuing to charge 1,025 customers for landline or broadband after they had cancelled their service, for a total of more than £500,000.
Companies' billing systems must recognise that a home phone or broadband line is "ceased" once the customer cancels the service.
An error in Plusnet's system meant that cancelled lines were still recognised as "live", Ofcom said.
The £880,000 fine must be paid to Ofcom within 20 working days and will be passed on to HM Treasury.
It includes a 20% reduction to reflect Plusnet's willingness to enter into a formal settlement.
Lindsey Fussell, Ofcom's consumer group director, said: "There can be no margin for error and no excuses when it comes to billing customers correctly.
"This fine should serve as a reminder to telecoms companies that they must adhere to Ofcom's billing rules at all times, or face the consequences."
Ofcom said Plusnet had made repeated attempts to refund all affected ex-customers by letter and phone, and had refunded 356 customers a total of £212,140, which included interest at a rate of 4% for each account.
It had donated the remaining funds to a dozen local charities in lieu of payments owed to customers it could not contact.
A Plusnet spokeswoman said: "We are very sorry and would like to apologise to the 1,025 customers affected.
"We reported this ourselves to Ofcom, and made every effort to contact these customers to arrange a full refund before the investigation started.
"We would also like to reassure all customers this was an isolated historic issue and we have implemented a number of new robust measures to make sure this doesn't happen again."