Almost 10,000 motorists have been caught twice for being distracted while driving over the past four years, including by using a mobile phone, figures show.
Nearly 240,000 people have been caught at least once since 2012, but just 284 received a driving ban as a direct result.
The figures for 2012 to 2015 were released by the Driver and Vehicle Licensing Agency to BBC Radio 5 Live following a freedom of information request.
They revealed that more than 600 people had been caught on three occasions, and one was caught five times.
Gary Rae, campaigns director for road safety charity Brake, said: "These are astonishing and worrying figures. It's further evidence that the authorities are not taking illegal use of mobiles behind the wheel seriously enough.
"We need tougher sanctions on drivers who use their devices when driving, and that includes increasing both the fines and penalty points."
On Monday, a lorry driver who killed a woman and three children by ploughing into their stationary car while scrolling through music on his mobile phone was jailed for 10 years.
Tomasz Kroker, 30, smashed into the vehicle carrying Tracey Houghton, 45, her sons Ethan, 13, and Josh, 11, and her stepdaughter Aimee Goldsmith, 11, at 50mph on August 10.
Their car was shunted underneath the back of a heavy goods vehicle and crushed to a third of its size, immediately killing the family, from Bedfordshire, on the A34 dual carriageway north of Newbury in Berkshire.
Ministry of Justice data shows the number of convictions for using a mobile phone while driving halved from 32,547 in 2010 to 16,093 last year.
The number of fixed penalty notices issued has also plummeted, with an 84% drop since 2011.
This is despite a survey of more than 1,700 UK drivers by the RAC showing that almost one in three (31%) admit using a handheld phone behind the wheel, compared with just 8% in 2014.
The breakdown organisation believes a 27% fall in the number of full-time dedicated roads policing officers in England and Wales (excluding London) between 2010 and 2015 has left drivers with no fear that they will be caught for offences not detected by automatic cameras.
Department for Transport figures show that a driver impaired or distracted by their phone was a contributory factor in 440 accidents in Britain last year, including 22 that were fatal and 75 classed as serious.
The Government has proposed doubling the punishment for illegal phone use by drivers.
Motorists caught using a handheld phone are currently given three penalty points and a minimum fine of £100, but this is set to be increased to six points and £200.