The company - well-known for its botched handling of its Olympics security contract - and rival security firm Serco overcharged the Government by tens of millions of pounds, including for monitoring dead offenders.
G4S now faces a criminal investigation after the firm was reported to the Serious Fraud Office when it refused to take part in an additional investigation to rule out any dishonesty.
Serco, meanwhile, has agreed to take part in the independent audit which will look at, among other areas, internal email trails between executives to establish what happened.
Auditors discovered the firms had charged the Government for tagging offenders who were back in prison, had had their tags removed, had left the country or had never been tagged in the first place but had been returned to court.
Justice Secretary Chris Grayling told the House of Commons: "The billing practices in question were clearly unacceptable and the Government will take all necessary steps to secure a refund for the taxpayer."
"The House will also be surprised and disappointed to learn that staff in the Ministry of Justice were aware of a potential problem and yet did not take adequate steps to address it."
Mr Grayling admitted that contract managers in the Ministry of Justice discovered issues with the contract as far back as 2008 - but did nothing to tackle the problem.
A Government-wide review of all contracts held by Serco and G4S, which both saw their shares plunge on London's leading FTSE 100 index on Thursday, will take place following the shock revelations.