The firm, run by former Conservative Party treasurer Michael Spencer, is the fourth organisation to be penalised following the scandal over the rigging of the inter-bank lending rate.
America's Commodity Futures Trading Commission found that ICAP brokers, including one known as "Lord Libor", helped fixed the rate for a period of at least four years.
It fined ICAP 65 million US dollars (£40.5 million), while in London the firm was fined £14 million by the Financial Conduct Authority (FCA).
In a simultaneous announcement, the US Department of Justice announced that it was charging former brokers Darrell Read - who lives in New Zealand - as well as Daniel Wilkinson and Colin Goodman, from England, with fraud.
ICAP is the first broking firm to be fined over the Libor scandal, following a £290 million hit on Barclays, £940 million for Swiss bank UBS and £391 million for the Royal Bank of Scotland.
The FCA said the misconduct by London-based ICAP Europe Limited (IEL) involved a "significant number of brokers" including two managers between October 2006 and November 2010.
It involved brokers colluding with traders at UBS to manipulate Japanese yen Libor rates for the benefit of the traders.
One broker received corrupt bonus payments at the instigation of a manager for his help in the rate-rigging, the FCA said.
Tracey Mcdermott, FCA director of enforcement and financial crime, said: "The misconduct in relation to Libor has cast a shadow over the financial services industry.
"The findings we publish today illustrate, once again, individuals within the industry acting with a cavalier disregard both for regulatory obligations and the interests of the markets.
"IEL's significant failings in culture and controls allowed that misconduct to flourish and fell far short of expectations."
The FCA said the misconduct was widespread and UBS made at least 330 written requests to ICAP brokers for "inappropriate submissions" in relation to the rates, as well as oral requests which cannot be counted precisely.
It said three brokers, including one manager, were "central to the collusion", though at least seven other individuals including another manager, spanning three desks, also participated.
In Washington, the CFTC said that ICAP brokers "knowingly disseminated false and misleading information" about yen borrowing rates to manipulate, at times successfully, the daily yen Libor rate.
It said the brokers, one of whom was known as "Lord Libor" or "Mr Libor" did so to help a "highly valued client" in his "relentless attempts to manipulate yen Libor".
David Meister, the CFTC's director of enforcement, said: "ICAP and other interdealer brokers are expected to be honest middlemen. Here, certain ICAP brokers were anything but honest.
"They repeatedly abused their trusted role when they infected the financial markets with false information to aid their top client's manipulation of Libor.
"As should be clear from today's action, any market participant who seeks to undermine the integrity of a global benchmark interest rate must be held accountable."
Meanwhile the US Department of Justice charged former brokers Read, Wilkinson and Goodman with conspiracy to commit wire fraud and two counts of wire fraud in a criminal complaint unsealed in a Manhattan criminal court. It said each face a maximum 30 years in jail on conviction.
Attorney General Eric Holder said: "By allegedly participating in a scheme to manipulate benchmark interest rates for financial gain, these defendants undermined the integrity of the global markets.
"They were supposed to be honest brokers, but instead, they put their own financial interests ahead of that larger responsibility.
"And as a result, transactions and financial products around the world were compromised, because they were tied to a rate that was distorted due to the brokers' dishonesty."
The prosecution is part of a drive launched by US president Barack Obama to wage an aggressive effort to take on financial crime.
ICAP said it had been co-operating fully with the Department of Justice investigation.
Chief executive Michael Spencer said: "We deeply regret and strongly condemn the inexcusable actions of the brokers who sought to assist certain bank traders in their efforts to manipulate Yen Libor.
"Their conduct contravenes all that ICAP stands for. As soon as their actions came to light, we provided assistance to regulators in the US and UK to understand what had happened.
"None of the three individuals at the centre of the activity remains with the firm. Others are either no longer with the company or are being disciplined.
"There were no findings that any senior management were involved in this matter nor that the firm engaged in deliberate misconduct.
"Both the CFTC and the FCA have acknowledged our co-operation and the FCA notes the significant improvements the entire Group has made to its control infrastructure since 2010."
He said ICAP, which employs 5,000 people around the world, had learned lessons and would further improve its risk and compliance systems.