The Financial Conduct Authority (FCA) said the bank broke reporting rules on more than a third of its wholesale market transactions for more than five years until February this year.
The FCA said many of the problems were compounded by the 81% state-owned bank's disastrous takeover of Dutch lender ABN Amro in 2007 - but said RBS should have been able to overcome these challenges and ensure that adequate systems and controls were in place.
The FCA said RBS failed to accurately - and in some cases failed altogether - to report financial transactions for assets such shares, government bonds and derivatives between it and other financial firms.
The fine is the latest blow for the semi-nationalised lender after its respected boss, Stephen Hester, revealed last month that he is leaving after being ousted by Chancellor George Osborne.
And it follows a £390 million fine imposed by US and UK authorities in February for rigging the Libor interbank lending rate.
Tracey McDermott, the FCA's director of enforcement and financial crime, said: "Effective market surveillance depends on accurate and timely reporting of transactions.
"We expect firms to get it right."
The FCA said RBS failed to properly report 44.8 million wholesale transactions between November 2007 and February 2013, and 804,000 transactions between November 2007 and February 2012 were not reported at all.
That meant 37% of its deals during the period breached reporting rules. Firms must report details including when a product has been traded, with whom, what price and the quantity.
These reports should have been submitted to the FCA's predecessor, the Financial Services Authority.
The ABN Amro deal at the peak of the market led to the collapse of RBS in November 2008, when the credit crunch forced a £45 billion government bailout.
But the FCA said that, despite the ABN problems, RBS had considerable resources to overcome these and ensure it met reporting rules.
The regulator said RBS's fine would have been more than £8 million if it had not settled early.
RBS said: "RBS fully cooperated with the regulator throughout the investigation. We regret the failings that were uncovered and have subsequently made significant investments to our systems and controls in this area."