The penalty was handed down by the Gambling Commission over a 2009 incident that went undiscovered for six years until 2015, when the regulator and police were alerted.
The commission would not confirm how much Camelot had paid out, but said that the £3 million penalty included £2.5 million "to represent the amount that would have been received by good causes had the prize claim not been paid".
It said while its investigation "could not be certain a fraud had taken place, it was more likely than not that a fraudulent prize claim had been made and paid out".
Camelot chief executive Andy Duncan apologised, saying: "We accept that, at the time, there were some weaknesses in some of the specific controls relevant to this incident and we're very sorry for that."
The commission said Camelot had already taken steps to ensure that a similar problem would not happen again.
The watchdog's chief executive, Sarah Harrison, said "The Gambling Commission's chief concern is to ensure the National Lottery is run with integrity and that player interests are protected.
"Camelot's failures in this case are serious and the penalty package reflects this. Importantly, the package also ensures that good causes will not lose out as a result of Camelot's licence breach.
"Lottery players can feel reassured that our investigations have found no evidence of similar events happening and that controls are in place today to mitigate against future prize payout failings of this type."
Camelot said police had decided earlier this year to take no further action over the alleged fraud.
Mr Duncan said: "It's really important that people understand that this allegation relates to a unique, one-off incident dating back to 2009 and involves a potentially fraudulent claim on a deliberately damaged ticket. It has nothing to do with the National Lottery draws themselves.
"We've strengthened our processes significantly since 2009 and are completely confident that an incident of this nature could not happen today. We welcome the Gambling Commission's confirmation that this is the case."