Cyber attacks and smuggling by criminal gangs contribute to £16 billion a year of tax fraud, HM Revenue and Customs (HMRC) estimates.
The figure, contained in a report by spending watchdog the National Audit Office, makes up almost half of the total £34 billion tax gap - the difference between the revenue that should be collected each year and the amount actually received.
The report suggested that criminal attacks were responsible for a loss of £5.1 billion a year of revenue, while evasion amounted to £4.4 billion.
The hidden economy - made up of "ghosts" whose entire income was unknown to HMRC and "moonlighters" who keep some sources of income secret - results in a £6.2 billion loss.
The NAO said the criminal attacks included smuggling genuine or counterfeit products, such as alcohol or tobacco, into the UK to evade duties.
It also included "cyber attacks on HMRC systems to fraudulently claim tax repayments".
The report found major contributors to the risks associated with evasion and the hidden economy were small and micro-businesses and individuals with complex tax affairs.
In 2014/15 HMRC reported £26.6 billion additional revenue from its work to improve compliance, including efforts to tackle tax fraud but also work to tackle other parts of the tax gap like error and legal tax avoidance.
The NAO found there was only partial data on how much of that revenue was from work to tackle fraud, estimating that the proportion was between 30% and 40%, or between £8 and £10.6 billion.
HMRC has met a target to increase its number of prosecutions by 1,000 by 2014/15, but the NAO said it "needs to better prioritise the cases it selects for criminal investigations".
The drive to meet the target had led to a focus on "less complex cases" with a "large number of prosecutions for evading income tax, VAT and tobacco duty, and lower value cases".
The NAO said HMRC had recognised it needed to "align the cases it selects for criminal investigation more closely with its analysis of risk and how effective a successful prosecution would be in creating a deterrent".
The report said it was difficult to assess the effects of the prosecutions and HMRC research had not been able to find evidence the cases led to changes in behaviour or increases in tax revenues.
Meg Hillier, chairwoman of the Commons Public Accounts Committee, said: "A staggering £16 billion is still lost to tax fraud every year.
"HMRC clearly needs to think harder about how it tackles tax evasion, the hidden economy and criminal attacks.
"The committee will be examining HMRC's work in this area throughout this Parliament.
"Time and time again we hear that Government departments don't have the data or information that they need to plan or evaluate their activities properly, despite them being responsible for setting up these projects or programmes in the first place.
"HMRC is no different in this respect.
"HMRC needs to use the powers and sanctions it has to make a public example of those who break the rules."
Sir Amyas Morse, the head of the NAO, said: "HMRC loses £16 billion a year due to tax fraud, but reducing these losses is not straightforward.
"HMRC has met its targets to raise more tax revenue in the short-term.
"It now needs to consider whether its overall strategy is designed to achieve the best long-term outcomes.
"We will be evaluating HMRC's performance in tackling different types of tax fraud in more depth.
"As we do so, we will be looking for further improvements in the way HMRC uses data and analysis to understand the effect of its actions in both the long and short-term."