The Health Secretary said he would not do anything that put public health at risk as he prepared to unveil a consultation on plans to ensure migrants contribute to health costs.
Proposals include a levy on visa charges of around £200 on any students or foreign workers who come to the UK for more than half a year.
Free access to GPs for tourists and those staying less than six months may also be curbed under the reforms.
Mr Hunt said it was "extraordinary" that visitors do not pay for GP and emergency care and he was determined to tackle illegal immigration.
"We have been clear that we are a national health service, not an international health service, and I am determined to wipe out abuse in the system," he said.
"It is right to ask whether that is fair to hard-working British families who pay about £5,000 a year in taxes or whether we need to have a fairer system."
Doctors claim the "rushed" reforms could put UK citizens' health at risk, end up costing more money by pushing people to A&E, and turn GP surgeries into border control posts.
"I don't think we should be turning the GP surgery into a border agency," she told BBC Radio 4's Today programme.
"It is a very rushed consultation. Whatever happens, we need to make sure that what comes out the other end is sensible, proportionate and fair, and doesn't cost us all much more money and put us at much more risk than the current situation which is one that, even at the worst estimates, is a tiny proportion of NHS costs," she said.
National AIDS Trust chief executive Deborah Jack said the changes would "undermine years of work" to encourage marginalised at-risk groups to access HIV testing and treatment.
"If they go ahead, they risk putting lives at risk and accelerating the spread of HIV in the general population," she said.
Mr Hunt told Today there would be an "exemption on all public health grounds" and pointed to other countries which charge for healthcare but do not suffer TB or other problems.
"One of the things we want to consult on is that we don't want to do anything that puts public health at risk" he said.
"But we are one of the only countries in the world that has completely free access to primary care and there are many other countries that do have a charging system that don't have public health problems like TB.
"So I think there is a way we can find to deal with this."
Foreign patients would not be refused treatment in emergency cases, he insisted.
Estimates of the cost of "health tourism" vary from the £12 million known to have been written off by the NHS in 2011/12 to tens of billions of pounds.
Mr Hunt, who has previously claimed the figure is closer to £200 million, conceded that an accurate figure would have to await an audit being carried out by the Department of Health.
His reforms are also looking to change rules so that hospitals are no longer individually required to recoup money from other countries for treatment of their nationals - which has led some not to declare patients as foreign to avoid the burden.
Fees could in future be collected centrally to remove the incentive to fiddle the figures.
"We want to make sure we have a non-bureaucratic system in place so that where someone should be paying for their care, we actually do charge them and we do collect the money later,"Mr Hunt told the BBC.
"That's what most other countries manage. We are the most generous country in the world when it comes to that kind of thing."
Asked about claims that costs would go up because of extra A&E visits, he pointed out that a £200 levy would pay for about 1,000 GPs.
Shadow health minister Liz Kendall said: "In its three years in power, the Government has a poor record on announcing policies that sound good but prove to be completely unworkable.
"We will have many questions to ask about the details when they are published but the key tests for their proposals are: can they be properly enforced and will they save more money than they cost to put in place?
"The public and NHS staff must be confident that any new measures are about getting taxpayers a better deal and ensuring fairness, not playing politics with our NHS."
Shadow public health minister Diane Abbott backed claims that the changes would put people at risk and suggested the Government was guilty of xenophobia.
"What price xenophobia? Stigmatising foreigners accessing NHS creates a public health risk," she wrote on Twitter.
"Stigmatising foreigners accessing NHS will undermine years of work to encourage at-risk groups to access HIV testing."
She said she was "not against people paying who are not entitled to free healthcare"; but added there were "obvious public health risks re communicable disease".
Downing Street said that David Cameron "emphatically rejects" Ms Abbott's assertions.
The Prime Ministers official spokesman said: "This is about fairness. We have a national health service, not an international health service. Why should visitors get healthcare for free while UK residents pay their taxes?
"The Prime Minister would very much underline the point that Jeremy Hunt was making today, which is that there will be an exemption on all public health grounds."
Asked whether Mr Cameron regarded it as inflammatory for Ms Abbott to suggest that the policy was xenophobic, the spokesman said: "He emphatically disagrees. It would be for other people to explain why they felt that was the right word to use."
The PM's spokesman said the Government had no precise figure for the cost of "health tourism" to the NHS.
"One of the things the Secretary of State was saying was that we don't actually know the cost" he said. "That is why he announced we will be doing an audit."
Mr Cameron told MPs:"British families pay about £5,000 a year in taxes to pay for our NHS and it is right to make sure that those people who don't have a right to use our NHS get properly charged for it.
"I had hoped there would be all-party support for it but Labour's public health minister has condemned it as xenophobic so I assume they are now going to oppose this sensible change which working people in this country will roundly support."
In a written statement to the House of Commons, Mr Hunt said the Department of Health consultation on the proposed changes to the NHS in England will last for eight weeks, closing at the end of August.
The Home Office will run a linked consultation on UK-wide proposals for regulations to migrant access to NHS care.
Leigh Daynes, executive director of humanitarian aid organisation Doctors of the World UK, said: "Restricting access to GPs is likely to cost the NHS more money.
"Barring sick migrants from GP surgeries will result in many more admissions to A&E departments, which is costly. It is easier and cheaper to treat most illnesses early, and it has better health outcomes for individuals and communities.
"Excluding migrants from care is an irrational misdiagnosis that will have serious unintended consequences on the health of the nation and public purse."
Dr Laurence Buckman, chair of the British Medical Association & GP Committee, said: "Introducing a new system to assess a patient's eligibility to use NHS services will require significant planning and resources. It must also be discussed with healthcare professionals and take into consideration the huge workload pressure that the NHS is already facing.
"The botched launch of NHS111 demonstrates what can happen if a system is rushed through because of politically driven deadlines.
"The BMA would strongly oppose any system where GPs are required to act as UK Border Force agents and enforce immigration checks. Doctors should spend their time treating patients and not acting as the arbitrators of whether patients are eligible to receive NHS care."