Michael Gove is facing calls to pull out of the Tory leadership race after admitting taking cocaine.
The Environment Secretary acknowledged he was "fortunate" not to have gone to prison after taking the class A drug on several occasions around 20 years ago.
Former Tory chairwoman Baroness Warsi said it was "completely inappropriate" for him to remain in the contest to be the next prime minister.
She said: "This case isn't just about drug taking, it is about trust, it is about hypocrisy of the highest order and it cannot be that we have somebody who is now mired in this issue of trust and hypocrisy feel that it is still appropriate for him to stand as leader of the Conservative Party and a prime minister of this country."
The former CPS solicitor told Channel 4 News it was "a matter for the Crown Prosecution Service" whether Mr Gove should face prosecution for the offence, which was committed while he was a journalist.
Lady Warsi does not have a vote in the first stage of the leadership contest, but said she would like Rory Stewart to make the final two.
Mr Gove said "drugs wreck lives", and his cocaine use was a "mistake which I profoundly regret".
He insisted that if he became prime minister, he would be allowed to travel to the United States, despite the country's strict visa rules about people who admit to taking drugs.
His campaign team were forced to take legal advice from a QC who said Mr Gove had "completed his forms correctly" for previous trips to America, which include one to interview Donald Trump in 2017, shortly before his inauguration as president.
The Cabinet minister was grilled about his cocaine use in a bruising appearance on the BBC's Andrew Marr Show.
Asked if he should have gone to prison, Mr Gove said: "I was fortunate in that I didn't, but I do think it was a profound mistake and I have seen the damage drugs do."
The former justice secretary said: "I have seen it close up and I have also seen it in the work that I have done as a politician.
"That is why I deeply regret the mistake that I made."
Metropolitan Police chief Cressida Dick has told middle-class drug users they have "blood on their hands" for financing a trade that results in gangland violence and the deaths of youngsters on the streets.
Mr Gove said: "I hugely respect Cressida Dick. One of the things that I would absolutely say is it is a mistake which I profoundly regret. Absolutely.
"One of the things that I also completely agree with is that the drugs trade is wrong, that drugs wreck lives and that's one of the reasons why I have sought, in office, to try to help people to move away from that."
Mr Gove faced claims of hypocrisy over a column he wrote for The Times hitting out at "London's liberal consensus" around decriminalisation of drugs.
According to the Mail on Sunday, Mr Gove hosted a party the night before the column was published in which the Class A drug was allegedly taken by guests.
Asked by Marr whether he was a hypocrite, Mr Gove said: "No. I think anyone can read the article and make their own minds up. The point that I made in the article is that if any of us lapse sometimes from standards that we uphold, that is human.
"The thing to do is not necessarily then to say that the standards should be lowered. It should be to reflect on the lapse and to seek to do better in the future."
Mr Gove said "no-one asked" about drug use when he became a government minister.
He said "I don't believe that I have ever on any occasion failed to tell the truth about this when asked directly".
Home Secretary Sajid Javid, a leadership rival to Mr Gove, told Sky News "It's not for me to pass judgment on fellow candidates", but he added: "People do need to know – it doesn't matter if you are middle class or not – anyone who takes class A drugs, they need to think about that supply chain that comes from Colombia, let's say, to Chelsea and the number of lives that are destroyed along the way."