Supreme Court President Lord Neuberger said in a lecture that cutting legal aid will deprive the people who most need the protection of the courts of the ability to get legal representation.
Confronted with the judge's comments during a Justice Committee hearing, Mr Grayling said he would be surprised if Lord Neuberger thought the cutbacks were "a good thing".
After reading the judge's comments, Plaid Cymru MP Elfyn Llwyd said: "Have you no regard for experts in the field?"
Mr Grayling replied: "I would be surprised if the president of the Supreme Court was saying legal aid cutbacks are a good thing.
"It would be a surprise if the Lord Chancellor was saying legal aid cutbacks are a good thing - I'm not.
"What I'm saying is this unfortunately, given the nature of the public spending challenge we face and the fact that we have relatively the most expensive system of its kind in the world, means that it is inevitable that we will have to bring the costs down.
Mr Grayling said in the past year the bill for legal aid has been £2 billion.
Mr Llwyd accused the Justice Secretary of attempting to mislead the Committee as the bill has actually fallen.
Mr Grayling said: "That has fallen by 10% so far since 2010.
"Over the period between 2010 and 2015/16, we will have reduced the spend on legal aid roughly in line with the budget of the department.
"We're not misleading anyone, we're telling you the truth.
"I'm telling you we face a huge financial challenge in this country."
Delivering the annual Tom Sargent Memorial Lecture, Lord Neuberger said: "If a person with a potential claim cannot get legal aid, there are two possible consequences.
"The first is that the claim is dropped: that is a rank denial of justice and a blot on the rule of law.
"The second is that the claim is pursued, in which case it will be pursued inefficiently, and will take up much more of the court staff's time and of the judge's time in and out of court.
"So that it means greater costs for the court system, and delay for other litigants."
Mr Llwyd said he had never seen a judge be so publicly opposed to a Government reform.
Elsewhere in his lecture, Lord Neuberger also warned the Government to take "particular care" when considering changes to a legal process which allows people to challenge decisions by public bodies.
The Supreme Court President said ministers should be "very careful" about cutting down people's right to seek judicial review.
He said courts had "no more important function" than protecting citizens from the abuses and excesses of public authorities.
Lord Neuberger said Government proposals aimed to reduce the cost and delay involved in judicial review applications.
"The desire to discourage weak applications is understandable, even laudable, and the desire to reduce delay and expense is plainly right, at least in principle," he said.
"However, one must be very careful about any proposals whose aim is to cut down the right to judicial review.
"The courts have no more important function than that of protecting citizens from the abuses and excesses of the executive - central government, local government, or other public bodies.
"With the ever-increasing power of Government, which now commands almost half the country's gross domestic product, this function of calling the executive to account could not be more important.
"I am not suggesting that we have a dysfunctional or ill-intentioned executive, but the more power that a government has, the more likely it is that there will be abuses and excesses which result in injustice to citizens, and the more important it is for the rule of law that such abuses and excesses can be brought before an impartial and experienced judge who can deal with them openly, dispassionately and fairly.
"While the Government is entitled to look at the way that judicial review is operating and to propose improvements, we must look at any proposed changes with particular care, because of the importance of maintaining judicial review, and also bearing in mind that the proposed changes come from the very body which is at the receiving end of judicial review."
Shadow justice secretary Sadiq Khan said: "Lord Neuberger is right to sound the alarm about the Government's plans for judicial review. Citizens of this country holding to account politicians, governments and public agencies is part and parcel of a healthy democracy.
"Judicial review is central to this, and time and again it exposes failings and acts as a check and balance on the excesses of governments. Making it harder to use judicial review will see governments insulated from challenge, and individual citizens increasingly powerless.
"This is yet another example of David Cameron's out-of-touch Government siding with the strong, and, by watering down the rights of citizens, it is failing to stand up for the weak."