A survey of IoD members found that just 27% feel HS2 represents good value for money, and 70% say the scheme will have no impact on the productivity of their business.
The survey also showed that there was little enthusiasm for the project even in the regions where the benefits are supposed to be strongest.
In August 2011 a survey of IoD members found 54% rated HS2 important to their business. This figure has now fallen to 41%, with the IoD saying that this illustrated "how businesses see high-speed rail as a lower priority than it was two years ago".
The IoD scepticism over the scheme, which will see a first, London to Birmingham, phase completed around 2026, follows a report last week by the Institute for Economic Affairs (IEA) which said the cost of HS2 could be as high as £80 billion.
There were also reports that the Treasury was working on a figure as high as £73 billion for the project which cuts through Tory heartlands in the Chilterns, with a Y-shaped scheme due to take the line to north-east and north-west England around 2032/33 .
The IoD said that a central part of the Government's current economic case for HS2 was that time spent on a train is unproductive. But the IoD said its research showed that this assumption was "wildly inaccurate", as only 6% of directors say they never work on a train.
Also, 48% of members say they spend at least half of the journey working, 26% work for between a quarter and half the time, and 21% spend up to a quarter of the journey time working productively.
Commenting on the research, IoD director general Simon Walker said: "Businesses up and down the country know value for money when they see it, and our research shows that they don't see it in the Government's case for HS2.
"Some of the specific claims that the Government has used to support its economic case for the project have been challenged by our members, who by and large do not feel that their business will benefit."
He went on: "We recognise that some of our members are in favour of this project, and there is a plurality of opinion among the businesses community. But overall there appears to be little enthusiasm among IoD members, not even in the regions where the benefits are supposed to be strongest. Indeed, our research shows that almost every region expects London to benefit the most.
"The IoD cannot support the Government's current economic case for HS2 when so many of our members are doubtful of the benefits.
"We agree with the need for key infrastructure spending, but the business case for HS2 simply is not there. The money would be far better spent elsewhere and in a way that will benefit much more of the country.
"Investment in the West and East Coast main lines combined with a variety of other infrastructure projects would be a far more sensible option."
Mr Walker continued: "Our members support increased investment in other aspects of our road and rail network, citing this as more important than investment in HS2.
"As many as 80% feel that investment in existing intercity rail services should be a priority, with just 41% saying the same for investment in HS2. A total of 63% believe the money should be spent on other transport projects.
"Station upgrades, inter-city improvements, tunnels, electrification and capacity improvements should all be considered alternatives. It is time for the Government to look at a thousand smaller projects instead of falling for one grand folly."
The opinion of IoD members is broadly similar across the country, and in no region of the UK do more than 35% think HS2 represents good value for money.
The £50 billion figure for HS2 includes £7.5 billion for the cost of the trains, with the line costing £42.6 billion.
HS2 Ltd chief executive Alison Munro said: " While we respect the right of the IOD to state its case, we believe that HS2 will provide value for money and will bring about a transformational change to the economic geography of our country through creating thousands of jobs and opportunities for regeneration in and around our core cities.
"While smaller schemes may have higher benefit cost ratios, by their very nature they only make small improvements to capacity and often just move the bottleneck elsewhere on the network."
She went on: "Passenger numbers are continuing to grow on our main north/south rail arteries and the time when we reach capacity crunch is fast approaching. Network Rail have made it clear that they cannot deliver the required capacity by piecemeal upgrades to the existing network. There are after all only so many trains you can fit into a section of track and the East Coast and West Coast main lines are virtually full already.
"The IoD have not yet released their polling data but it is clear that their members are strongly in favour of increasing capacity with 80% supporting investment in long-distance trains and 41% supporting HS2.
"There is no other alternative that delivers the benefits of HS2. Through building a world-class 21st century high-speed rail network that will link our great cities, both north and south, as never before, we are focused on delivering within our budget of £42.6 billion to provide the capacity we need on those routes and free up space to expand commuter services and freight."
A Department for Transport spokeswoman said: "We need to build HS2 to free up valuable space for passengers and freight because, without it, our existing rail network will be full by the mid-2020s at a cost to passengers and businesses up and down country.
"The scheme is forecast to generate over £50 billion of benefits for the UK economy but we know we must maximise every economic benefit HS2 has to offer.
"That is why the HS2 Growth Taskforce was created, headed up by Lord Deighton, to work with city and business leaders to ensure we are capitalising on every opportunity to help regeneration, job creation, investment opportunities and in building a skilled UK economy."
The IoD surveyed 1,323 members online between August 1 and 11.