A survey of almost 1,500 consumers found that four out of five would be concerned if big branches were moved.
The research, by watchdog Consumer Futures, follows controversial plans to franchise 70 so-called Crown post offices, the larger sites usually found on high streets.
The Communication Workers Union has held a series of strikes against the move, warning that jobs and services will be hit.
More than nine out of 10 of those who took part in the latest poll said a town centre location was important for them.
Almost half of pensioners said it would be difficult for them to visit an out-of-town post office, with one in five believing they would have to rely on a friend or relative to give them a lift.
Consumer Futures urged the Post Office to retain main branches in town centres rather than move them to sites such as hypermarkets.
Spokesman Andy Burrows said: "Consumers value and trust the Post Office and large post office branches moving to out-of-town locations would mean considerable inconvenience and disruption.
"Post Office branches must provide high quality services from locations that people can get to easily and we are calling for a 'town centre first' approach with Post Office Ltd seeking town centre premises before it considers an out-of-town-centre location, for example an out-of-town hypermarket.
"The forthcoming changes to branches over the next few years have the potential to bring improvements, but it is key that the Post Office gets off on the right foot and commits to flagship branches remaining in town and city centres."
Martin Blackwell, chief executive of the Association of Town & City Management, commented: " It is essential to the aims of building an inclusive society that post offices remain accessible to everyone via a range of different modes of transport, and not just to those with cars.
"Furthermore, retaining post offices on the high street strengthens the chances for the success of other businesses through increased visits. This insight from Consumer Futures reminds us of the importance of ensuring our communities are sustainable."
A Post Office spokesman said: "We are absolutely committed to keeping Post Office services in the communities they currently serve. Indeed we are investing around £70 million in improving services in our high street and city centre branches over the course of the next 18 months, alongside the overall transformation of our network."
"Where we are currently in the process of franchising 70 of our high street branches, we are looking to partner with quality retailers that will maintain and modernise services to communities in locations in very close proximity to their current location. Any changes will go through full public consultation."
A Business Department spokesman said: "As Consumer Futures' report makes clear, branch locations and their proximity to town centres are subject to full public consultation. It is in everyone's interest that branches are located where customers can conveniently access them and these public consultations help to ensure this.
"In addition, Post Office is required to meet strict Government set access criteria that sees over 99% of the urban population within one mile of a branch. We are also making a significant investment to maintain a network of at least 11,500 branches and to modernise thousands of branches right across the country."