A "change of culture" is needed so walking and cycling become the norm instead of using cars, health experts have said.
A Faculty of Public Health (FPH) report said councils should take the lead in redesigning urban areas to encourage residents to reduce the amount they use their vehicles.
FPH president Professor John Middleton said: "Although we no longer have the pea-soupers that killed 12,000 people in the 1950s, 40,000 deaths each year in the UK are attributable to exposure to outdoor air pollution."
"It is also evident that it is disproportionately the poorest of our communities which are most exposed and vulnerable to air pollution," he continued. "Local authorities are responsible for improving the health of their communities.
"One way they do so is to ensure that town and city centres are designed to reduce the health harms of cars to their residents. Everyone in public health, local authorities and across the health and social care sector needs to work together to reduce the health harms of driving.
For the sake of our health now and generations to come, we need a change in culture so that walking or cycling becomes part of our daily routine, rather than spending hours sitting in cars."
Professor Jonathan Grigg, of the Royal College of Paediatrics and Child Health, said: "Measures that reduce the exposure of the population, especially of vulnerable groups such as children, are of utmost priority given the widespread damaging effects of air pollution."