A survey of 2,200 adults found that most think older people have something positive to offer the economy.
The Prince's Initiative for Mature Enterprise, a charity aimed at helping the over-50s consider self-employment, said its research suggests there is a "strong perception" of prejudice in the workplace.
Spokesman Teresa Folkes said: "Conversations around unemployment tend to focus on the high number of young people without a job. While this is a terrible situation, we've got to remember that older people are suffering in the job market too.
"People over the age of 50 have a wealth of skills and experience to make a positive contribution to their local communities, as well as the wider economy, and we would like to see more efforts to support them get back to work."
A Department for Work and Pensions spokesman said: "We agree that the over 50s often have a wealth of experience to offer the workplace which is why we are working with organisations like the age action alliance to promote the benefits of employing older workers.