The Prince's Trust also found around one in five have "abandoned their ambitions" due to a lack of success at school, while those with fewer than five GCSEs are almost twice as likely as their peers to believe that they will "never amount to anything".
The survey of people aged 16-25 has prompted the trust to call for more vocational support for those leaving school with few qualifications, to help them into jobs.
Last year, two-fifths of young people across England did not achieve five GCSEs graded A*-C, including English and maths.
Martina Milburn, chief executive of the youth charity, said: "Thousands of young people's ambitions are crushed by exam results each year.
"Many of these young people have faced problems at home or bullying at school, so their exam results don't reflect their true potential.
"It is now more important than ever to invest in vocational support and training for young people who are not academically successful.
The Prince's Trust says three-quarters of young people who enroll on its intensive training schemes for struggling school leavers move into work, education or training.
More than 2,000 people were questioned as part of the online survey, which also showedone in 10 young people believe their exam results will "always" hold them back. This increases to a quarter (26%) among those who left school with poor grades.
Last month David Cameron hailed a "rigorous, engaging and tough" new national curriculum, which he said was critical to Britain's future economic success.
The Prime Minister described the changes, due to be introduced in September 2014, as a ''revolution in education''.
Shadow education secretary Stephen Twigg said: "David Cameron doesn't have a plan for all young people.
"With A-level results out this week, we know many young people have high ambitions. But sadly, this Government is holding them back by cutting careers advice, threatening school standards, and leaving nearly a million young people out of work.
"While David Cameron has done nothing for the 50% of young people who don't go to university, Labour would give them a gold standard vocational option through our planned Tech Bacc.
"Labour would ensure that all young people get high quality qualifications and there is a jobs guarantee for young people out of work for a year. The Prince's Trust are right to highlight these issues."
Lucie Russell, director of campaigns and policy at the charity YoungMinds, which works to improve the emotional wellbeing and mental health of children and young people, said: "These figures highlight just some of the everyday pressures young people face. Young people are growing up in a toxic climate where bleak employment prospects, exam pressures and a 24/7 online culture where they can never switch off are placing a significant strain on their emotional wellbeing.
"Children and young people feeling such despair about their future is not good for their mental health and could lead to problems later down the line. These are unprecedented times for children and young people and we must support them practically as The Prince's Trust recommends but also give much better support to improve their emotional wellbeing."
A Department for Education spokesman said: "We are taking decisive action to transform vocational education by funding a place in education or training for every 16 and 17-year-old who wants one, raising the participation age to 18, and investing in quality traineeships and apprenticeships.
"We are also overhauling vocational qualifications to recognise only high-quality courses that lead to a skilled trade or profession.
"Studio Schools and University Technical Colleges already allow employers to help shape the curriculum, while the new Tech Levels and our Technical Baccalaureate will finally put high-quality vocational education on a par with A- levels."