Some 770,000 people settled in the UK do not speak English, Sajid Javid said as he defended the Government's proposals to encourage social integration.
The plans, backed by £50 million of public money, include measures to boost English language skills and proposals to require schools whose pupils come from a single ethnic or religious community to ensure they mix with children from other backgrounds.
Communities Secretary Mr Javid said there was a "segregation problem" in some of England's schools and action was needed to tackle it.
The proposed Integrated Communities Strategy calls on schools to teach "British values" and sets out plans to boost English language skills and encourage women from minority communities to find jobs.
A consultation paper on the plans follows the 2016 Casey Review, which warned that social cohesion could not be taken for granted in the multicultural UK.
Mr Javid told BBC Radio 4's Today there were about 770,000 people settled in Britain who "speak no or very poor English".
"Just imagine the opportunities they have given up on, the inability they have to socially mix with others and really contribute to society.
" It's not fair on them and it's not fair on the rest of society," he said.
Mr Javid, whose family came to the UK from Pakistan, said his mother took about 10 years to learn English.
"I remember as a six or seven-year-old going to the doctor's surgery with her so I could interpret for her," he said, but learning the language "absolutely transformed her life".
Pressed on whether the money promised to support the strategy was enough, Mr Javid said: "It's not just about the £50 million, there's actually a substantial amount the Government already spends on helping people learn English."
But "there hasn't been enough of a joined-up approach" between Whitehall and local government, he acknowledged.
Highlighting the "segregation problem" in schools, Mr Javid said: "We believe about 60% of ethnic minority pupils ... they go to schools where ethnic minority pupils are in the majority.
"If you just think about that and the amount of segregation that has been caused by schools, something new has to be done."
Five pilot areas - Blackburn, Bradford, Peterborough, Walsall and the London borough of Waltham Forest - will develop local integration plans allowing new strategies to be tested as the programme develops.
The author of the 2016 report, Dame Louise Casey, welcomed measures to promote English tuition, telling Today: "It is really important that as a nation we are all able to speak a common language and that common language is English."
Lady Casey said the strategy also included "very important, very powerful" proposals to tighten registration of home-schooling to protect children against exposure to extremist ideology.
But she warned that funding of £50 million over two years would not be enough to deliver the "seismic" shift needed.
Mr Javid's Ministry of Housing, Communities and Local Government is seeking responses to proposed initiatives set out in a new green paper.
Among the proposals are:
- A new community-based English language programme, with a network of conversation clubs and support for councils to improve provision of tuition.
- Personalised skills training to help women from "isolated" communities into work.
- Measures to ensure young people have the opportunity to mix and form lasting relationships with those from different backgrounds.
- Promotion of British values across the school curriculum.
- Increased take-up of the National Citizen Service.
Education Secretary Damian Hinds said: "We want to make sure that all children learn the values that underpin our society - including fairness, tolerance and respect.
"These are values that help knit our communities together, which is why education is at the heart of this strategy."
Think tank British Future released polling data suggesting a majority of voters would back schools teaching pluralistic British values (76%), more support to learn English (67%) and a zero-tolerance approach to hate crime and prejudice (79%).
Some 63% said the Government should use national events like St George's Day, St David's Day and St Andrew's Day to bring people together.
British Future director Sunder Katwala said: "Integration isn't just about British Muslims - it's an issue for all of us.
"So it's welcome that this green paper moves on from the Casey Review and broadens the integration debate.
"It could be an important step towards the national integration strategy that we've been missing - provided it's followed up by action."
- ICM questioned 3,657 GB adults online between June 9 and 14 2017 for British Future.