The jobless rate will rise from 4.8% this year to 5.4% in 2018 and 5.8% in 2019, said the EY Item Club forecasting group.
Employment will fall for the first time since 2009, albeit only by 0.1% next year, as the supply of workers slows, said the report.
Martin Beck, senior economic adviser to the EY Item Club, said: "The UK labour market may be starting to become a victim of its own success.
"As the proportion of people in work has climbed ever higher, firms may have found it more difficult to fill vacancies, resulting in greater utilisation of the existing workforce and slower jobs growth.
"On a positive note, slower growth in the workforce may deliver a boost to what has been a long period of insipid productivity growth.
"With the flow of potential workers slowing, firms are likely to have more incentive to invest in improving efficiency or labour-saving technology."
Average earnings are expected to increase by 2.75% this year and at similar rates in 2017 and 2018, although prospects for growth in real pay, taking inflation into account, look less bright, it was predicted.
Official figures on unemployment and average earnings will be published by the Office for National Statistics on Wednesday.