Casual workers are five times more likely to drop out of the labour market than those on permanent contracts, a new study finds.
Research for the TUC also showed that casual work was "rarely" a stepping stone to finding a better job.
Workers without regular hours were more likely to be unsatisfied with their jobs and suffer greater levels of anxiety.
Work by the Universities of Sheffield and Greenwich for the union organisation revealed that young, black or ethnic minority people were more likely to be in casual work.
The TUC said the findings backed up separate polling which showed the main reason people were on zero hours contracts was because that was the only type of work available.
The study of almost 20,000 workers by the two universities showed that people in casual employment for at least a year were at greater risk of becoming jobless than permanent employees.
TUC general secretary Frances O'Grady said: "People are often told that casual contracts are a stepping stone to finding better work.
"But for many this isn't the case. Instead of improving their career prospects, lots of casual workers find themselves dropping out of work altogether, or they're at the mercy of bad bosses who treat them like disposable labour.
"Every job should be a great job , but far too many workers in the UK feel insecure at work."
Dr Kirsty Newsome from the University of Sheffield said the survey revealed a "pernicious set of interlocking pressures at work and home", adding: "Many feel worthless, vulnerable and afraid in the face of unbridled employer power.
"Low and unpredictable pay causes them hardship. In too many instances the pressures at work are mirrored at home by rising personal and household debt."
A retail worker interviewed for the study said: "All our hours change from day to day or week to week. A chap on my department came into work at six in the morning thinking that he was due in, but they changed the rota without telling him and said, no, you've got to go home".