They will hit a "ceiling" where despite working more, their disposable income after childcare costs will fail to increase or even fall in some cases as their benefits are steeply withdrawn, the Joseph Rowntree Foundation (JRF) found.
The report was commissioned to probe whether Universal Credit, which combines six different benefits and tax credits into one simplified payment, will achieve its goal of making work pay. It suggested that people without children will generally have stronger incentives to work.
Moving into "mini jobs" of up to 10 hours a week would see families better off under the shake-up, but working beyond this threshold results in a slow climb towards a higher disposable income, it found. Families could end up "trapped" on inadequate funds to get by.
The system risks being undermined by high childcare costs combined with low wages and sharp cuts in Universal Credit once families earn above certain thresholds, the report, titled Does Universal Credit Enable Households To Reach A Minimum Income Standard? said.
Universal Credit started its early roll-out in April and the Government said that six new areas - Hammersmith, Rugby, Inverness, Harrogate, Bath and Shotton - will put claimants on to the scheme from October.
The report looked at how families incomes changed by working more hours. It found that a couple with two young children, where one person works full time on the minimum wage, after receiving Universal Credit, would have disposable income of £346 a week. This falls £103 a week short of what they actually need to live on, according to people's perceptions of acceptable living standards.
If the second earner worked three days a week on the minimum wage, they would still be £72 short of the acceptable standard of living. And if the second earner upped their hours to five days a week they would actually be worse off, falling £74 short, the research found.
Julia Unwin, chief executive of the JRF, said: "For families to move from welfare to work, Universal Credit must restore a situation where working full-time is properly rewarded. It is vital that families are able to improve their disposable incomes by working more hours. The risk otherwise is it will trap families on inadequate incomes where they don't have enough to makes end meet, with little prospect of them progressing to a better standard of living."