I think the "work" he talks about was unpaid and enforced. Stuff armed people forced you to do, that took up time you needed for your subsistence farming.

I don't think that's true. There are some subsistence societies, blessed with abundant resources and neighbors with no interest in taking them, where people worked less. In many societies people worked more. (And of course, in all premodern societies, it was common to have famines where many of your neighbors would starve to death because they couldn't get enough food no matter how much they worked.)

The type of work people used to do was a lot more varied too. Even specialists would have to do more work than just their speciality

As people have specialized more and more we spend more time on a much more narrowly defined set of tasks

And we spend a lot more time per week doing just those tasks