... I feel like if an authoritarian government could extract the family classifier info from local, they could just extract the damn messages.

And if you think that handful of megabytes is too much, they could extract a million other signals that tell them this, such as who you have pinned in the relevant apps.