Hacker News

points by karmickoala 4 days ago | 0 comments

I know, I mentioned it because there's a category of models that I'd be okay with, but I can't exactly name it. I thought the difference lied in the encoder/decoder distinction. As a sibling comment mentioned, generative models do offer more privacy risks.

But it's not only that. Model size, for instance, also plays a role. Will modern BERT keep their lossy compression nature? Or could there be a privacy risk as well?

Thus, what are the characteristics of privacy-aware models? So far: discriminative vs. generative, model size, and lossy transforms (related to size). Perhaps looking at pre-2020 models could help to narrow other necessary attributes.

[–] whimsicalism 4 days ago

Model size, definitely. Not so sure about the discriminative vs. generative distinction (technically 'generative' models are discriminators as well) - I suspect the data is extractable if you have access to the weights, although easier to extract from inferencing in a decoder model.

My guess is that the pairing of all of these things is due to the fact that Google has likely consolidated everything under the same style of decoder model.