points by whimsicalism 4 days ago | 0 comments
BERT is an LLM. I don't see why Google needs to cater to increasingly fine parsing of politically acceptable vs. not ML workflows, because that is really what this is about.
[–] deepsun 4 days ago
[–] whimsicalism 4 days ago
thank you for the reply, it is a good grounding distinction and i will think more on this perspective.
[–] karmickoala 4 days ago
I know, I mentioned it because there's a category of models that I'd be okay with, but I can't exactly name it. I thought the difference lied in the encoder/decoder distinction. As a sibling comment mentioned, generative models do offer more privacy risks.
But it's not only that. Model size, for instance, also plays a role. Will modern BERT keep their lossy compression nature? Or could there be a privacy risk as well?
Thus, what are the characteristics of privacy-aware models? So far: discriminative vs. generative, model size, and lossy transforms (related to size). Perhaps looking at pre-2020 models could help to narrow other necessary attributes.
[–] whimsicalism 4 days ago
Model size, definitely. Not so sure about the discriminative vs. generative distinction (technically 'generative' models are discriminators as well) - I suspect the data is extractable if you have access to the weights, although easier to extract from inferencing in a decoder model.
My guess is that the pairing of all of these things is due to the fact that Google has likely consolidated everything under the same style of decoder model.
And frankly I see the distinction in only number of model parameters vs number of people in cohort it was trained. So 50k-param model trained on 1b emails from 1 million people is ok, encoding/compression loses anything personal. But huge LLMs are closer to lossless compressors that can just spew all my passwords, prescriptions etc verbatim.