> Another dumb question: can the model be smarter than the data?

There's a mathematical sense in which it cannot be: the data-processing inequality (https://en.wikipedia.org/wiki/Data_processing_inequality). The extent to which that sense is totally counterintuitive to how things normally work in ML nowadays has led to investigating different foundations for information theory: https://arxiv.org/abs/2601.03220.