The author is wading into an age-old debate, between instrumentalism and (scientific) realism in science. Briefly, instrumentalists are happy to have accurate models, realists also want to have explanatory theories.

I am far from well-informed on the intricacies of this debate myself, however I class myself as a realist. Not only do I have to know how and why things work, I also observe that an explanatory theory is at least as useful as an accurate model (of the same phenomenon). Assuming equal accuracy, the explanatory theory is just as good at predicting future observations as the model, and on top of that it is capable of explaining past observations. More, we seem to always be able to improve our predictive accuracy, compared to non-explanatory models, once we have an explanatory theory and can understand why models work and why they fail. Understanding ultimately improves prediction but prediction alone does not necessarily improve understanding.

I think this is the crux of the debate with AI mathematics: AI proofs are nice to have but if we don't understand them, what are we going to do with them to further advance our knowledge?

I see this also in the way neural nets work in general. We can train very powerful neural nets, but can we reuse one trained neural net to train another? Not unless we use the first net to label some data. I mean to say, we can't just use a trained neural net as a feature of another neural net. Which is why most of the time we have to learn everything from scratch, end-to-end, and consequently why neural nets need so much damn training data.

Finally, say that AI takes over mathematics research. What happens a few years down the line when all we have is yes/no answers? How are we going to make new mathematics from that? How are we going to keep training AI mathematicians? Let alone human ones?