Okay, it's not quite the same. I could figure out how the thermometer works, but I cannot figure out how the 1 trillion parameter AI works. But the AI didn't give me an answer directly, it gave me a proof that I can run through a verifier. And I could figure out how the verifier works. That is a partial understanding.
The barrier to my understanding is not the verifier. So it's the opaque AI model? But human brains are blackboxes too. I cannot tell what's going on in another person's head when they're writing a proof either. I can only look at their behavior, the language they produce.
An AI provides a proof written in the same symbols humans write in. It's really long and complicated, more than human-written proofs. It would be nice if I had an excuse for my inability to internalize the proof in a timely manner. I could form a community to make the proofs we don't fully understand yet = "not knowledge".
This is delusional self-soothing, and it's just as damaging to my capacity to understand reality as delegating thought to another entity, regardless of whether that entity is a computer or a human.