They do not respect prior knowledge in any way, and think it has no value.

Of course they don't do standard, table stakes things like making their crawler smart. They don't have to. They're better than that or something.