Part of the issue was that Herodotus reported what his sources said. So rather than it being 'Herodotus' as a single reliable or unreliable source, it was more like a collection of unvetted sources.

As a result, you got stories about dog sized ants that were nonsense but then also stories that were legit.

People tend to be rather binary with how they process information, so when there's things that are wrong they group everything in the 'wrong' category.

Definitely a lot of false negatives with how Herodotus was regarded. Much like how the prevailing consensus was that Troy was a fictional city until it was discovered.