That's the historical wisdom, but there are reasons to believe the opposite: that past a certain point more parameters are always better. Basically, that overfitting don't matter, because a large enough model will memorize and build the general solution simultaneously. Then weight decay and SGD noise encourage forgetting of the memorized solution so the general solution takes over.

https://youtu.be/lKoJJxjUfdw is a really excellent lecture that changed how I think about it.