The fallacy comes into play when you use those characteristics to prejudge individual humans.

Tall women are taller than short men, so if you're hiring a librarian who needs to be able to reach books on a high shelf, you still need to look at the individual. Society as a whole suffers if you run employment ads saying "Women need not apply," so we rightfully deter that.