They can't climb all surfaces. They especially can't climb on dirty and dusty surfaces or crumbling paint.

But more than that, they fell into the hole without leaving a pheromone trail to follow back to the nest. Without that, they have no way of navigating. Only random exploration could get them back out, and that would only help one ant at a time.