This test still says something about how well the model adapts to subverted expectations. Some models get more stuck in local regions of their latent spaces than others.

But I agree witih you an important piece of the generalisation puzzle is missing.