Broken global state between Hydra runs

[rsokl]

I don't know what is going on here. It only occurs for pre-release, and re-running the job often fixes it. It occurs in tests/test_launch/test_validation.py and is caused by:

 hydra.errors.MissingConfigException: In 'hydra/config': Could not find 'hydra/sweeper/basic'

@jgbos any ideas?

[jgbos]

That is weird. I'll try to look at it this afternoon.

[rsokl]

@jgbos I think there is a real issue here, although I'm unsure what the culprit is. I'd like to get a fix in for launch before v0.12 is released. Any chance you'd be able to take a look this week?

[rsokl]

I am actually less sure that this is a "real" issue in that I do not think it is affecting every day use of launch. This is a very weird issue that might have something to do with hydra and pytest? Not sure. Anyway, I am going to release v0.12 regardless as this issue certainly is no different in v0.11 than it is in v0.12.

[rsokl]

This was really hard to repro -- still not sure the circumstances that cause this vs not -- but there is some broken statefulness in Hydra between runs which causes Hydra's store to misbehave with the basic sweeper. Fortunately there is a simple fix - running the following clears ~all global state from Hydra!

from hydra.core.singleton import Singleton

# this should go in a `finally:` after each launch call
Singleton._instances.clear()

Because this clears all global state, Hydra will populate everything anew on subsequent runs.

We should clear state this way within launch and zen.hydra_main. This would have to wait for v0.13 as it is potentially compat-breaking (but only in the most weird ways?)