Flyte domains cache separation - alternative "stacked" mode?

[fg91]

Flyte has a concept called domain. From the docs:

Domains provide an abstraction to isolate resources and feature configuration for different deployment environments.

The domain is part of the cache key (source).

---

In our ML experimentation platform, we associate feature branches with the development domain, the main branch with the staging domain, and tags/releases with the production domain.

While we wouldn't want to "pollute" e.g. the staging/main branch cache with feature branch results, the opposite - reusing a result computed on master - on a feature branch would at least for us be desirable. This is especially the case when certain experiment results are expensive to compute.

This would mean:

• Development: first check development cache, then staging, then production
• Staging: Check staging cache, then production
• Production: Only check production cache

While this is not something where I'd ever propose to change the current behaviour by default, I'd like to start a discussion whether such an opt-in feature would be feasible and useful for others in the community.