Persistant storage

[cynicaljoy]

Rather than cache/artifact assets needing to be constantly uploaded/downloaded, wouldn't it be better to have a persistent storage volume that's owned by the repo owner that gets mounted every execution?

Whether the workflow stores data to the volume or not could be completely up to the workflow. The usage of this volume would be accounted for as part of the "Storage for Actions and Packages" within our account billing.

This should result in less complexity in the Workflow definition file as well as speed up Workflow execution times.

[iskunk]

I am trying to build a Workflow that, unfortunately, is not well served by the cache construct. There are at least two significant problems with the way caches are implemented:

• The storage and retrieval medium: a literal tarball that is persisted in blob storage. This has to be downloaded, unpacked, then (re)packed, and (re)uploaded---a noticeably slow operation. Inefficient, too: If my cache has 5 GB of data, and a particular Workflow run only needs a 1 kB file from it? Still gotta move and unpack 5 GB of data. Compare that to e.g. NFS, where just the 1 kB goes over the wire.

• Cache content is immutable once uploaded. So if your cache behaves like a typical cache, where new entries are added and old entries expired as time goes on, you have to do this bizarre dance with restore-keys: to create a whole new updated copy of the cache from the old one at each iteration. Housekeeping of obsolete cache revisions is left as an exercise for the reader.

GitHub caches are a reasonable solution if (1) your Workflow regularly needs most/all of the cached content, and (2) the thing you are caching remains stable over time. But that's a fairly narrow use case. And if you're doing something not ideally suited to that, it gets ugly pretty quickly---it's not a particularly general or flexible sort of mechanism that lends itself to a wide variety of use cases.

A persistent storage volume, as @cynicaljoy described, is exactly what I want. Conceptually, it serves the same need as volumes in a Docker container. Not only would that let me implement caching and persistent state in a manner that suits my application, it would be flexible enough to support the immutable-caching scenario (just offer an API call to switch read-only status) for the occasions when that makes sense.

(Frankly, I don't understand why this approach wasn't implemented to begin with. The current way of doing things is so much more complex and hard to understand than "just" mounting a volume that there has to be some non-obvious reasoning behind it.)