Allow specifying top-level file and format for KVStore

[GeeWee]

It's very practical to be able to keep all your local configuration in separate files, if they are e.g. generated by automatic scripts. This extends the fastly.toml kv_store configuration to accept either the current inline API or a new file and format table, that allows loading of an external JSON file.

Part of #364

[kpfleming]

Thank you for the contribution! We're actually in the midst of some discussions about how to redesign the way that Viceroy handles 'resources' needed by the service, so we'll have to hold off on reviewing this PR for a few weeks until we've reached some decisions there. The good news is that the idea you've presented here is directly in line with our goals :-)

Please stay tuned, we'll get back to you soon.

[GeeWee]

Gotcha - thanks for the feedback :)

[kpfleming]

Overall this looks fine, but as you can probably see from the conflicts the term 'object store' shouldn't be used for this feature, it should be 'kv store' (in various forms) everywhere. If you get the PR updated I'll give it a formal review and work towards getting it merged. Thanks!

[GeeWee]

@kpfleming the test file is reasonably inconsistent (many tests still refer to object store). However I've fixed my changes on top of the latest main and changed to use kv-store instead of object store in my tests.

I've rebased and squashed my changes into one commit. Let me know if you don't care about that (e.g. if you'll end up squashing when merging anyways) and I'll save the effort :)

[JakeChampion]

Another approach we can take is to point a KV Store to a folder and use the file-paths as the KV Store Keys and the file contents as the value -- This would allow for simpler setup for the scenario where applications are wanting to store their website static assets in a KV Store

[JakeChampion]

Another approach we can take is to point a KV Store to a folder and use the file-paths as the KV Store Keys and the file contents as the value -- This would allow for simpler setup for the scenario where applications are wanting to store their website static assets in a KV Store

I have a branch which implements this at https://github.com/fastly/Viceroy/tree/jake/kv-store-folder but I stalled out on the work unfortunately

[GeeWee]

Definitely also an option - I do wonder if keeping parity with the dictionary interface isn't a good thing though, considering
that we can't be the only migrating I imagine :)

[kpfleming]

A few changes you can commit directly, and some others you can make in a followup commit. You can squash or not squash, whichever is easier for you.

[kpfleming]

Another approach we can take is to point a KV Store to a folder and use the file-paths as the KV Store Keys and the file contents as the value -- This would allow for simpler setup for the scenario where applications are wanting to store their website static assets in a KV Store

I have a branch which implements this at https://github.com/fastly/Viceroy/tree/jake/kv-store-folder but I stalled out on the work unfortunately

We've got another PoC implementation around which does the same thing as a WASM component :-)

For now I'm OK with merging the approach in this PR because it doesn't add a new 'style' of data population, it just replicates the existing style offered for config stores. We can talk about offering other styles in the larger topic of redesigning the package manifest/configuration file.

[GeeWee]

@kpfleming I believe I've incorporated all of your feedback :)

[kpfleming]

The tests all pass now, I'll add an entry for CHANGELOG.md and get this merged.