Implement cache in Znoyder

[sdatko]

It turned out that the upstream jobs specification does not change that often like we initially assumed. However, with passing time and project growing, we begin to be overwhelmed by number of calls to GitHub and OpenDev for fetching the Zuul sources. This pull request introduces cache mechanism that would allow us to rely on a single local file providing us all the information about upstream jobs. Such file can be generated/updated periodically. It will be added to repository with a separate pull request.

Example of generated cache:

---
discover_upstream_jobs('files-upstream/stable/2023.1/gnocchixyz/python-gnocchiclient'): []
discover_upstream_jobs('files-upstream/stable/2023.1/openstack-dev/hacking'): []
discover_upstream_jobs('files-upstream/stable/2023.1/openstack-infra/requestsexceptions'): []
discover_upstream_jobs('files-upstream/stable/2023.1/openstack-infra/shade'): []
discover_upstream_jobs('files-upstream/stable/2023.1/openstack-k8s-operators/edpm-image-builder'): []
discover_upstream_jobs('files-upstream/stable/2023.1/openstack-k8s-operators/tcib'): []
discover_upstream_jobs('files-upstream/stable/2023.1/openstack/ansible-role-atos-hsm'):
  - !ZuulJob
    name: openstack-tox-linters
    parameters: {}
    pipeline: check
  - !ZuulJob
    name: openstack-tox-linters
    parameters: {}
    pipeline: gate
...

List of changes:

– Move NestedDumper to dedicated submodule

There will be additional Dumper introduced later,
so lets keep them in the same place.

– Move utils to lib.utils submodule

There are two utils submodules right now,
so lets merge them together.

– Add persistent cache

This commit introduces additional class that implements
the persistent cache. It will be useful for reducing
the actual queries to GitHub and OpenDev, as it turned out
that there not many/often changes to list of upstream jobs.

– Make ZuulJob dumpable with PyYAML

It is all about inheriting from PyYAML base class
and setting custom tag name and declaring this object
as recognized by SafeLoader.

– Use cache in generator

Later we can preserve the generated cache
and stop querying GitHub hundreds of times.

– Make cache decorator callable in two ways

It is possible now to specify it both with
and without parentheses, e.g. @cache and @cache().
This opens a way to extend the functionality
further with additional options passed.

– Add selective cache option

This commit customizes the cache decorator so it is possible
to specify what parameters will be used as cache keys ~ i.e.
for function that we actually care about selected parameters that
makes the actual difference we can consider only specified ones
(by default all function parameters are considered).
Additionally it is possible to produce a human-readable string
instead of a calculated hash – this would make the generated
cache file browseable for us in case we need that...

– Make cache for generator predictable

This commit adds sorting of ZuulJobs entries so when cached
the produced output is always predictable and repeatable
across multiple calls to generator, no matter who, where
and when executes the code. This should allow us to include
small incremental changes in cache file when regenerating
the cache periodically for any upstream changes.

– Use relative paths in generator

The existing code relied on absolute paths to files,
which would result in generated cache to be specific
for a given environment where the Znoyder is called.
Using relative paths would make the calls universal
and properly cacheable.

[adrianfusco]

I just can say this is wonderful +1