Scale observation building with massive amount of results

[amirylm]

AUTO-9416

Description

Currently we sort the entire set of staged upkeeps by random seed order based on the OCR round/seq number, then take a limit of items to be added to observation.
As part of the sorting we are doing string shuffling with the random seed, which is CPU+memory intensive work that makes the entire operation slow, and breaks when scaling as we repeat the same expensive operation to get random order in each round. Eventually resulting in multiple “Observation is taking too long” warnings or even crushes the nodes.

We will use a random number every x rounds (e.g. 10) and maintain a cache for the shuffled work ids.
A map will be used to store shuffled ids, it will get cleaned every time the random source changes (10 rounds).

Changes

• Added staging hook state with cache and last src
• Populate cache before sorting
• Use cache when sorting

[infiloop2]

PR looks great to me!

Could we add some unit test coverage for the updated logic