refactor: simple substate fetching optimizations (#777)

Description
---
* Refactored the `transaction_autofiller` in the `indexer_lib` to spawn
`tokio` tasks in parallel for each substate request to the network.
* Created a new `SubstateCache` trait and a `SubstateFileCache`
implementation using `cacache-rs`
* Refactored the existing `SubstateScanner` to read and write to the new
cache accordingly. The decision to access the cache from this level was
both to encapsulate as well as maximize the usage of the cache across
all applications.
* Both the indexer and the validator node application need to build a
`SubstateFileCache` on startup (it will use a new `./substate_cache`
subfolder in the data root folder)
* The indexer needs it for transaction autofilling when submitting
transactions, as well as for monitoring substates.
* The validator node needs it as a dependency for the dry run
transaction processing.

Motivation and Context
---
Currently we are experiencing poor performance in the Tari testnet. A
big part of the problem is due to a lack of optimization around substate
fetching, particularly to get the latest version of a substate:
* Indexer auto-filler fetches substates sequentially, when we could do
it in parallel
* There is no caching of previous versions of a substate, so each time
we want to know the last version of a substate we start querying again
from substate 0 onwards.

This PR aims to solve this problems by:
* Implementing parallel fetching of substates in the indexer auto-filler
* Implementing a cache system for previous versions of substates, so we
only need to query for possible newer versions. We want the cache system
to be file-based to persist between restarts.

How Has This Been Tested?
---
Running a local network, doing transactions and inspecting the logs for
the caching operations.

What process can a PR reviewer use to test or verify this change?
---
Start up a new local network, do transactions and inspect the new
substate caching logs.

Breaking Changes
---

- [x] None
- [ ] Requires data directory to be deleted
- [ ] Other - Please specify

---------

Co-authored-by: Stan Bondi <[email protected]>

applications/tari_dan_app_utilities/Cargo.toml

@@ -27,9 +27,13 @@ tari_template_builtin = { path = "../../dan_layer/template_builtin" }
 tari_template_lib = { path = "../../dan_layer/template_lib" }
 tari_transaction = { path = "../../dan_layer/transaction" }
 tari_validator_node_client = { path = "../../clients/validator_node_client" }
+tari_bor = { path = "../../dan_layer/tari_bor" }
+tari_indexer_lib = { path = "../../dan_layer/indexer_lib" }
 
 anyhow = "1.0.53"
+async-trait = "0.1.74"
 bytes = "1"
+cacache = "12.0.0"
 chrono = "0.4.22"
 futures = { version = "^0.3.1" }
 log = { version = "0.4.8", features = ["std"] }

applications/tari_dan_app_utilities/src/lib.rs

@@ -22,5 +22,6 @@
 
 pub mod base_layer_scanner;
 pub mod consensus_constants;
+pub mod substate_file_cache;
 pub mod template_manager;
 pub mod transaction_executor;