feat(cat-gateway): Sync indexing and recovery (#917)

* fix(docs): Fix up docs issues

* fix(backend): Huge refactor to prep for scylladb config management

* fix(backend): Clean up logging a little, and add build info logs as required for production.

* Refactor and setup cassandra config/session

* feat(backend): Index DB schema setup seems to work

* WIP

* fix(rust): Format fixes

* fix(rust): Build fixes

* fix(rust): Adjust index DB so we can index without querying, and can optimize on first detected spend.

* fix(rust): add more docs

* fix(rust): basic new follower integration

* fix(rust): wip

* fix(ci): Bump rust compiler version to match CI

* ci(backend): Bump rust version to match CI

* fix(backend): Fix code format and lints

* feat(backend): simple new block indexer just to test the logic works

* feat(gateway): Simple indexing with cassandra seems to work

* refactor(backend): Remove lazy and once_cell in favor of new standard library replacements

* fix(backend): WIP indexing for stake addresses and unstaked ada

* fix(backend): indexing WIP

* fix(backend): Add support for log control with env vars, default to mainnet, adjust `justfile` to properly select preprod and also refresh git dependencies.

* feat(backend): Make local test scylla db run with 4 nodes, not 1

* fix(backend-lib): Add stop for cassandra db cluster

* refactor(backend-lib): Remove c509-certificate because its moved to catalyst-libs

* fix(backend): Remove dependencies from Workspace, and move into project

* fix(backend): Use temporary cat-ci branch for rust builders

* fix(backend): Remove obsolete common crates subdirectory

* fix(backend): Don't use pre-packaged mithril snapshots in integration tests

* fix(backend): Fix code so it builds with latest chain follower code.

Also eliminates redundant logic now incorporated into chain follower.

* fix(backend): Fix broken reference to catalyst libs

* ci(ci): Bump all earthfiles to latest WIP cat-ci branch

* fix(frontend-pkg): Ignore .dart_tool directory in frontend files checking markdown

* fix(ci): Fix spelling

* fix(spelling): Add more project words and properly sort list

* fix(backend): Sync rust configs and add target to make it easier in future

* fix(backend): Enable all features of Scylla for now.

* fix(frontend-pkg): Fix markdown table having too many columns

* ci(spelling): Fix spelling issues

* fix(docs): Bump docs to latest WIP cat-ci version

* feat(gateway): Add low resource scylla db instance for local testing

* feat(gateway): Add and update developer convenience functions for backend

* fix(backend): Fix code format

* fix(backend): Fix spelling issues in CQL files

* fix(spelling): Remove duplicates from the project words dictionary

* fix(backend): Get the backend building properly with earthly.

* feat(backend): remove obsoleted postgres logic for chain indexing

* revert(event-db): Revert extension changes to sql files after fixing sqlfluff version

* fix(frontend): Regenerate the dart api interface file, and add doing that to the pre-push just command

* fix(backend): temporarily disable API tests

* fix(backend): Also temporarily stop workflow consuming test reports that are disabled

* fix(ci): Try and stop coveralls running for api-tests

* ci(general): Replace temp CI branch with tagged release

* feat: Add Handler for Permissionless Auth (#825)

* docs(cips): Add Formal Defintion of auth token

* fix(docs): Fix comments in cddl file

* fix(docs): sig size

* fix(docs): Rename CDDL for the auth token

* docs(docs): Add auth-header documentation

* docs(docs): Fix markdown line length error

* docs(general): Fix spelling

* fix(backend-lib): Bump to catalyst-libs tagged version

* fix(backend): stub out obsolete code (to be removed in follow up PR).

* fix(backend-lib): code format

* fix(backend): remove unused crate dependencies

* feat: auth token (#723)

* feat(auth token encode and decode): permissionless auth

* feat(auth token encode and decode): permissionless auth

* feat(auth token encode and decode): permissionless auth

* feat(auth token encode and decode): permissionless auth

* feat(auth token encode and decode): permissionless auth

* iron out tests

* iron out tests

* refactor(auth token encode and decode): ed25519 Signature cbor fields

Sig over the preceding two fields - sig(cbor(kid), cbor(ulid))

* refactor(auth token encode and decode): ed25519 Signature cbor fields

Sig over the preceding two fields - sig(cbor(kid), cbor(ulid))

* feat(cat security scheme): open api

* feat(cat security scheme): open api

* feat(mock cert state): given kid from bearer return pub key

* feat(auth token): cache TTL

* feat(auth token): cache TTL

* feat(auth token): cache TT

* ci(spell check): fix

* ci(spell check): fix

* ci(spell check): fix

* refactor(clippy): housekeeping tidy

* refactor(clippy): housekeeping tidy

* refactor(clippy): housekeeping tidy

* refactor(clippy): housekeeping tidy

* fix(backend): Re-enable dependent crates used by this code

* fix(backend): clippy lints

* fix(backend): spelling

---------

Co-authored-by: Steven Johnson <[email protected]>
Co-authored-by: Steven Johnson <[email protected]>

* feat: Update GET staked_ada endpoint to fetch from ScyllaDB (#728)

* feat: get staked ada from scylladb

* chore: revert justfile changes

* chore: filter TXOs in rust instead of filtering in ScyllaDB query

* fix(backend): spelling

* fix(backend): Eliminate lint errors from Derived function

* fix(backend): code format

* fix(backend): Udate autogenerated dart code

* chore(cat-voices): fix tests

---------

Co-authored-by: Steven Johnson <[email protected]>
Co-authored-by: Steven Johnson <[email protected]>
Co-authored-by: Dominik Toton <[email protected]>

* feat: DB Indexing for  CIP-36 registrations (#788)

* feat: add schema for cip-36 registration tables

* feat: index cip-36 by stake address

* feat: index cip-36 registrations by vote key

* fix: use TxiInserParams::new when adding txi data

* fix: remove unused cfg attributes

* fix: refactor Cip36RegistrationInsertQuery::new

* fix(backend): Refactor queries and add multiple tables for cip36 registration indexes

* fix(backend): Cip36 Primary key is stake key. Stake Key N->1 Vote Key

* fix(backend): code format

---------

Co-authored-by: Steven Johnson <[email protected]>
Co-authored-by: Steven Johnson <[email protected]>

* docs(general): Cleanup project dictionary

* docs(spelling): Fix spelling

* fix(backend): remove obsolete clippy lint cfg

* docs(backend): Improve field documentation so its not ambiguous.

* docs(backend): Fix comment

* docs(backend): Improve comment

* fix(backend): Vote Key index logic, and update comments

* fix(backend): Earthfile needs to be executed from root of repo, to properly pick up secrets

* fix(backend): make generic saturating value converter and use it instead of type specific ones

* test(cat-gateway): Add tests for float conversion and better docs about functions limitations.

* fix(cat-gateway): Developer lints in release mode, and also refer to correct local release binary

* fix(cat-gateway): CIP36 index schema error

* fix(cat-gateway): Cip36 indexing working, improve bad cassandra query reporting.

* refactor(cat-gateway): Make settings a sub-module

* refactor(cat-gateway): Break up Envvar handling into multiple files

* refactor(cat-gateway): Fix code format

* feat(cat-gateway): Add chain-sync downloader options to cat-gateway env vars

* test(cat-gateway): Make debug logs visible in local testing

* fix(cat-gateway): Minimum timeout for mithril downloads

* test(cat-gateway): Silence gratuitous debug log in cassandra queries

* refactor(cat-gateway): Decrease verboseness of the string env var logging code

* fix(general): Bump alpine version and pin to fix missing upstream containers

* ci(general): Bump cat-ci to v3.2.07

* ci(general): fix docker in docker container image

* feat(cat-gateway): Autogenerate the cassandra schema version and add schema sync progress table

* fix(cat-gateway): Bump rust library dependency versions

* fix(cat-gateway): Make cassandra namespace compliant.

* refactor(cat-gateway): rename utilities to utils

* refactor(cat-gateway): use ensure! instead of if/bail

* ci(general): Bump cat-ci version to latest

* fix(flutter): Again, try and see if caching is working ok for flutter builds with cat-ci

* ci(general): Bump CI to latest version to try and fix docs build issues

* docs(cat-gateway): Better document the generate_cql_schema_version function

* fix(cat-gateway): Use a string array reference rather than a vec of strings for strings to uuid generation

* refactor(cat-gateway): fix code format

* fix(cat-gateway): Bump ci version and update rust earthfile to check for cache breaks

* test(cat-gateway): Add parameters better suited to syncing mainnet

* feat(cat-gateway): Add indexing insert queries for blockchain sync state

* fix(cat-gateway): Bump to cat-ci v3.2.13

* feat(cat-gateway): Sync status indexing and reading now working.

* feat(cat-gateway): Add recoverable sync logic

* fix(cat-gateway): Update CI to fix issue with SyncConfig

* fix(cat-gateway): spelling

* fix(cat-gateway): Adjust the chain indexing parameters and defaults

* fix(cat-gateway): Fix the logic at the end of indexing the immutable chain.

---------

Co-authored-by: cong-or <[email protected]>
Co-authored-by: Felipe Rosa <[email protected]>
Co-authored-by: Dominik Toton <[email protected]>
Co-authored-by: Joaquín Rosales <[email protected]>

.config/dictionaries/project.dic

@@ -239,6 +239,7 @@ Subkey
 submiting
 subosito
 svgs
+syncable
 SYSROOT
 tablestats
 tacho

Earthfile

@@ -1,8 +1,8 @@
 VERSION 0.8
 
-IMPORT github.com/input-output-hk/catalyst-ci/earthly/mdlint:v3.2.10 AS mdlint-ci
-IMPORT github.com/input-output-hk/catalyst-ci/earthly/cspell:v3.2.10 AS cspell-ci
-IMPORT github.com/input-output-hk/catalyst-ci/earthly/postgresql:v3.2.10 AS postgresql-ci
+IMPORT github.com/input-output-hk/catalyst-ci/earthly/mdlint:v3.2.14 AS mdlint-ci
+IMPORT github.com/input-output-hk/catalyst-ci/earthly/cspell:v3.2.14 AS cspell-ci
+IMPORT github.com/input-output-hk/catalyst-ci/earthly/postgresql:v3.2.14 AS postgresql-ci
 
 FROM debian:stable-slim
 

catalyst-gateway/Earthfile

@@ -1,19 +1,42 @@
 VERSION 0.8
 
-IMPORT github.com/input-output-hk/catalyst-ci/earthly/rust:v3.2.10 AS rust-ci
+IMPORT github.com/input-output-hk/catalyst-ci/earthly/rust:v3.2.14 AS rust-ci
 
 #cspell: words rustfmt toolsets USERARCH stdcfgs
 
+# COPY_SRC:
+# Copies the source code into the builder.
+COPY_SRC:
+ FUNCTION
+
+ COPY --keep-ts --dir \
+ .cargo .config Cargo.toml clippy.toml deny.toml rustfmt.toml \
+ bin \
+ .
+
+# builder : Set up our target toolchains, and copy our files.
+builder:
+ DO rust-ci+SETUP
+
 # sync-cfg: Synchronize local config with CI version.
 # Must be run by the developer manually.
 sync-cfg:
+ FROM +builder
  DO rust-ci+SYNC_STD_CFG
 
-# builder : Set up our target toolchains, and copy our files.
-builder:
- DO rust-ci+SETUP
+builder-src:
+ FROM +builder
+
+ # Don't build the src in the home directory itself, because it contains stuff.
+ WORKDIR $HOME/build
+ RUN rm -rf *
 
- COPY --dir .cargo .config Cargo.* clippy.toml deny.toml rustfmt.toml bin .
+ # Cached copy of the source we build.
+ DO +COPY_SRC
+
+ # Generate Checksums of the source
+ DO rust-ci+FINGERPRINT_SRC
+ SAVE ARTIFACT ../src_fingerprint.txt 
 
 ## -----------------------------------------------------------------------------
 ##
@@ -24,7 +47,7 @@ builder:
 # check : Run check using the most efficient host tooling
 # CI Automated Entry point.
 check:
- FROM +builder
+ FROM +builder-src
 
  DO rust-ci+EXECUTE --cmd="/scripts/std_checks.py"
 
@@ -36,12 +59,12 @@ all-hosts-check:
 # build : Build the catalyst-gateway service
 build:
  # Build the service
- FROM +builder
+ FROM +builder-src
 
  DO rust-ci+EXECUTE \
  --cmd="/scripts/std_build.py" \
  --args1="--bins=cat-gateway/cat-gateway" \
- --args2="--cov_report=$HOME/coverage-report.info" \
+ --args2="--cov_report=$HOME/build/coverage-report.info" \
  --output="release/cat-gateway" \
  --junit="cat-gateway.junit-report.xml" \
  --coverage="cat-gateway.coverage.info" \
@@ -81,4 +104,26 @@ package-cat-gateway:
 get-api-locally:
  FROM scratch
  COPY +build/doc/cat-gateway-api.json cat-gateway-api.json
- SAVE ARTIFACT cat-gateway-api.json AS LOCAL cat-gateway-api.json
+ SAVE ARTIFACT cat-gateway-api.json AS LOCAL cat-gateway-api.json
+
+# build-src-check: Check for any caching issues with the source we are building against.
+check-builder-src-cache:
+ FROM +builder
+
+ # Don't build the src in the home directory itself, because it contains stuff.
+ WORKDIR $HOME/build
+ RUN rm -rf * 
+
+ COPY +builder-src/src_fingerprint.txt ..
+
+ RUN --no-cache echo "Cache Disabled"
+
+ # Uncached copy of the source we build.
+ DO +COPY_SRC
+
+ # Generate Checksums of the source
+ DO rust-ci+FINGERPRINT_SRC --FINGERPRINT=src_fingerprint_uncached.txt
+
+ RUN diff ../src_fingerprint.txt ../src_fingerprint_uncached.txt \
+ || (echo "ERROR: Source fingerprints do not match. Caching Error Detected!!" && exit 1) \
+ && echo "Source fingerprints match. Caching OK."

catalyst-gateway/Justfile

@@ -50,6 +50,8 @@ run-cat-gateway: build-cat-gateway
 
 # Run cat-gateway natively on mainnet
 run-cat-gateway-mainnet: build-cat-gateway
- CHAIN_FOLLOWER_SYNC_TASKS="1" \
+ CHAIN_FOLLOWER_SYNC_TASKS="32" \
+ CHAIN_FOLLOWER_DL_CONNECTIONS="64" \
+ CHAIN_FOLLOWER_DL_CHUNK_SIZE="4" \
  RUST_LOG="error,cat_gateway=debug,cardano_chain_follower=debug,mithril-client=debug" \
  ./target/release/cat-gateway run --log-level debug