add carp funnel with stake delegation pool tracking #246

ecioppettini · 2023-10-24T18:21:50Z

TODO

remove tmp_carp_client and use the version from npm once published
make the presync work
parametrize the known shelley time or something (potentially the carp service could provide this instead)

Example settings

Currently I'm setting things up like this:

extensions.yml

extensions:
   - name: "Cardano Stake Delegation"
     type: cardano-stake-delegation 
     pools: ["367d927a91eb9ecf15648779beeec8c710171566edcce18feb048265"]
     startSlot: 32032590  
     scheduledPrefix: cd

.env (example)

CARP_URL=http://localhost:3000
CARDANO_NETWORK=preview
CARDANO_CONFIRMATION_DEPTH=10 
BATCHER_CARDANO_ENABLED_POOLS="095dd39da2d8534f9ddb93759a1931288e5dd79ae04fa5914e157bd6,093de39da2d8534f9ddb93759a1931288e5dd79a404fa5914e157b99"
BATCHER_CARP_URL=http://host.docker.internal:3000

Ideally the carp setting should be a single one, but I needed this to make it work with the batcher running in docker.

Parsing

I had to setup a parsing rule like this:

cardanoDelegation  = cd|address|pool

(the lenghts are not tight bounds, just as an example for testing)

const cardanoDelegation: ParserRecord<CardanoStakeDelegation> = {
  address: PaimaParser.NCharsParser(0, 100),
  pool: PaimaParser.OptionalParser(null, PaimaParser.NCharsParser(0, 100)),
};

packages/engine/paima-funnel/src/funnels/carp/funnel.ts

SebastienGllmt · 2023-11-07T04:45:24Z

packages/engine/paima-funnel/src/cde/reading.ts

  );
  return allData;
 }

 async function getSpecificCdeData(
-  extension: ChainDataExtension,
+  extension: ChainDataExtension & { startBlockHeight: number },


What was the need to add this? It's not used anywhere in the function

I don't understand this question, it's used in the first line of the function actually

packages/engine/paima-funnel/src/funnels/BaseFunnel.ts

packages/engine/paima-funnel/src/funnels/block/funnel.ts

SebastienGllmt · 2023-11-16T00:07:44Z

packages/engine/paima-funnel/src/funnels/carp/funnel.ts

+import { query, getErrorResponse } from '@dcspark/carp-client/client/src/index';
+import { Routes } from '@dcspark/carp-client/shared/routes';
+
+const confirmationDepth = '10';


Should have a comment rationale for why 10 was picked. If this isn't an ENV, it should be mentioned in the docs

I made it an env... Since I think it should be configurable, although it's a bit problematic since it shouldn't really be changed without resyncing.

packages/engine/paima-funnel/src/funnels/carp/funnel.ts

packages/engine/paima-sm/src/index.ts

packages/paima-sdk/paima-db/migrations/up.sql

packages/node-sdk/paima-db/migrations/down.sql

just to test until the up to date version is published on npm

SebastienGllmt · 2023-12-03T04:05:38Z

packages/engine/paima-rest/src/tsoa/routes.ts

@@ -1,16 +1,7 @@
 /* tslint:disable */


This file is automatically generated and shouldn't be checked in. The reason this file changed is because sometimes it gets modified by automatic code formatting and I can't figure out why sometimes the rule to ignore this file for formatting works and sometimes it doesn't 😕

SebastienGllmt · 2023-12-03T04:07:03Z

packages/engine/paima-runtime/src/runtime-loops.ts

+  if (!ENV.CARP_URL) {
+    delete presyncBlockHeight[Network.CARDANO];
  }


Could this not cause a subtle issue where if you specify a Cardano CDE but the CARP_URL is missing that it just silently doesn't track anything? I feel if you're using task that requires Carp and you didn't specify the CARP_URL it should be an error

yeah, makes sense, changed it for an error

packages/engine/paima-runtime/src/runtime-loops.ts

just to reduce the number of rpc calls

the cast is probably clearer than the alternatives

gostkin · 2023-12-05T11:05:09Z

packages/engine/paima-funnel/src/funnels/carp/funnel.ts

+  );
+
+  let grouped = groupCdeData(
+    Network.EVM,


why not cardano?

well... It doesn't actually matter, I think.

The discriminant was introduced so that during the presync we can update the cardano cde tracking table instead of the other one, but during the normal sync just knowing the EVM block gives you the slot, so the field is not used.

But I guess we should put cardano here anyway to be consistent.

gostkin · 2023-12-05T12:54:05Z

how do I get pool id to put in the config in BATCHER_CARDANO_ENABLED_POOLS? what is the format? e.g. I have pool id 3867a09729a1f954762eea035a82e2d9d3a14f1fa791a022ef0da242, how to put it there? 8200581ceeef9389ba6f3971425a59af99c56d1c9c227b3a2c2c8f52a10dd8ag looks like a different format and i can't query it in the explorer

ecioppettini · 2023-12-05T13:58:29Z

how do I get pool id to put in the config in BATCHER_CARDANO_ENABLED_POOLS? what is the format? e.g. I have pool id 3867a09729a1f954762eea035a82e2d9d3a14f1fa791a022ef0da242, how to put it there? 8200581ceeef9389ba6f3971425a59af99c56d1c9c227b3a2c2c8f52a10dd8ag looks like a different format and i can't query it in the explorer

The 8200581ceeef9389ba6f3971425a59af99c56d1c9c227b3a2c2c8f52a10dd8ag format was a mistake actually, it should be a string of 56 chars. I corrected it in the docs, but I forgot to do it here. It should be what shows as pool id in cardanoscan.

@gostkin

SebastienGllmt · 2023-12-05T14:13:33Z

packages/engine/paima-runtime/src/cde-config/utils.ts

-  const startBlockheights = config.map(cde => cde.startBlockHeight).filter(sbh => !!sbh);
+  const startBlockheights = config
+    .map(cde => {
+      if (cde.cdeType != ChainDataExtensionType.CardanoPool) {
+        return cde.startBlockHeight;
+      } else {
+        return null;
+      }
+    })
+    .filter(sbh => !!sbh) as number[];
  const minStartBlockheight = Math.min(...startBlockheights);
  return isFinite(minStartBlockheight) ? minStartBlockheight : -1;


All of this should probably be a single .reduce call instead (same with the function below)

packages/engine/paima-runtime/src/cde-config/utils.ts

ecioppettini self-assigned this Oct 24, 2023

ecioppettini force-pushed the carp-funnel-stake-delegation-cde branch from 081aeba to a1831ef Compare October 31, 2023 00:32

ecioppettini marked this pull request as ready for review November 3, 2023 00:53

ecioppettini force-pushed the carp-funnel-stake-delegation-cde branch from 9b76490 to 2a9f066 Compare November 7, 2023 02:30

gostkin reviewed Nov 14, 2023

View reviewed changes

packages/engine/paima-funnel/src/funnels/carp/funnel.ts Outdated Show resolved Hide resolved

gostkin reviewed Nov 14, 2023

View reviewed changes

packages/engine/paima-funnel/src/funnels/carp/funnel.ts Show resolved Hide resolved

SebastienGllmt reviewed Nov 16, 2023

View reviewed changes

ecioppettini force-pushed the carp-funnel-stake-delegation-cde branch from 079776d to 5ead855 Compare November 22, 2023 03:50

gostkin reviewed Nov 27, 2023

View reviewed changes

packages/node-sdk/paima-db/migrations/down.sql Show resolved Hide resolved

ecioppettini added 11 commits November 29, 2023 14:53

add carp funnel with stake delegation pool tracking

e62b7ab

handle presync in carp funnel

ba77b28

expose cde utils get cardano delegation function

e12f963

use local (tmp) carp client lib in the funnel

40d5e8e

just to test until the up to date version is published on npm

add delegation check in batcher's address validator

4265a95

add stopSlot setting to stop fetching

f1cbd29

improve logging and make presync concurrent

3413015

address review comments

d7f4540

fix markCardanoCdeSlotProcessed query

77f7ac2

presync checkpointing for cardano cde

444771d

drop cde_tracking_cardano in down.sql

879218d

ecioppettini force-pushed the carp-funnel-stake-delegation-cde branch from 6080ca7 to 879218d Compare November 30, 2023 02:39

SebastienGllmt reviewed Dec 3, 2023

View reviewed changes

packages/engine/paima-runtime/src/runtime-loops.ts Outdated Show resolved Hide resolved

ecioppettini added 4 commits December 4, 2023 14:20

address new review comments

709ead0

replace carp local client with 2.3.0 from npm

06a4493

cache the startingSlot and the previous range upper end

72d896d

just to reduce the number of rpc calls

remove old TODO

6f70baf

the cast is probably clearer than the alternatives

gostkin reviewed Dec 5, 2023

View reviewed changes

SebastienGllmt reviewed Dec 5, 2023

View reviewed changes

ecioppettini added 2 commits December 5, 2023 11:31

replace map-filter with reduce for getEarlistSlot/Blockheight

17b265f

add CARDANO_CONFIRMATION_DEPTH config

a2c3b33

SebastienGllmt reviewed Dec 5, 2023

View reviewed changes

packages/engine/paima-runtime/src/cde-config/utils.ts Outdated Show resolved Hide resolved

ecioppettini added 4 commits December 5, 2023 13:59

refactor getEarliestStart* functions

d63f1f1

default case of min should be Infinity

91c3247

document timestampToAbsoluteSlot and fix missing slot offset

e334a4d

reinitialize the cache entry in case of error

b6de71c

SebastienGllmt approved these changes Dec 12, 2023

View reviewed changes

SebastienGllmt merged commit a578dca into master Dec 12, 2023

SebastienGllmt deleted the carp-funnel-stake-delegation-cde branch December 12, 2023 21:55

ecioppettini mentioned this pull request Dec 14, 2023

cardano-presync: avoid error when there was no previous sync started #271

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add carp funnel with stake delegation pool tracking #246

add carp funnel with stake delegation pool tracking #246

ecioppettini commented Oct 24, 2023 •

edited

Loading

SebastienGllmt Nov 7, 2023

ecioppettini Dec 7, 2023

SebastienGllmt Nov 16, 2023 •

edited

Loading

ecioppettini Dec 5, 2023

SebastienGllmt Dec 3, 2023

ecioppettini Dec 4, 2023

SebastienGllmt Dec 3, 2023

ecioppettini Dec 5, 2023

gostkin Dec 5, 2023

ecioppettini Dec 5, 2023

gostkin commented Dec 5, 2023

ecioppettini commented Dec 5, 2023 •

edited

Loading

SebastienGllmt Dec 5, 2023

add carp funnel with stake delegation pool tracking #246

add carp funnel with stake delegation pool tracking #246

Conversation

ecioppettini commented Oct 24, 2023 • edited Loading

TODO

Example settings

extensions.yml

.env (example)

Parsing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SebastienGllmt Nov 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gostkin commented Dec 5, 2023

ecioppettini commented Dec 5, 2023 • edited Loading

Choose a reason for hiding this comment

ecioppettini commented Oct 24, 2023 •

edited

Loading

SebastienGllmt Nov 16, 2023 •

edited

Loading

ecioppettini commented Dec 5, 2023 •

edited

Loading