First node for CCF cluster failed to come up in CACI

[gaurav137]

Hit this issue on CACI where the start node itself has not come up. Looking at the container logs I see:
[fail ] node/quote_endorsements_client.h:136 | Giving up retrying fetching attestation endorsements from 10.92.0.12 after 3 attempts.

The node/state endpoint reports state as Initialized and hasn't transitioned to PartOfNetwork.

Filing the bug to track this issue.

2024-09-04T16:32:05.763122Z        0   [info ] node/quote_endorsements_client.h:104 | Fetching endorsements for attestation report at http://10.92.0.12:2377/metadata/THIM/amd/cert
ification?platformId=6c9871d8cd30ddb602591f746b7971db7e4457ee477be925e44c68753178492c48447f8c9cdee4f61fdc5907d4767c7903ad7490aed302a7a0c7584d13c17375&tcbVersion=d315000000000004
 
2024-09-04T16:32:05.770874Z -0.003 0   [info ] node/quote_endorsements_client.h:104 | Fetching endorsements for attestation report at http://10.92.0.12:2377/metadata/THIM/amd/certification?platformId=6c9871d8cd30ddb602591f746b7971db7e4457ee477be925e44c68753178492c48447f8c9cdee4f61fdc5907d4767c7903ad7490aed302a7a0c7584d13c17375&tcbVersion=d315000000000004
 
2024-09-04T16:32:08.778701Z -0.004 0   [info ] node/quote_endorsements_client.h:104 | Fetching endorsements for attestation report at http://10.92.0.12:2377/metadata/THIM/amd/cert
ification?platformId=6c9871d8cd30ddb602591f746b7971db7e4457ee477be925e44c68753178492c48447f8c9cdee4f61fdc5907d4767c7903ad7490aed302a7a0c7584d13c17375&tcbVersion=d315000000000004
 
2024-09-04T16:32:11.778697Z -0.004 0   [fail ] node/quote_endorsements_client.h:136 | Giving up retrying fetching attestation endorsements from 10.92.0.12 after 3 attempts

[gaurav137]

Perhaps logic can retry more and/or give a longer gap between the retries. The above logs show very fast retries. To debug this on doing curl command for the same endpoint in the container it worked, so would be an intermittent issue.

[achamayou]

@gaurav137 I think what we want here mainly is:

1. More retries.
2. Node shut down if all attempts fail.

Does that make sense to you?

[gaurav137]

@achamayou yes sounds good. I'd also suggest to have the retry attempts/timeouts configurable via the cchost config file and not baked into the code. That way tweaking them does not require new ccf builds/images.

[achamayou]

@gaurav137 yes, please see #6478