SSLEOFError when connecting web3isgner

[hentai8]

I followed the walkthrough exactly, without customizing anything, but I got an SSLEOFError when I tried web3signer_status as follows

2024-12-05T06:33:17.808Z
LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html
2024-12-05T06:33:17.808Z
[ERROR] Exception: exception happened: HTTPSConnectionPool(host='signer.devnitrovalidator.private', port=443): Max retries exceeded with url: /upcheck (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1147)'))) Traceback (most recent call last): File "/var/task/lambda_function.py", line 146, in lambda_handler raise Exception("exception happened: {}".format(e))

I also tried running tests/e2e/e2e_setup.sh at the same time, but got the same error.
I ran cdk destroy before each deploy to make sure that no resource errors from the previous deploy will affect this one.

I really want to use this technique so I've tried it over and over again but it's always this SSLEOFError, please let me know if you need more info, thanks a lot!

[hentai8]

The log from running the e2e test is as follows

+ ./scripts/generate_key_policy.sh nitro_validator_output.json
+ aws kms put-key-policy --policy-name default --key-id arn:aws:kms:us-east-1:727392871343:key/57c7ec84-39ab-480c-ac77-7adc638830dd --policy file://key_policy.json
+ aws lambda invoke --function-name arn:aws:lambda:us-east-1:727392871343:function:devNitroValidator-NitroInvokeLambda398BB8E0-U7iDjgMpV3tz --cli-binary-format raw-in-base64-out --payload '{"operation": "set_tls_key"}' lambda-output
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
+ ./scripts/start_signing_service.sh nitro_validator_output.json
i-0ad8f7a5a340c6a0b:
● nitro-signing-server.service - Nitro Enclaves Signing Server Loaded: loaded (/etc/systemd/system/nitro-signing-server.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2024-12-16 08:31:23 UTC; 15s ago Main PID: 4057 (python3) Tasks: 5 Memory: 27.7M CGroup: /system.slice/nitro-signing-server.service ├─4057 python3 /home/ec2-user/app/watchdog.py └─4092 /bin/nitro-cli run-enclave --cpu-count 2 --memory 3806 --eif-path /home/ec2-user/app/server/signing_server.eif --enclave-cid 16 --debug-mode Dec 16 08:31:23 ip-10-0-99-156.ec2.internal systemd[1]: Started Nitro Enclaves Signing Server. Dec 16 08:31:23 ip-10-0-99-156.ec2.internal watchdog.py[4057]: Start allocating memory... Dec 16 08:31:25 ip-10-0-99-156.ec2.internal watchdog.py[4057]: Started enclave with enclave-cid: 16, memory: 3806 MiB, cpu-ids: [1, 3]
i-0fb1175f37089eccd:
● nitro-signing-server.service - Nitro Enclaves Signing Server Loaded: loaded (/etc/systemd/system/nitro-signing-server.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2024-12-16 08:31:23 UTC; 15s ago Main PID: 4011 (python3) Tasks: 5 Memory: 27.9M CGroup: /system.slice/nitro-signing-server.service ├─4011 python3 /home/ec2-user/app/watchdog.py └─4043 /bin/nitro-cli run-enclave --cpu-count 2 --memory 3806 --eif-path /home/ec2-user/app/server/signing_server.eif --enclave-cid 16 --debug-mode Dec 16 08:31:23 ip-10-0-183-85.ec2.internal systemd[1]: Started Nitro Enclaves Signing Server. Dec 16 08:31:23 ip-10-0-183-85.ec2.internal watchdog.py[4011]: Start allocating memory... Dec 16 08:31:25 ip-10-0-183-85.ec2.internal watchdog.py[4011]: Started enclave with enclave-cid: 16, memory: 3806 MiB, cpu-ids: [1, 3]
{
    "Version": 2,
    "Tier": "Standard"
}

(16/12/2024 08:31:41) service has been started and is healthy
+ ./tests/e2e/web3signer_status.sh nitro_validator_output.json

16/12/2024 08:31:41: sending request
{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST"
}
result: {"errorMessage": "exception happened: HTTPSConnectionPool(host='signer.devnitrovalidator.private', port=443): Max retries exceeded with url: /upcheck (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1147)')))", "errorType": "Exception", "requestId": "8e74ab26-3d20-420d-b857-ded30a7a99fa", "stackTrace": ["  File \"/var/task/lambda_function.py\", line 146, in lambda_handler\n    raise Exception(\"exception happened: {}\".format(e))\n"]}

16/12/2024 08:31:57: sending request
{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST"
}
result: {"errorMessage": "exception happened: HTTPSConnectionPool(host='signer.devnitrovalidator.private', port=443): Max retries exceeded with url: /api/v1/eth2/publicKeys (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1147)')))", "errorType": "Exception", "requestId": "873b8d38-4f0f-44c5-aa3f-fa1e45dd1caf", "stackTrace": ["  File \"/var/task/lambda_function.py\", line 162, in lambda_handler\n    raise Exception(\"exception happened: {}\".format(e))\n"]}

16/12/2024 08:32:14: sending request
{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST"
}

[hentai8]

[peterpan0708]

Hi @dpdornseifer Sir, can you take a look at this issue. We have been stuck here for a long time and still have no idea, much appreciate!

[dpdornseifer]

Hey @hentai8 @peterpan0708,
the solution was still based on IMDSv1 and was missing a few general updates such as the changed docker security model.

Created a new PR including different changes. Can you please try this branch and let me know if you are still facing the issue #12 : https://github.com/aws-solutions-library-samples/guidance-for-secure-blockchain-validation-using-aws-nitro-enclaves/tree/feature/IMDSv2_enablement

PR will be merged to #main soonish.

Cheers

[hentai8]

@dpdornseifer Thank you so much, sir. I will try it later.

[peterpan0708]

Hey @hentai8 @peterpan0708, the solution was still based on IMDSv1 and was missing a few general updates such as the changed docker security model.

Created a new PR including different changes. Can you please try this branch and let me know if you are still facing the issue #12 : https://github.com/aws-solutions-library-samples/guidance-for-secure-blockchain-validation-using-aws-nitro-enclaves/tree/feature/IMDSv2_enablement

PR will be merged to #main soonish.

Cheers

Hi sir, when I'm using python3.9 i got this error:

after I upgrade my python version to 3.12, i get this error:

./scripts/start_signing_service.sh output.json
i-020b2e3625ec8d694:
----------ERROR------- Unit nitro-signing-server.service could not be found. failed to run commands: exit status 4
i-04b887bd468479ac6:
----------ERROR------- Unit nitro-signing-server.service could not be found. failed to run commands: exit status 4
{
"Version": 2,
"Tier": "Standard"
}

[dpdornseifer]

@peterpan0708 did you delete the previous stack and do the entire setup/walkthrough (https://github.com/aws-solutions-library-samples/guidance-for-secure-blockchain-validation-using-aws-nitro-enclaves/blob/main/docs/walkthrough.md) again?

[peterpan0708]

@peterpan0708 did you delete the previous stack and do the entire setup/walkthrough (https://github.com/aws-solutions-library-samples/guidance-for-secure-blockchain-validation-using-aws-nitro-enclaves/blob/main/docs/walkthrough.md) again?

yes, i run cdk destroy prodNitroValidator command

[peterpan0708]

@peterpan0708 did you delete the previous stack and do the entire setup/walkthrough (https://github.com/aws-solutions-library-samples/guidance-for-secure-blockchain-validation-using-aws-nitro-enclaves/blob/main/docs/walkthrough.md) again?

On the new branch , after the deployment, I login in to the prodNitroValidator, there is also no /etc/systemc/system/nitro-signing-server.service.

It just seems failed to bootstrap the workflow.
On the main branch, it works

[dpdornseifer]

Can you please remove the ECR image and run tests/e2e_setup.sh. What platform are you deploying from arm or x86?

[peterpan0708]

Can you please remove the ECR image and run tests/e2e_setup.sh. What platform are you deploying from arm or x86?

I'm using x86(t3.small)

I run the e2e_setup.sh, and got this error:

It says The maximum number of VPCs has been reached.
how many vpcs do i need?

[peterpan0708]

Can you please remove the ECR image and run tests/e2e_setup.sh. What platform are you deploying from arm or x86?

I'm using x86(t3.small)

I run the e2e_setup.sh, and got this error:

It says The maximum number of VPCs has been reached. how many vpcs do i need?

My mistake, the region in the e2e_setup.sh is us-east-1 by default, we already have 5 vpcs in that area.

I run e2e_setup.sh in ap-east-1 and I still got this :

[hentai8]

The new branch you provided is working fine and I can get the pubkey from web3signer in enclaves, thanks a lot for your help!

Hey @hentai8 @peterpan0708, the solution was still based on IMDSv1 and was missing a few general updates such as the changed docker security model.

Created a new PR including different changes. Can you please try this branch and let me know if you are still facing the issue #12 : https://github.com/aws-solutions-library-samples/guidance-for-secure-blockchain-validation-using-aws-nitro-enclaves/tree/feature/IMDSv2_enablement

PR will be merged to #main soonish.

Cheers

[dpdornseifer]

@peterpan0708 @hentai8 realized that there were still some potential issues with multi region environments and also solved a compatibility issue in the e2e_test script causing an error running on Linux. Pushed a minor update to the branch.

[peterpan0708]

@peterpan0708 @hentai8 realized that there were still some potential issues with multi region environments and also solved a compatibility issue in the e2e_test script causing an error running on Linux. Pushed a minor update to the branch.

@dpdornseifer sir, still can't run correctly in ap-east-1.

error is still:
starting signing service
i-035f795d75d994b4b:
----------ERROR------- Unit nitro-signing-server.service could not be found. failed to run commands: exit status 4
i-0f3e55ea6dabe4caa:
----------ERROR------- Unit nitro-signing-server.service could not be found. failed to run commands: exit status 4
{
"Version": 2,
"Tier": "Standard"
}

(23/12/2024 05:57:02) service has been started and is healthy