Cannot start the node after upgrade

[tajelp]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

Hello, after I have upgraded the node to the latest docker image, I get these errors:

us-docker.pkg.dev/oplabs-tools-artifacts/images/op-geth:v1.101308.2

errexit         on
noglob          off
ignoreeof       off
monitor         off
noexec          off
xtrace          off
verbose         off
noclobber       off
allexport       off
notify          off
nounset         off
errtrace        off
vi              off
pipefail        off
Waiting for Bedrock node to initialize...
INFO [05-05|16:38:49.969] Starting geth on an OP network...        network=op-mainnet
INFO [05-05|16:38:49.969] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [05-05|16:38:49.969] Enabling metrics collection
INFO [05-05|16:38:49.970] Enabling metrics export to InfluxDB
INFO [05-05|16:38:49.971] Maximum peer count                       ETH=50 total=50
INFO [05-05|16:38:49.972] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
Fatal: Failed to raise file descriptor allowance: operation not permitted
Fatal: Failed to raise file descriptor allowance: operation not permitted

us-docker.pkg.dev/oplabs-tools-artifacts/images/op-node:v1.7.2

errexit         on
noglob          off
ignoreeof       off
monitor         off
noexec          off
xtrace          off
verbose         off
noclobber       off
allexport       off
notify          off
nounset         off
errtrace        off
vi              off
pipefail        off
Waiting for Bedrock node to initialize...
t=2024-05-05T16:38:46+0000 lvl=warn msg="Unknown env var" prefix=OP_NODE env_var="OP_NODE__RPC_ENDPOINT=http://172.17.0.1:10124"
t=2024-05-05T16:38:46+0000 lvl=warn msg="Unknown env var" prefix=OP_NODE env_var="OP_NODE__L1_BEACON=http://172.17.0.1:10125"
t=2024-05-05T16:38:46+0000 lvl=warn msg="Unknown env var" prefix=OP_NODE env_var="OP_NODE__RPC_TYPE=basic"
t=2024-05-05T16:38:46+0000 lvl=info msg="Rollup Config" l2_chain_id=10 l2_network=OP-Mainnet l1_chain_id=1 l1_network=mainnet l2_start_time=1686068903 l2_block_hash=0xdbf6a80fef073de06add9b0d14026d6e5a86c85f6d102c36d3d8e9cf89c2afd3 l2_block_number=105235063 l1_block_hash=0x438335a20d98863a4c0c97999eb2481921ccd28553eac6f913af7c12aec04108 l1_block_number=17422590 regolith_time="@ genesis" canyon_time="@ 1704992401 ~ Thu Jan 11 17:00:01 UTC 2024" delta_time="@ 1708560000 ~ Thu Feb 22 00:00:00 UTC 2024" ecotone_time="@ 1710374401 ~ Thu Mar 14 00:00:01 UTC 2024" fjord_time="(not configured)" interop_time="(not configured)"
t=2024-05-05T16:38:46+0000 lvl=info msg="Initializing rollup node" version=v0.0.0-99a53381-1710933867
t=2024-05-05T16:38:46+0000 lvl=info msg="Connected to L1 Beacon API, ready for EIP-4844 blobs retrieval." version=Lighthouse/v5.1.3-3058b96/x86_64-linux
t=2024-05-05T16:38:46+0000 lvl=warn msg="failed to dial address, but may connect later" addr=http://op-geth:8551
t=2024-05-05T16:38:47+0000 lvl=warn msg="failed to dial address, but may connect later" addr=http://op-geth:8551
t=2024-05-05T16:38:49+0000 lvl=warn msg="failed to dial address, but may connect later" addr=http://op-geth:8551
t=2024-05-05T16:38:53+0000 lvl=warn msg="failed to dial address, but may connect later" addr=http://op-geth:8551
t=2024-05-05T16:39:01+0000 lvl=warn msg="failed to dial address, but may connect later" addr=http://op-geth:8551
t=2024-05-05T16:39:12+0000 lvl=warn msg="failed to dial address, but may connect later" addr=http://op-geth:8551
t=2024-05-05T16:39:22+0000 lvl=warn msg="failed to dial address, but may connect later" addr=http://op-geth:8551
t=2024-05-05T16:39:32+0000 lvl=warn msg="failed to dial address, but may connect later" addr=http://op-geth:8551
t=2024-05-05T16:39:42+0000 lvl=warn msg="failed to dial address, but may connect later" addr=http://op-geth:8551
t=2024-05-05T16:39:52+0000 lvl=warn msg="failed to dial address, but may connect later" addr=http://op-geth:8551
t=2024-05-05T16:39:52+0000 lvl=eror msg="Error initializing the rollup node" err="failed to init L2: failed to setup L2 execution-engine RPC client: operation failed permanently after 10 attempts: address unavailable (http://op-geth:8551)"
t=2024-05-05T16:39:52+0000 lvl=crit msg="Application failed" message="failed to setup: unable to create the rollup node: failed to init L2: failed to setup L2 execution-engine RPC client: operation failed permanently after 10 attempts: address unavailable (http://op-geth:8551)"

I don't understand where this http://op-geth:8551 comes from.

Here's my .env:

###############################################################################
#                                ↓ REQUIRED ↓                                 #
###############################################################################

# Network to run the node on ("op-mainnet" or "op-sepolia")
NETWORK_NAME=op-mainnet

# Type of node to run ("full" or "archive"), note that "archive" is 10x bigger
NODE_TYPE=full

###############################################################################
#                            ↓ REQUIRED (BEDROCK) ↓                           #
###############################################################################

# L1 node that the op-node (Bedrock) will get chain data from
OP_NODE__RPC_ENDPOINT=http://172.17.0.1:10124

# L1 beacon endpoint, you can setup your own or use Quicknode
OP_NODE__L1_BEACON=http://172.17.0.1:10125

# Type of RPC that op-node is connected to, see README
OP_NODE__RPC_TYPE=basic

# Reference L2 node to run healthcheck against
HEALTHCHECK__REFERENCE_RPC_PROVIDER=https://mainnet.optimism.io

###############################################################################
#                            ↓ OPTIONAL (BEDROCK) ↓                           #
###############################################################################

# Optional provider to serve legacy RPC requests, see README
OP_GETH__HISTORICAL_RPC=

# Set to "full" to force op-geth to use --syncmode=full
OP_GETH__SYNCMODE=snap

###############################################################################
#                                ↓ OPTIONAL ↓                                 #
###############################################################################

# Feel free to customize your image tag if you want, uses "latest" by default
# See here for all available images: https://hub.docker.com/u/ethereumoptimism
IMAGE_TAG__L2GETH=
IMAGE_TAG__DTL=
IMAGE_TAG__HEALTCHECK=
IMAGE_TAG__PROMETHEUS=
IMAGE_TAG__GRAFANA=
IMAGE_TAG__INFLUXDB=
IMAGE_TAG__OP_GETH=
IMAGE_TAG__OP_NODE=

# Exposed server ports (must be unique)
# See docker-compose.yml for default values
PORT__L2GETH_HTTP=8585
PORT__L2GETH_WS=8588
PORT__DTL=8606
PORT__HEALTHCHECK_METRICS=8607
PORT__PROMETHEUS=8601
PORT__GRAFANA=8602
PORT__INFLUXDB=8603
PORT__TORRENT_UI=8604
PORT__TORRENT=8605
PORT__OP_GETH_HTTP=8595
PORT__OP_GETH_WS=8598
PORT__OP_GETH_P2P=8599
PORT__OP_NODE_P2P=8600

Protocol Description

op-geth v1.101308.2
op-node op-node:v1.7.2
mainnet/op
10
1

Node Logs

No response

Additional Information

No response

Feedback

No response

[Chomtana]

Failed to raise file descriptor allowance seem to be related to ulimit

Try this one: https://docs.sisense.com/main/SisenseLinux/increasing-docker-ulimits.htm

[Chomtana]

It's very weird that root can't increase the limit. Maybe docker is doing the same so it result in the same error.

From https://bitsanddragons.wordpress.com/2023/09/27/ulimit-open-files-cannot-modify-limit-operation-not-permitted-on-centos-7/ There is an alternative way

We modify /etc/security/limits.conf by adding this before the “End of file” message. Like this

* hard nofile 100000
* soft nofile 100000

[tajelp]

I still get this from op-geth

Waiting for Bedrock node to initialize...
INFO [05-06|19:25:55.866] Starting geth on an OP network...        network=op-mainnet
INFO [05-06|19:25:55.867] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [05-06|19:25:55.867] Enabling metrics collection
INFO [05-06|19:25:55.868] Enabling metrics export to InfluxDB
INFO [05-06|19:25:55.870] Maximum peer count                       ETH=50 total=50
INFO [05-06|19:25:55.872] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
Fatal: Failed to raise file descriptor allowance: operation not permitted
Fatal: Failed to raise file descriptor allowance: operation not permitted

and this from op-node:

t=2024-05-06T19:26:29+0000 lvl=warn msg="failed to dial address, but may connect later" addr=http://op-geth:8551

[Chomtana]

It's probably from the host provider if even the root can't change the limit. Which cloud provider have you used?

[tajelp]

hetzner

[Chomtana]

Hmm very weird. I'm also using Hetzner (EX130-S) and it works without setting the ulimit. Are you using VPS or Bare metal?

If using Bare metal, probably try EX line EX101. Or even try with the same model in case the disk is corrupted.