Merge branch 'main' into synchronised-flag

Consensys · Mar 25, 2024 · d22d6f1 · d22d6f1
2 parents 45ff5f5 + df116fb
commit d22d6f1
Show file tree

Hide file tree

Showing 5 changed files with 108 additions and 29 deletions.
diff --git a/docs/how-to/troubleshoot/network.md b/docs/how-to/troubleshoot/network.md
@@ -67,7 +67,7 @@ View the [Prysm guide](https://docs.prylabs.network/docs/prysm-usage/p2p-host-ip
 
 :::
 
-### Advertised IP address issues 
+### Advertised IP address issues
 
 A possible reason for incoming peers being unable to connect could be an incorrect address specified using the
 [`--p2p-advertised-ip`](../../reference/cli/index.md#p2p-advertised-ip) option. Teku auto-detects the address to use by
@@ -78,7 +78,7 @@ correct firewall and forwarding settings, this could potentially be the cause.
 ### Network gateway issues
 
 A potential reason for incoming peers not being able to connect could be the use of a different port on your network
-gateway (router or modem). 
+gateway (router or modem).
 This usually happens because only one service can listen on a port. Therefore, if you're running multiple beacon nodes, you'll
 need to open multiple ports on your gateway. The simplest solution is to use the same port on your gateway as specified
 in your [`--p2p-port`](../../reference/cli/index.md#p2p-port) (9000 by default). However, if necessary, users can also
@@ -106,6 +106,50 @@ Common issues include:
  misses, typically anything over 10 Mbps upstream is acceptable.
 
 
+## Excessive late block import warnings due to time skew
+
+In Ethereum, every proposed block is expected to propagate through the network and reach every beacon node within
+four seconds into the current slot. Whenever Teku receives a block after the expected period, it prints a warning to
+the logs that looks like:
+
+```
+2024-03-18 17:32:27.363 WARN - Late Block Import *** Block: a0ad54151e1e629ac4a3c23d768e100a9f017b229c927c23ea90111f6399cbdf (8659360) proposer 858815 arrival 4083ms, gossip_validation +4ms, pre-state_retrieved +3ms, processed +259ms, execution_payload_result_received +0ms, begin_importing +1ms, transaction_prepared +0ms, transaction_committed +0ms, completed +13ms
+```
+
+The `arrival` value in the message indicates the time the block was received by the node. In this particular
+case, the block arrived 4083ms after the start of the slot (more than four seconds). Therefore, Teku printed
+the warning message.
+
+Even on a healthy network, some late blocks are expected. It's impossible to completely eliminate them, as most of
+the time, a block being late has nothing to do with your node specifically. However, if you're seeing multiple late
+block warnings in the logs, it's possible that your server's timing configuration is incorrect, causing your node
+to perceive blocks as late when, in reality, the server's clock is misaligned with the rest of the network. This
+is why it's important to use a service like NTPD or Chrony to keep your server's clock synchronized.
+
+If you suspect your server clock is out-of-sync, use a dashboard like the
+[Grafana Node Exporter Dashboard](https://grafana.com/grafana/dashboards/1860-node-exporter-full/) to check.
+Look for the **System Timesync** panel and examine the **Time Synchronized Drift** chart, which shows how much
+your server clock is drifting from other nodes in the network. A higher drift indicates a greater deviation between
+your system clock and other nodes, potentially causing issues for Teku.
+
+Here is an image that shows the **Time Synchronized Drift** chart before and after the server clock being
+adjusted using Chrony:
+
+![Time Synchronized Drift](../../images/time_synchronized_drift_chart.png)
+
+:::note
+
+Having zero time drift is impossible in practice. The Ethereum protocol has been designed to withstand up to
+500ms of variance between nodes.
+
+:::
+
+References:
+- [Monitoring a Linux host with Prometheus and node_exporter](https://grafana.com/docs/grafana-cloud/send-data/metrics/metrics-prometheus/prometheus-config-examples/noagent_linuxnode/)
+- [Node Exporter Grafana Dashboard](https://grafana.com/grafana/dashboards/1860-node-exporter-full/)
+- [Using `chrony`` to configure NTP](https://ubuntu.com/blog/ubuntu-bionic-using-chrony-to-configure-ntp)
+- [Why clock sync matters in Ethereum 2.0](https://hackmd.io/@ericsson49/BJfLjEX-8)
+
 ## Address missing attestations or non-inclusion issues
 
 * No peers might have been present on the attestation subnet. Check for a log message when attempting to

diff --git a/docs/images/time_synchronized_drift_chart.png b/docs/images/time_synchronized_drift_chart.png
diff --git a/docs/reference/cli/index.md b/docs/reference/cli/index.md
@@ -121,7 +121,7 @@ builder-bid-compare-factor: 50
  </TabItem>
 </Tabs>
 
-The builder bid compare factor. The default is 100 (100%).
+The builder bid compare factor. The default is 90 (90%).
 
 Execution layer clients in [Capella-enabled networks](https://notes.ethereum.org/@launchpad/withdrawals-faq#Q-What-is-ShanghaiCapella) provide the execution payload and the payload value. The beacon node compares this value against the builder bid to maximize the validator's profit or decrease network censorship at a low or no cost.
 
@@ -241,7 +241,7 @@ If it can't download the finalized state, it tries to download the genesis state
 See [this community-maintained list of checkpoint state endpoints](https://eth-clients.github.io/checkpoint-sync-endpoints/).
 :::
 
-When this option is set, and `--deposit-snapshot-enabled` is also not set or disabled, 
+When this option is set, and `--deposit-snapshot-enabled` is also not set or disabled,
 the `--checkpoint-sync-url` value will be used to determine the deposit snapshot.
 
 ### `config-file`
@@ -504,7 +504,6 @@ data-validator-path: "/home/me/me_validator"
 
 Path to the validator client data. The default is `<data-base-path>/validator` where `<data-base-path>` is specified using [`--data-base-path`](#data-base-path-data-path).
 
-
 ### `deposit-snapshot-enabled`
 
 <Tabs>
@@ -659,7 +658,7 @@ ee-jwt-claim-id: "foobar"
 </Tabs>
 
 A unique identifier for the consensus layer client. When using the JSON-RPC API engine, this identifier is added to JWT claims as an `id` claim.
- 
+
 ### `ee-jwt-secret-file`
 
 <Tabs>
@@ -886,7 +885,7 @@ exit-when-no-validator-keys-enabled: true
 </Tabs>
 
 The default setting is `false`.
-If this option is set to `false`, Teku continues running even when no validator keys are loaded. 
+If this option is set to `false`, Teku continues running even when no validator keys are loaded.
 
 If this option is set to `true`, Teku automatically exits if no validator keys are loaded, or there are no active validators.
 
@@ -1385,7 +1384,7 @@ metrics-categories: ["BEACON", "JVM", "PROCESS"]
  </TabItem>
 </Tabs>
 
-Categories for which to track metrics. Options are `JVM`, `PROCESS`, `BEACON`, `DISCOVERY`, `EVENTBUS`, `EXECUTOR`, `LIBP2P`, `NETWORK`, `STORAGE`, `STORAGE_HOT_DB`, `STORAGE_FINALIZED_DB`, `REMOTE_VALIDATOR`, `VALIDATOR`, `VALIDATOR_PERFORMANCE`, `VALIDATOR_DUTY`.  All but `VALIDATOR_DUTY` categories are enabled by default.
+Categories for which to track metrics. Options are `JVM`, `PROCESS`, `BEACON`, `DISCOVERY`, `EVENTBUS`, `EXECUTOR`, `LIBP2P`, `NETWORK`, `STORAGE`, `STORAGE_HOT_DB`, `STORAGE_FINALIZED_DB`, `REMOTE_VALIDATOR`, `VALIDATOR`, `VALIDATOR_PERFORMANCE`, `VALIDATOR_DUTY`. All but `VALIDATOR_DUTY` categories are enabled by default.
 
 When `metrics-categories` is used, only the categories specified in this option are enabled (all other categories are disabled).
 
@@ -1644,16 +1643,16 @@ The default is `mainnet`.
 
 Possible values are:
 
-| Network | Chain | Type | Description |
-| :-- | :-- | :-- | :-- |
-| `mainnet` | Consensus layer | Production | Main network |
-| `minimal` | Consensus layer | Test | Used for local testing and development networks |
-| `goerli` | Consensus layer | Test | Multi-client testnet |
-| `gnosis` | Consensus layer | Production | Network for the [Gnosis chain](https://www.gnosis.io/) |
-| `holesky` | Consensus layer | Test | Multi-client testnet |
-| `sepolia` | Consensus layer | Test | Multi-client testnet |
-| `chiado` | Consensus layer | Test | Gnosis [testnet](https://docs.gnosischain.com/concepts/networks/chiado) |
-| `lukso` | Consensus layer | Production | Network for the [Lukso chain](https://lukso.network/) |
+| Network  | Chain  | Type  | Description  |
+| :-------- | :-------------- | :--------- | :---------------------------------------------------------------------- |
+| `mainnet` | Consensus layer | Production | Main network  |
+| `minimal` | Consensus layer | Test  | Used for local testing and development networks  |
+| `goerli`  | Consensus layer | Test  | Multi-client testnet  |
+| `gnosis`  | Consensus layer | Production | Network for the [Gnosis chain](https://www.gnosis.io/)  |
+| `holesky` | Consensus layer | Test  | Multi-client testnet  |
+| `sepolia` | Consensus layer | Test  | Multi-client testnet  |
+| `chiado` | Consensus layer | Test  | Gnosis [testnet](https://docs.gnosischain.com/concepts/networks/chiado) |
+| `lukso`  | Consensus layer | Production | Network for the [Lukso chain](https://lukso.network/)  |
 
 Predefined networks can provide defaults such as the initial state of the network, bootnodes, and the address of the deposit contract.
 
@@ -1764,6 +1763,44 @@ p2p-advertised-udp-port: 1789
 
 The advertised UDP port to external peers. The default is the port specified in [`--p2p-advertised-port`](#p2p-advertised-port) if it is set. Otherwise, the default is the port specified in [`--p2p-port`](#p2p-port).
 
+### `p2p-direct-peers`
+
+<Tabs>
+ <TabItem value="Syntax" label="Syntax" default>
+
+```bash
+--p2p-direct-peers=<ADDRESS>[,<ADDRESS>...]...
+```
+
+ </TabItem>
+ <TabItem value="Example" label="Example" >
+
+```bash
+--p2p-direct-peers=/ip4/151.150.191.80/tcp/9000/p2p/16Ui...aXRz,/ip4/151.150.191.80/tcp/9000/p2p/16Ui...q6f1
+```
+
+ </TabItem>
+ <TabItem value="Environment variable" label="Environment variable" >
+
+```bash
+TEKU_P2P_DIRECT_PEERS=/ip4/151.150.191.80/tcp/9000/p2p/16Ui...aXRz,/ip4/151.150.191.80/tcp/9000/p2p/16Ui...q6f1
+```
+
+ </TabItem>
+ <TabItem value="Configuration file" label="Configuration file" >
+
+```bash
+p2p-direct-peers: ["/ip4/151.150.191.80/tcp/9000/p2p/16Ui...aXRz",
+ "/ip4/151.150.191.80/tcp/9000/p2p/16Ui...q6f1"]
+```
+
+ </TabItem>
+</Tabs>
+
+List of comma-separated [multiaddresses](https://docs.libp2p.io/concepts/appendix/glossary/#multiaddr) of direct peers
+with which to establish and maintain connections. Direct peers are static peers with which this node will always
+exchange full messages, regardless of peer scoring mechanisms. Such peers will also need to enable you as direct
+in order to work.
 
 ### `p2p-discovery-bootnodes`
 
@@ -1801,8 +1838,6 @@ p2p-discovery-bootnodes: ["enr:-Iu4QG...wgiMo",
 
 List of comma-separated Ethereum Node Records (ENRs) for P2P discovery bootstrap.
 
-
-
 ### `p2p-discovery-enabled`
 
 <Tabs>
@@ -2103,7 +2138,6 @@ p2p-port: 1789
 
 Specifies the P2P listening ports (UDP and TCP). The default is `9000`.
 
-
 ### `p2p-private-key-file`
 
 <Tabs>
@@ -2181,7 +2215,8 @@ p2p-static-peers: ["/ip4/151.150.191.80/tcp/9000/p2p/16Ui...aXRz",
  </TabItem>
 </Tabs>
 
-List of comma-separated [multiaddresses](https://docs.libp2p.io/concepts/appendix/glossary/#multiaddr) of static peers.
+List of comma-separated [multiaddresses](https://docs.libp2p.io/concepts/appendix/glossary/#multiaddr) of static peers
+with which to establish and maintain connections.
 
 ### `p2p-subscribe-all-subnets-enabled`
 
@@ -2919,7 +2954,6 @@ validator-api-port: 5052
 
 The [validator REST API](../rest.md#enable-the-validator-client-api) listening port (HTTP). The default is 5052.
 
-
 ### `validators-builder-registration-default-enabled`
 
 <Tabs>

diff --git a/package-lock.json b/package-lock.json
diff --git a/project-words.txt b/project-words.txt
@@ -569,6 +569,7 @@ templating
 temurin
 thanos
 therox
+timesync
 tmpdir
 toolset
 toplevel