Add Limit Trie Logs Early Access Feature Guide (#1538)

* Add deprecation note

* Rewrite PR

* add-info

* Minor rewrite of bonsai trie logs

* apply Eng review suggestions

Signed-off-by: Joan E <[email protected]>

* Add admonition

Signed-off-by: Joan E <[email protected]>

* Add trie logs words to list 

Signed-off-by: Joan E <[email protected]>

* fix broken link

Signed-off-by: Joan E <[email protected]>

* update formatting

Signed-off-by: Joan E <[email protected]>

* cleanup

Signed-off-by: Joan E <[email protected]>

* change wording

Co-authored-by: Simon Dudley <[email protected]>
Signed-off-by: Joan E <[email protected]>

* change format

Signed-off-by: Joan E <[email protected]>

* fix spacing

Signed-off-by: Joan E <[email protected]>

* Apply suggestions from code review

Co-authored-by: Alexandra Tran Carrillo <[email protected]>
Signed-off-by: Joan E <[email protected]>

* Apply suggestions from code review

Co-authored-by: Alexandra Tran Carrillo <[email protected]>
Signed-off-by: Joan E <[email protected]>

* change heading

Signed-off-by: Joan E <[email protected]>

---------

Signed-off-by: Joan E <[email protected]>
Co-authored-by: Simon Dudley <[email protected]>
Co-authored-by: Alexandra Tran Carrillo <[email protected]>

docs/public-networks/how-to/bonsai-limit-trie-logs.md

@@ -0,0 +1,208 @@
+---
+title: Reduce storage for Bonsai Tries
+sidebar_position: 12
+description: Enable this feature to reduce the size of your database when using Bonsai Tries
+tags:
+  - public networks
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+To decrease the database size when using the [Bonsai Trie](../concepts/data-storage-formats#bonsai-tries) data storage format, enable the early access feature `--Xbonsai-limit-trie-logs-enabled`. 
+When enabled, this feature can reduce database growth by more than 3 GB each week on Mainnet.
+
+## Limit and prune trie logs
+
+:::caution
+
+The following commands are examples. Before executing these example commands on your node, modify them to apply to your node's configuration.
+
+:::
+
+1. Add the `--Xbonsai-limit-trie-logs-enabled` option to the [Besu configuration file](use-configuration-file).
+
+    :::note
+
+    If you are using a `systemd` service file, as recommended by [CoinCashew](https://www.coincashew.com/coins/overview-eth/guide-or-how-to-setup-a-validator-on-eth2-mainnet/part-i-installation/step-3-installing-execution-client/besu) 
+    and [Somer Esat](https://someresat.medium.com/guide-to-staking-on-ethereum-ubuntu-teku-f09ecd9ef2ee), ensure you execute `sudo systemctl daemon-reload`.
+
+    :::
+
+1. Stop Besu.
+1. (Optional) Run the Besu trie log prune command. Specify the Bonsai Trie data storage format and the data directory for your Besu database:
+    ```bash
+    sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=X_SNAP storage x-trie-log prune
+    ```
+1. Start Besu.
+1. Look for `Limit trie logs enabled: retention: 512; prune window: 30000` in your Besu configuration printout at startup.
+
+### Prune outdated trie logs
+
+When you start Besu with `--Xbonsai-limit-trie-logs-enabled`, it continuously prunes the unnecessary trie log data, removing it one block at a time.
+This process begins after an initial reduction in the database size during startup.
+
+Enabling `--Xbonsai-limit-trie-logs-enabled` on a long-running node does not immediately clear your backlog of trie logs in the same way resyncing does. 
+Instead of resyncing, you can run an offline command to immediately prune old trie logs. 
+To run the offline command, you must shut down Besu for a minimal period. 
+If the `--Xbonsai-limit-trie-logs-enabled` option is enabled, you do not need to run the offline command again after initially running it.
+
+For minimal downtime, we recommend running the offline command before restarting Besu with `--Xbonsai-limit-trie-logs-enabled`.
+
+If you are following the guides by [Somer Esat](https://someresat.medium.com/guide-to-staking-on-ethereum-ubuntu-teku-f09ecd9ef2ee) or [CoinCashew](https://www.coincashew.com/coins/overview-eth/guide-or-how-to-setup-a-validator-on-eth2-mainnet/part-i-installation/step-3-installing-execution-client/besu), you have set the following options in your `besu.service` or `execution.service` systemd file:
+
+```
+...
+ExecStart=/usr/local/bin/besu/bin/besu \
+...
+  --sync-mode=SNAP \
+  --data-path="/var/lib/besu" \
+  --data-storage-format=BONSAI \
+...
+```
+To prune trie logs, the command should look similar to the following:
+
+```
+sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=SNAP storage x-trie-log prune
+```
+
+The logs should look similar to the following:
+
+```
+2024-02-02 05:45:41.162+00:00 | main | INFO  | KeyPairUtil | Attempting to load public key from /data/besu/key
+ ...
+2024-02-02 05:45:43.433+00:00 | main | INFO  | TrieLogSubCommand | Estimating trie logs size before pruning...
+2024-02-02 05:45:43.837+00:00 | main | INFO  | TrieLogSubCommand | Estimated trie logs size before pruning: 9 GiB
+2024-02-02 05:46:09.863+00:00 | main | INFO  | TrieLogHelper | Starting pruning: retain 512 trie logs, processing in 1 batches...
+2024-02-02 05:46:09.918+00:00 | main | INFO  | TrieLogHelper | Saving trie logs to retain in file /data/besu/database/trieLogsToRetain-1 (batch 1)...
+2024-02-02 05:46:09.926+00:00 | main | INFO  | TrieLogHelper | Obtaining trielogs from db, this may take a few minutes...
+2024-02-02 05:46:10.100+00:00 | main | INFO  | TrieLogHelper | Clear trie logs...
+2024-02-02 05:46:10.155+00:00 | main | INFO  | TrieLogHelper | Restoring trie logs retained from batch 1...
+2024-02-02 05:46:10.222+00:00 | main | INFO  | TrieLogHelper | Key(0): 0xcd50706da7f6f2db7f9d54f3589122760900d9ab2508c20a4ca40b496d930368
+... 
+2024-02-02 05:46:10.336+00:00 | main | INFO  | TrieLogHelper | Key(511): 0x238f9649b59616430ad7e43b8f3cf65bc97cac4aa54a3eddf3ad6ee666ce733e
+2024-02-02 05:46:10.441+00:00 | main | INFO  | TrieLogHelper | Deleting files...
+2024-02-02 05:46:10.446+00:00 | main | INFO  | TrieLogSubCommand | Finished pruning. Re-estimating trie logs size...
+2024-02-02 05:46:11.023+00:00 | main | INFO  | TrieLogSubCommand | Estimated trie logs size after pruning: 0 B (0 B estimate is normal when using default settings)
+2024-02-02 05:46:11.023+00:00 | main | INFO  | TrieLogSubCommand | Prune ran successfully. We estimate you freed up 9 GiB!
+Prune ran successfully. We estimate you freed up 9 GiB!
+```
+
+If you are using a TOML configuration file, you can run a command similar to the following:
+
+```bash
+sudo /usr/local/bin/besu/bin/besu --config-file=besu-config.toml storage x-trie-log prune
+```
+
+## Troubleshoot
+
+Troubleshoot common errors that can occur when using the trie log prune command to reduce your database size for Bonsai Tries.
+To minimize errors, ensure your command specifies the following:
+
+- [`--data-storage-format`](../reference/cli/options.md#data-storage-format)
+- [`--data-path`](../reference/cli/options.md#data-path)
+- [`--sync-mode`](../reference/cli/options.md#sync-mode)
+
+### Prune command for Mainnet
+
+The prune command should look similar to the following:
+
+```bash
+sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=SNAP storage x-trie-log prune
+```
+
+Ensure you stop Besu before running the command.
+
+###  Subcommand not working
+
+`java.lang.IllegalArgumentException: Subcommand only works with data-storage-format=BONSAI`
+
+The `--data-storage-format=BONSAI` may be missing. 
+To resolve, add the storage format. 
+The command should look similar to the following:
+
+```bash
+sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=SNAP storage x-trie-log prune
+```
+
+### Column handle not found for segment `TRIE_BRANCH_STORAGE`
+
+`java.lang.RuntimeException: Column handle not found for segment TRIE_BRANCH_STORAGE`
+
+Ensure you specify `--data-path`. 
+Your command should look similar to the following:
+
+```bash
+sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=SNAP storage x-trie-log prune
+```
+
+### Database not detected
+
+`java.lang.IllegalArgumentException: Trying to retain more trie logs than chain length (0), skipping pruning`
+
+Ensure you specify the correct `--data-path` for your node. 
+Your command should look similar to the following:
+
+```bash
+sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=SNAP storage x-trie-log prune
+```
+
+### Cannot store generated private key
+
+`java.lang.IllegalArgumentException: Cannot store generated private key`
+
+Ensure you specify the correct `--data-path` for your node. 
+Your command should look similar to the following:
+
+```bash
+sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=SNAP storage x-trie-log prune
+```
+
+### Valid keypair not provided
+
+`java.lang.IllegalArgumentException: Supplied file does not contain valid keyPair pair.`
+
+Check your file permissions and try running a `sudo` command to resolve the issue:
+
+```bash
+sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu storage --sync-mode=SNAP x-trie-log prune
+```
+
+### Column handle not found for segment `WORLD_STATE`
+
+`java.lang.RuntimeException: Column handle not found for segment WORLD_STATE`
+
+Ensure you are using `--data-storage-format=BONSAI` instead of `--data-storage-format=FOREST` on an existing Bonsai database.
+
+### Resource temporarily unavailable
+
+`org.hyperledger.besu.plugin.services.exception.StorageException: org.rocksdb.RocksDBException: While lock file: /data/besu/database/LOCK: Resource temporarily unavailable`
+
+Check if Besu is already running. 
+You must shut down the Besu client before running the subcommand.
+
+### Unable to change the sync mode
+
+`java.lang.IllegalStateException: Unable to change the sync mode when snap sync is incomplete, please restart with snap sync mode`
+
+Check that you have specified `--sync-mode`. 
+The default is `--sync-mode=FAST`. 
+Most Mainnet users use `X_SNAP` or `X_CHECKPOINT`. 
+
+### Cannot run trie log prune
+
+`java.lang.RuntimeException: No finalized block present, can't safely run trie log prune`
+
+This message may appear if your node is relatively new or recently resynced. 
+To resolve this error, ensure that your node is fully synced and correctly configured to recognize finalized blocks.
+
+### Block does not match stored chain data
+
+`org.hyperledger.besu.util.InvalidConfigurationException: Supplied genesis block does not match chain data stored in /data/besu.`
+
+Check if you are running the command for a network other than Mainnet. 
+To specify a network, run a command that looks similar to the following:
+
+```bash
+`sudo /usr/local/bin/besu/bin/besu --network=holesky --sync-mode=SNAP --data-storage-format=BONSAI --data-path=/var/lib/besu storage x-trie-log prune`
+```

project-words.txt

@@ -171,6 +171,7 @@ entrypoints
 envr
 epochlength
 errnametoolong
+Esat
 esbenp
 esbuild
 eslintcache
@@ -530,6 +531,7 @@ reqs
 requesttimeout
 requesttimeoutseconds
 requireindex
+resynced
 resyncs
 retesteth
 retrocompatibility
@@ -575,6 +577,7 @@ sluggifies
 sluggify
 smartcard
 solana
+Somer
 spâce
 ssszzz
 stackblitz