Performance degeneration with time?

[slawomir-pryczek]

Hi Everyone, i noticed a pattern when hosting solana rpc node (not full validator). After some time perfrormance degrates to a point where it's no longer possible for the node to timely process new data, so it starts lagging more and more and is never able to catch up again. Datadirs needs to be cleaned-up, new snapshot downloaded and the node needs to be started again. Not sure what could be the case maybe it's DB or RAM fragmentation.

EPYC 73F3 x2
1.5TB Ram
solana-validator 1.18.18

Proposed solution:
Any possibility to run some cleanup / defragment on a working node from time to time, or maybe my config is incorrect?

./solana-validator \
    --identity ~/validator-keypair.json \
    --ledger /home2/solana/data/ledger \
    --accounts /home2/solana/data/wallet \
    --rpc-port 8899 \
    --enable-rpc-transaction-history \
    --private-rpc \
    --dynamic-port-range 8000-8020 \
    --entrypoint entrypoint.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint2.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint3.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint4.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint5.mainnet-beta.solana.com:8001 \
    --expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d \
    --wal-recovery-mode skip_any_corrupted_record \
    --limit-ledger-size 450000000 \
    --account-index program-id \
    --account-index spl-token-mint \
    --account-index spl-token-owner \
    --bind-address XXXXXXXXXXXXX \
    --rpc-bind-address 0.0.0.0 \
    --maximum-local-snapshot-age 999999999 \
    --enable-cpi-and-log-storage \
    --full-rpc-api \
    --geyser-plugin-config /home/solana/yellowstone/config.json

[github-actions]

This repository is no longer in use. Please re-open this issue in the agave repo: https://github.com/anza-xyz/agave