Subscribe Very Slow / Hanging [Rust & Python]

[hub-il]

Subscribing to a stream is very very to return a result. Sometimes it will return in 45 seconds and sometimes it returns nothing after 3 hours. I'm running Solana 2.0.18 and Yellowstone v3.0.1+solana.2.0.18.

How fast should blocks be returned and what could be causing such a significant delay?

Example can run for hours and return nothing:

./client-ubuntu-22.04 -e http://127.0.0.1:10000 subscribe --blocks

Responses not requiring a subscription return instantly:

./client-ubuntu-22.04 -e http://127.0.0.1:10000 get-block-height
[2024-12-11T18:02:03Z INFO  client] Connected
[2024-12-11T18:02:03Z INFO  client] response: GetBlockHeightResponse { block_height: 285119088 }

Yellowstone Config:

{
  "libpath": "/nodedata/solana_data/yellowstone-grpc-geyser-release_v3.0.1+solana.2.0.18/lib/libyellowstone_grpc_geyser.so",
  "log": {
    "level": "info"
  },
  "grpc": {
    "address": "127.0.0.1:10000",
    "max_decoding_message_size": "100_000_000",
    "snapshot_plugin_channel_capacity": null,
    "snapshot_client_channel_capacity": "50_000_000",
    "channel_capacity": "100_000",
    "unary_concurrency_limit": 100,
    "unary_disabled": false
  },
  "prometheus": {
    "address": "0.0.0.0:8999"
  }
}

[Juanito87]

What's the latency between producer and consumer? do you have enough bandwith for your filters?

[hub-il]

@Juanito87 thanks for the reply. Both producer and consumer are on the same machine if that's what you mean. Client is pointing to 127.0.0.1.

As far as bandwidth for your filters I'm not sure what you mean. Are you talking about IO bandwidth, network bandwidth, CPU, RAM? Network, RAM, and IO should all be good. CPU shows 32 cores in HTOP.

Using the rust client my request looks just like this ./client-ubuntu-22.04 -e http://127.0.0.1:10000 subscribe --blocks

Using Python my request looks like this:

blocks_filter = geyser_pb2.SubscribeRequestFilterBlocks(include_transactions=True, include_accounts=True, include_entries=True)
            stream_request = [geyser_pb2.SubscribeRequest(
                slots=[],
                accounts={},
                transactions={},
                transactions_status={},
                entry={},
                blocks={"client": blocks_filter},
                blocks_meta={},
                commitment=CommitmentLevel.FINALIZED,
                accounts_data_slice=[],
                ping=None
            )]

[hub-il]

@Juanito87 I have been able to get some data to stream but not consistently. The machine I am using now has 64 cores and 512 GB of ram. The cores are AMD 2.5GZ boosting to 3.0GZ. Local NVME drives stripped and separate drives for accounts and the ledger.

I am having the same issues with Websockets, the Rust client, and Python. Seems like others are running with these configs just fine. I just don't know what the issue could be.

My agave-validator config looks like this:

agave-validator \
--identity /nodedata/solana_data/validator-keypair.json \
--log - \
--ledger /nodedata/solana_data/ledger_data \
--accounts /nodedata/solana_data/accounts_data \
--entrypoint entrypoint.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint2.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint3.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint4.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint5.mainnet-beta.solana.com:8001 \
--known-validator Certusm1sa411sMpV9FPqU5dXAYhmmhygvxJ23S6hJ24 \
--known-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 \
--known-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ \
--known-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S \
--expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d \
--expected-shred-version 50093 \
--only-known-rpc \
--full-rpc-api \
--no-voting \
--no-wait-for-vote-to-start-leader \
--private-rpc \
--enable-cpi-and-log-storage \
--no-skip-initial-accounts-db-clean \
--dynamic-port-range 8100-8200 \
--rpc-bind-address 0.0.0.0 \
--rpc-port 7699 \
--no-port-check \
--enable-rpc-transaction-history \
--wal-recovery-mode skip_any_corrupted_record \
--use-snapshot-archives-at-startup when-newest \
--limit-ledger-size \
--rpc-pubsub-enable-block-subscription \
--enable-extended-tx-metadata-storage \
--geyser-plugin-config /nodedata/solana_data/yellowstone_grpc_config_latest.json

And my Yellowstone config looks like this:

{
    "libpath": "/nodedata/solana_data/yellowstone-grpc-geyser-release_v3.0.1+solana.2.0.18/lib/libyellowstone_grpc_geyser.so",
    "log": {
        "level": "info"
    },
    "grpc": {
        "address": "0.0.0.0:7505",
        "compression": {
        "accept": [
            "gzip"
        ],
        "send": [
            "gzip"
        ]
        },
        "max_decoding_message_size": "50_000_000",
        "snapshot_plugin_channel_capacity": null,
        "snapshot_client_channel_capacity": "50_000_000",
        "channel_capacity": "100_000",
        "unary_concurrency_limit": 100,
        "unary_disabled": false
    },
    "prometheus": {
        "address": "0.0.0.0:8999"
    }
}

[OldTyT]

I also noticed that gRPC works slowly. I compared it with OrbitFlare. In my estimation, gRPC works 2000 times slower than Orbit. Perhaps to speed things up, the Solana node needs to be staked?

[OldTyT]

My agave-validator config:

/home/solana/agave/2.0.21-1.78.0-bookworm/agave-validator \
  --identity /home/solana/keys/identity.json \
  --known-validator Fzc7TuiVneTK26Dx1qVjczWR7kk18DC5s2sAwPsF1Gms \
  --known-validator mcs1kpUkWeqoruxWwtCskY1GGF4Bx1t3MMtHSHoSLyC \
  --known-validator 37nZfUbZkjnsJChjqNi6h1n7fqMpYGUs3PpFvJUPpuXD \
  --known-validator 7Lp6sJYnJL2p33oPshdfAx5FSvYDFYRpng2gHJDTEQeN \
  --known-validator E8vh8n4eAYZ67nrFHk3czi8iFG1QoY8vsbvNZGUHPhSV \
  --known-validator scs2tFfBCobDRjkYfhQ59saMg91teNACbje18SraSwg \
  --known-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 \
  --known-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ \
  --known-validator DE1bawNcRJB9rVm3buyMVfr8mBEoyyu73NBovf2oXJsJ \
  --known-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S \
  --only-known-rpc \
  --full-rpc-api \
  --no-voting \
  --ledger /solana/tmpfs/ledger \
  --accounts /solana/accounts/accounts/ \
  --log - \
  --rpc-port 8899 \
  --rpc-bind-address 127.0.0.1 \
  --private-rpc \
  --dynamic-port-range 8000-8020 \
  --limit-ledger-size 5000000 \
  --entrypoint entrypoint.mainnet-beta.solana.com:8001 \
  --entrypoint entrypoint2.mainnet-beta.solana.com:8001 \
  --entrypoint entrypoint3.mainnet-beta.solana.com:8001 \
  --entrypoint entrypoint4.mainnet-beta.solana.com:8001 \
  --entrypoint entrypoint5.mainnet-beta.solana.com:8001 \
  --expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d \
  --no-snapshot-fetch \
  --wal-recovery-mode skip_any_corrupted_record \
  --account-index program-id spl-token-owner \
  --account-index-exclude-key kinXdEcpDQeHPEuQnqmUgtYykqKGVFq6CeVX5iAHJq6 \
  --account-index-exclude-key TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA \
  --snapshots /solana/ledger/snapshots \
  --accounts-hash-cache-path /solana/tmpfs/accounts_hash_cache \
  --accounts-index-path /solana/tmpfs/accounts_index \
  --tower /solana/tmpfs/tower \
  --geyser-plugin-config /home/solana/yellowstone-grpc-geyser-config.json \
  --enable-rpc-transaction-history \
  --enable-rpc-bigtable-ledger-storage \
  --maximum-full-snapshots-to-retain 1 \
  --maximum-incremental-snapshots-to-retain 1 \
  --enable-bigtable-ledger-upload

My yellowstone config:

{
   "libpath":"/home/solana/yellowstone-grpc/v2.0.0+solana.2.0.18-1.78.0-bookworm/libyellowstone_grpc_geyser.so",
   "log":{
      "level":"info"
   },
   "grpc":{
      "address":"0.0.0.0:10001",
      "channel_capacity":"100_000",
      "unary_concurrency_limit":100,
      "unary_disabled":false,
      "max_decoding_message_size":"4_194_304"
   },
   "prometheus":{
      "address":"0.0.0.0:10002"
   }
}