Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Interx memory leak and resource usage optimization. #243

Open
MrLutik opened this issue Dec 1, 2024 · 0 comments
Open

Interx memory leak and resource usage optimization. #243

MrLutik opened this issue Dec 1, 2024 · 0 comments
Assignees
@kmlbgn @golnar-boosty
Labels
bug Something isn't working enhancement New feature or request

Comments

@MrLutik
Copy link
Collaborator

MrLutik commented Dec 1, 2024 •
edited
Loading

The interx consumed all the memory and was killed by OOM. The last logs are follows:

10:09AM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af timestamp=2024-12-01T10:09:12Z
10:09AM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" timestamp=2024-12-01T10:09:12Z
10:09AM INF Starting Wrap Response... timestamp=2024-12-01T10:09:12Z
10:09AM INF Starting 'GetResponseSignature' request... timestamp=2024-12-01T10:09:12Z
10:09AM INF Cache hit for `QueryStatus` request. error=null endpoint=/api/status method=GET params= timestamp=2024-12-01T10:09:12Z
10:09AM INF Finished 'GetResponseSignature' request. timestamp=2024-12-01T10:09:12Z
10:09AM INF Starting 'GetInterxRequest' request... endpoint=/api/valopers?address=Unknown method=GET timestamp=2024-12-01T10:09:12Z
10:09AM INF Finished 'GetInterxRequest' request. timestamp=2024-12-01T10:09:12Z
10:09AM INF Starting SearchCache timestamp=2024-12-01T10:09:12Z
10:09AM INF `SearchCache` Config Path chainIDHash=c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed endpointHash=5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5 requestHash=42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a timestamp=2024-12-01T10:09:12Z
10:09AM INF Starting 'GetCache' request... endpoint=5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5 timestamp=2024-12-01T10:09:12Z
10:09AM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a timestamp=2024-12-01T10:09:12Z
10:09AM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" timestamp=2024-12-01T10:09:12Z
10:09AM INF Starting Wrap Response... timestamp=2024-12-01T10:09:12Z
10:09AM INF Starting 'GetResponseSignature' request... timestamp=2024-12-01T10:09:12Z
10:09AM INF Finished 'GetResponseSignature' request. timestamp=2024-12-01T10:09:12Z
10:09AM INF Starting `getStatus` request... rpc Addr=http://sekai.local:26657 timestamp=2024-12-01T10:09:13Z
10:09AM INF Starting 'AddBlockTime' request... timestamp=2024-12-01T10:09:13Z
10:09AM INF Starting 'GetBlockTime' request... timestamp=2024-12-01T10:09:13Z
10:09AM INF Starting `QueryStatus` request... timestamp=2024-12-01T10:09:14Z
10:09AM INF Starting 'GetInterxRequest' request... endpoint=/api/status method=GET timestamp=2024-12-01T10:09:14Z
10:09AM INF Finished 'GetInterxRequest' request. timestamp=2024-12-01T10:09:14Z
10:09AM INF Starting SearchCache timestamp=2024-12-01T10:09:14Z
10:09AM INF `SearchCache` Config Path chainIDHash=c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed endpointHash=8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1 requestHash=d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854 timestamp=2024-12-01T10:09:14Z
10:09AM INF Starting 'GetCache' request... endpoint=8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1 timestamp=2024-12-01T10:09:14Z
10:09AM INF Starting `saveGenesis` request... timestamp=2024-12-01T10:09:14Z
10:09AM INF Finished 'AddBlockTime' request. timestamp=2024-12-01T10:09:16Z
10:09AM INF Starting 'AddBlockNanoTime' request... timestamp=2024-12-01T10:09:16Z
10:09AM ERR [saveGenesis][getGenesisCheckSum] Failed to fetch genesis checksum. error=null timestamp=2024-12-01T10:09:16Z
10:09AM INF Starting 'GetBlockNanoTime' request... timestamp=2024-12-01T10:09:16Z
10:09AM ERR [GetGenesisResults][Unmarshal] Failed to unmarshal genesis doc. error=null timestamp=2024-12-01T10:09:16Z
10:09AM INF Starting 'GetKiraStatus' request... success={"node_info":{"channels":"40202122233038606100","id":"74d5bc5f3c4bae154777e0fc793974931ecbea09","listen_addr":"tcp://0.0.0.0:26656","moniker":"KIRA_TEST_LOCAL_VALIDATOR_NODE","network":"testnet-1","other":{"rpc_address":"tcp://0.0.0.0:26657","tx_index":"on"},"protocol_version":{"app":"0","block":"11","p2p":"8"},"version":"0.37.2"},"sync_info":{"catching_up":false,"earliest_app_hash":"E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855","earliest_block_hash":"676A01DD1EF03DF6E285741D8212BC01CDA3B16B6832603C3AF3D1DEBFF82D32","earliest_block_height":"1","earliest_block_time":"2024-09-12T09:14:20.01232859Z","latest_app_hash":"90B8821FD6F3840862BAF2CFA06F5D3DF0E1116134ADABDF197EC0F511D9A76B","latest_block_hash":"32BF75789FFD67C6AA84567101B3E0056BA764B1F33834BAFF0E04894440FE45","latest_block_height":"1179619","latest_block_time":"2024-12-01T10:09:09.186655507Z"},"validator_info":{"address":"E65C8993E9293DF88FD8DC685FBB1BFD9895DAD3","pub_key":{"type":"tendermint/PubKeyEd25519","value":"re6kzu8reGXPaFKbZifcFMEgcEEek81yAVCQ+KTGv5M="},"voting_power":"1"}} timestamp=2024-12-01T10:09:16Z
10:09AM INF Processed `QueryStatus` request. error=null endpoint=/api/status method=GET params= timestamp=2024-12-01T10:09:16Z
10:09AM INF Starting Wrap Response... timestamp=2024-12-01T10:09:16Z
10:09AM INF Starting 'PutCache' request... endpoint=8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1 timestamp=2024-12-01T10:09:16Z
10:09AM INF Finished 'AddBlockNanoTime' request. timestamp=2024-12-01T10:09:20Z
10:09AM INF Starting 'GetBlockNanoTime' request... timestamp=2024-12-01T10:09:20Z
10:09AM INF Finished 'PutCache' request. timestamp=2024-12-01T10:09:20Z
10:09AM INF Starting 'GetResponseSignature' request... timestamp=2024-12-01T10:09:20Z
10:09AM ERR [CacheHeaderCheck] Failed to check cache headers error="remove /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1: directory not empty" timestamp=2024-12-01T10:09:20Z
10:09AM INF Finished 'GetResponseSignature' request. timestamp=2024-12-01T10:09:20Z
10:09AM INF Finished `QueryStatus` request. timestamp=2024-12-01T10:09:20Z
10:09AM INF Starting `GetBlockNanoTime` call endpoint=http://sekai.local:26657/block?height=1179617 timestamp=2024-12-01T10:09:21Z
10:09AM INF Starting 'AddBlockTime' request... timestamp=2024-12-01T10:09:21Z
10:09AM INF Starting 'GetBlockTime' request... timestamp=2024-12-01T10:09:21Z
10:09AM INF Starting 'GetInterxRequest' request... endpoint=/api/valopers?address=Unknown method=GET timestamp=2024-12-01T10:09:22Z
10:09AM INF Finished 'GetInterxRequest' request. timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting SearchCache timestamp=2024-12-01T10:09:22Z
10:09AM INF `SearchCache` Config Path chainIDHash=c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed endpointHash=5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5 requestHash=42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting 'GetCache' request... endpoint=5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5 timestamp=2024-12-01T10:09:22Z
10:09AM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a timestamp=2024-12-01T10:09:22Z
10:09AM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting Wrap Response... timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting 'GetInterxRequest' request... endpoint=/api/valopers?all=true method=GET timestamp=2024-12-01T10:09:22Z
10:09AM INF Finished 'GetInterxRequest' request. timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting SearchCache timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting 'GetResponseSignature' request... timestamp=2024-12-01T10:09:22Z
10:09AM INF Finished 'GetResponseSignature' request. timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting `QueryStatus` request... timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting 'GetInterxRequest' request... endpoint=/api/status method=GET timestamp=2024-12-01T10:09:22Z
10:09AM INF Finished 'GetInterxRequest' request. timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting SearchCache timestamp=2024-12-01T10:09:22Z
10:09AM INF `SearchCache` Config Path chainIDHash=c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed endpointHash=8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1 requestHash=d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854 timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting 'GetCache' request... endpoint=8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1 timestamp=2024-12-01T10:09:22Z
10:09AM INF `SearchCache` Config Path chainIDHash=c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed endpointHash=aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091 requestHash=ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting 'GetCache' request... endpoint=aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091 timestamp=2024-12-01T10:09:22Z
10:09AM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af timestamp=2024-12-01T10:09:22Z
10:09AM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting Wrap Response... timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting 'GetResponseSignature' request... timestamp=2024-12-01T10:09:22Z
10:09AM INF Finished SearchCache timestamp=2024-12-01T10:09:22Z
10:09AM INF Starting 'GetResponseSignature' request... timestamp=2024-12-01T10:09:22Z
10:09AM INF Finished 'GetResponseSignature' request. timestamp=2024-12-01T10:09:22Z
10:09AM INF Cache hit for `QueryStatus` request. error=null endpoint=/api/status method=GET params= timestamp=2024-12-01T10:09:22Z
10:09AM INF Finished 'GetResponseSignature' request. timestamp=2024-12-01T10:09:22Z
10:09AM INF Finished 'AddBlockTime' request. timestamp=2024-12-01T10:09:24Z
10:09AM INF Starting 'AddBlockNanoTime' request... timestamp=2024-12-01T10:09:24Z
10:09AM INF Starting 'GetBlockNanoTime' request... timestamp=2024-12-01T10:09:24Z
2024/12/01 10:09:25 Server is starting on :8081...

My guess would be that the caching limit was exceeded.

Interx Cache CacheDir                  :  /home/chaos/interx/cache
Interx Cache MaxCacheSize              :  2147483648
Interx Cache CachingDuration           :  5
Interx Cache DownloadFileSizeLimitation:  10485760

The function responsible for the cache check.

  • The func uses infinite main loop.
  • The inner loop also infinite.
  • Re-computation of the cache size in this func is quite intense especially with infinite loop
  • Mutex lock used inconsistently. Why to unlock while checking file stats and then lock for deletion.

Suggestion: refactor the func.

Next. We start these helping funcs here The thing is that the value for loggin is hardcoded and set to false. So we need to change that. Verbosity should be a choice

Tree of the cache dir:

.
├── cache
│   ├── db
│   │   ├── block
│   │   │   └── BlockData
│   │   ├── blocknano
│   │   │   └── BlockNanoData
│   │   ├── faucet
│   │   ├── proposals
│   │   │   └── proposals
│   │   ├── ref
│   │   │   └── ReferenceData
│   │   ├── token-aliases.json
│   │   └── transactions
│   │       ├── kira13lsmpt73lf4p8ye4pj009whz2qagjwt5hv3ya5
│   │       ├── kira13lsmpt73lf4p8ye4pj009whz2qagjwt5hv3ya5-inbound
│   │       ├── kira143q8vxpvuykt9pq50e6hng9s38vmy844n8k9wx
│   │       └── kira143q8vxpvuykt9pq50e6hng9s38vmy844n8k9wx-inbound
│   ├── reference
│   │   └── genesis.json
│   └── response
│       └── c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed
│           └── 8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1
│               └── d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854

Size of the cache dir:

4.0K	./cache/db/ref
84K	./cache/db/transactions
4.0K	./cache/db/faucet
39M	./cache/db/block
8.0K	./cache/db/proposals
46M	./cache/db/blocknano
85M	./cache/db
8.0K	./cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1
12K	./cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed
16K	./cache/response
12K	./cache/reference
85M	./cache

After rebuilding the interx with all true value here
It crashes without any error messages.
I collected all errors which preceding the crash:

5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a timestamp=2024-12-01T17:09:12Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" timestamp=2024-12-01T17:09:12Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af timestamp=2024-12-01T17:09:12Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" timestamp=2024-12-01T17:09:12Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854 timestamp=2024-12-01T17:09:14Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854: no such file or directory" timestamp=2024-12-01T17:09:14Z
5:09PM ERR [saveGenesis][getGenesisCheckSum] Failed to fetch genesis checksum. error=null timestamp=2024-12-01T17:09:14Z
5:09PM ERR [GetGenesisResults][Unmarshal] Failed to unmarshal genesis doc. error=null timestamp=2024-12-01T17:09:14Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a timestamp=2024-12-01T17:09:22Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" timestamp=2024-12-01T17:09:22Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af timestamp=2024-12-01T17:09:22Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" timestamp=2024-12-01T17:09:22Z
5:09PM ERR [saveGenesis][getGenesisCheckSum] Failed to fetch genesis checksum. error=null timestamp=2024-12-01T17:09:24Z
5:09PM ERR [GetGenesisResults][Unmarshal] Failed to unmarshal genesis doc. error=null timestamp=2024-12-01T17:09:24Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854 timestamp=2024-12-01T17:09:32Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854: no such file or directory" timestamp=2024-12-01T17:09:32Z
5:09PM ERR [saveGenesis][getGenesisCheckSum] Failed to fetch genesis checksum. error=null timestamp=2024-12-01T17:09:32Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a timestamp=2024-12-01T17:09:32Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" timestamp=2024-12-01T17:09:32Z
5:09PM ERR [GetGenesisResults][Unmarshal] Failed to unmarshal genesis doc. error=null timestamp=2024-12-01T17:09:32Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af timestamp=2024-12-01T17:09:32Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" timestamp=2024-12-01T17:09:32Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854 timestamp=2024-12-01T17:09:42Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854: no such file or directory" timestamp=2024-12-01T17:09:42Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a timestamp=2024-12-01T17:09:42Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" timestamp=2024-12-01T17:09:42Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af timestamp=2024-12-01T17:09:42Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" timestamp=2024-12-01T17:09:42Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854 timestamp=2024-12-01T17:09:44Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/8ae288159346d481d8322cca0b1e1cf86fd36e096bad714f648b8d0ad0e844d1/d4f7e316bac8c7c8bbe7d58c244aae4aaee5d5af9067610405b51f1ac83b3854: no such file or directory" timestamp=2024-12-01T17:09:44Z
5:09PM ERR [saveGenesis][getGenesisCheckSum] Failed to fetch genesis checksum. error=null timestamp=2024-12-01T17:09:45Z
5:09PM ERR [saveGenesis][getGenesisCheckSum] Failed to fetch genesis checksum. error=null timestamp=2024-12-01T17:09:45Z
5:09PM ERR [GetGenesisResults][Unmarshal] Failed to unmarshal genesis doc. error=null timestamp=2024-12-01T17:09:45Z
5:09PM ERR [GetGenesisResults][Unmarshal] Failed to unmarshal genesis doc. error=null timestamp=2024-12-01T17:09:45Z
5:09PM ERR [CacheHeaderCheck] Failed to check cache headers error="remove /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed: directory not empty" timestamp=2024-12-01T17:09:49Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af timestamp=2024-12-01T17:09:52Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/aecb22ab26b9c48eadd2daca023e7abdce9737cf17e43179b00df6d4ce89a091/ec4d709902c6b0441ad2856300479d92978e657fae737ec548564f2032f169af: no such file or directory" timestamp=2024-12-01T17:09:52Z
5:09PM ERR [GetCache] Failed to read data from the named file. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a: no such file or directory" file_Path=/interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba41e4b5a4f8829ea15ac7b02276b4a3cf06ce518599d92c343e5dbd5/42f671ed10790a6ab162fd78cb42401d413d67a76103819808746c7917d9ed6a timestamp=2024-12-01T17:09:52Z
5:09PM ERR [SearchCache] Failed to run GetCache function. error="open /interx/cache/response/c2e2368cae8d3edd8406b2c324cbfb5edd470db71204959ef00a1bd36dc939ed/5ac7d3dba4r

What would I expect:

On start the stack consumes less than ~4Gb. So let's add 2 Gb cache. Withing the limit of ~6Gb the stack should work stable without crashing and extra memory consumption.
@MrLutik MrLutik added bug Something isn't working enhancement New feature or request labels Dec 1, 2024
@MrLutik MrLutik changed the title Interx memory leak issue and resource usage optimization. Interx memory leak and resource usage optimization. Dec 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kmlbgn kmlbgn

@golnar-boosty golnar-boosty

Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kmlbgn @MrLutik @golnar-boosty