Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CPU consumption and DoS on calling get_outputs rpc #3791

Open
aglkm opened this issue Jun 5, 2024 · 4 comments
Open

CPU consumption and DoS on calling get_outputs rpc #3791

aglkm opened this issue Jun 5, 2024 · 4 comments

Comments

@aglkm
Copy link
Contributor

aglkm commented Jun 5, 2024

  1. Call get_block rpc to ensure it can produce the expected output in time.
    curl -u grin:secret --data '{"jsonrpc":"2.0","method":"get_block","params":[2824162, null, null],"id":1}' 127.0.0.1:3413/v2/foreign
  2. Call get_outputs rpc.
    curl -u grin:secret --data '{"jsonrpc":"2.0","method":"get_outputs","params":[["08b7e57c448db5ef25aa119dde2312c64d7ff1b890c416c6dda5ec73cbfed2edea"], null, null, true, true],"id":1}' 127.0.0.1:3413/v2/foreign
  3. Check CPU consumption.
  4. Call get_block rpc again and notice long time response until get_outputs finishes its job.
    curl -u grin:secret --data '{"jsonrpc":"2.0","method":"get_block","params":[2824162, null, null],"id":1}' 127.0.0.1:3413/v2/foreign
@yeastplume
Copy link
Member

yeastplume commented Jun 11, 2024

Thanks for reporting, just a quick video below of me running these commands against a docker image using a monitoring utility I've been putting together:

OliveTin.-.Personal.-.Microsoft.Edge.2024-06-11.11-07-06.mp4

As you can see, the call to get_outputs is taking a couple of seconds (which definitely could be looked into to see how to speed it up), but it doesn't seem anywhere close to DoS territory from a single request.

Is the node in the middle of syncing when you're performing this call? Also, if you could give me as much detail about the machine the node is running on, including as much detail about processor specs, ram, disk etc.

@aglkm
Copy link
Contributor Author

aglkm commented Jun 11, 2024

If you want to re-create a longer period of time waiting from the get_outputs, then send the rpc to grinnode public node API. I was able to re-create the same long waiting time against my archival node. My guess, this depends on whether a node is archival or not, causing deep searching over the outputs comparing to non archival node.

Also, even with shorter waiting time periods from get_outputs, it still causing the issue by locking(?) the db, making node struggling to sync or handle other rpc requests, e.g. get_block. So, if someone constantly sending you get_outputs one after another (not even parallel requests are needed), your node is not able to be in sync with the network or handle certain other requests, obviously this is way worse for the archival node.

Some other observations regarding the issue:

this is blocking the db:

curl -u grin:secret --data '{"jsonrpc":"2.0","method":"get_outputs","params":[["08b7e57c448db5ef25aa119dde2312c64d7ff1b890c416c6dda5ec73cbfed2edea"], null, null, true, true],"id":1}' 127.0.0.1:3413/v2/foreign

this is not blocking the db:

curl -u grin:secret --data '{"jsonrpc":"2.0","method":"get_outputs","params":[null, 1, 2833321, true, true],"id":1}' 127.0.0.1:3413/v2/foreign

@yeastplume
Copy link
Member

Okay, thank you for the extra information, I'll profile to see exactly what's going on with this call

@yeastplume
Copy link
Member

See fix in #3792, once again thanks for bringing this to our attention

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants