Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cuda prover error: "InCompleteMessage" when generating proof #167

Open
lightning-li opened this issue Oct 15, 2024 · 0 comments
Open

Cuda prover error: "InCompleteMessage" when generating proof #167

lightning-li opened this issue Oct 15, 2024 · 0 comments

Comments

@lightning-li
Copy link

  • The ec2 instance: G6.2XLARGE L4 24GB
  • The op-succint commit id: 6452ee623dc5463f8de70009f198ee2b6002beeb
  • The command: RUST_LOG=info just run-multi 18570000 18570002 false true
  • The error log:
thread 'main' panicked at /root/ryan/sp1/crates/cuda/src/lib.rs:219:82:
called `Result::unwrap()` on an `Err` value: ReqwestError(reqwest::Error { kind: Request, url: "http://localhost:3000/twirp/api.ProverService/ProveCore", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: Recipe `run-multi` failed with exit code 101
  • Before the error log, there're normal logs:
.......
2024-10-15T15:36:57.123576Z  INFO prove_core:clone: close time.busy=940µs time.idle=7.37µs
2024-10-15T15:36:57.215280Z  INFO prove_core:execute: close time.busy=91.6ms time.idle=2.27µs
2024-10-15T15:36:57.220563Z  INFO prove_core:create memory checkpoint: close time.busy=5.24ms time.idle=3.56µs
2024-10-15T15:36:57.450347Z  INFO prove_core:clone: close time.busy=924µs time.idle=8.13µs
2024-10-15T15:36:57.550878Z  INFO prove_core:execute: close time.busy=100ms time.idle=4.09µs
2024-10-15T15:36:57.554019Z  INFO prove_core:create memory checkpoint: close time.busy=3.09ms time.idle=3.67µs
2024-10-15T15:36:57.858283Z  INFO prove_core:clone: close time.busy=671µs time.idle=6.06µs
2024-10-15T15:36:57.932027Z  INFO prove_core:execute: └╴54,655,801 cycles
2024-10-15T15:36:57.932114Z  INFO prove_core:execute: ┌╴output-root
2024-10-15T15:36:57.945074Z  INFO prove_core:execute: └╴83,347 cycles
stdout: Validated derivation and STF. Output Root: 0xd437b81415e7eb94191bb9ef1535f3630ea4528c0678cf8a30df77ec596e8da6
2024-10-15T15:36:57.978004Z  INFO prove_core:execute: close time.busy=120ms time.idle=2.96µs
2024-10-15T15:36:58.184846Z  INFO prove_core:create memory checkpoint: close time.busy=207ms time.idle=4.41µs
2024-10-15T15:37:20.011567Z  INFO prove_core: ┌╴execution-instantiation
2024-10-15T15:37:20.013139Z  INFO prove_core: └╴5,068 cycles
2024-10-15T15:37:20.013272Z  INFO prove_core: ┌╴block-execution
2024-10-15T15:37:23.930891Z  INFO prove_core: ┌╴payload-derivation
2024-10-15T15:37:24.728756Z  INFO prove_core: ┌╴block-execution
2024-10-15T15:37:43.447528Z  INFO prove_core: ┌╴output-root
2024-10-15T15:37:43.488556Z  INFO prove_core: └╴83,347 cycles

It seems the error comes from the underlying network library, any ideas to fix this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant