Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include error-message as part of the GET/LIST endpoints to help with troubleshooting #396

Closed
sabbyanandan opened this issue Aug 21, 2023 · 9 comments · Fixed by #397
Closed
Labels
enhancement New feature or request released

Comments

@sabbyanandan
Copy link

As a user, when the actively observing chainhooks fail to trigger either because of the errors in Chainhook itself or due to the issues when triggering the HTTP endpoint for a given event, I would like to know what and why it failed, so I can troubleshoot and course-correct.

The interruptions could be:
a: Errors in Chainhook
b: Having an incorrect HTTP endpoint as a trigger
c: Having an unresponsive or erroring HTTP endpoint as a trigger
d: Any runtime errors between the Chainhook node, Chainhook API, or HTTP endpoints

If we can include the error-message and can define a category for those errors, the Hiro Platform can handle them and gracefully let users know of the problem.

@MicaiahReid
Copy link
Contributor

@lgalabru I'm wondering if we can store any errors that occur in checking or triggering a predicate and return them with PredicateStatus

@MicaiahReid
Copy link
Contributor

Upon further investigation, we already store and return errors when evaluating a predicate:

let status = PredicateStatus::Interrupted(format!(

@BLuEScioN / @sabbyanandan - whenever the predicate status is "Interrupted", it will include an error. Do you think that will be sufficient for the platform to display more info to the user?

@sabbyanandan
Copy link
Author

Woah! Awesome, I think this should suffice. Now, if we are able to trigger those errors, we can see how much of the information is floating through the pipes, and how we may want to massage them for the end-users. As far as the "caused by" or equivalent is clearer, that is all it matters to the end users.

@smcclellan
Copy link
Contributor

smcclellan commented Aug 23, 2023

@sabbyanandan If I'm following here, this issue is a no-op since we already surface the errors? Can it be closed?

@sabbyanandan
Copy link
Author

@smcclellan: Before you close it, perhaps you can verify the functionality and share the sample JSON responses with the error messages. That way @BLuEScioN can review the JSON and explore how to further handle it gracefully for the end-users with the help of @andromeda995.

cc: @andresgalante

@MicaiahReid
Copy link
Contributor

I've added some additional errors in #397

@MicaiahReid
Copy link
Contributor

Here are the possible errors:

  • Unable to evaluate predicate on Stacks chainstate: {e}. e will have the following options:
    • Chainhook specification must include fields 'start_block' when using the scan command
    • Unable to retrieve block {current_block_height}
    • Unable to retrieve block {current_block_height}: {e}
    • Scan aborted (consecutive action errors >= 3): {e}. e will have some further reasons on why:
      • unable to build http client: {e}
      • unable to serialize payload {e}
      • unable to send request after several retries. most recent error: Trigger {url} failed with status {http_post_status}
      • unable to create file {file_name}: {e}
      • unable to retrieve current_dir: {e}
      • unable serialize bytes as utf8 string {e}
      • unable to open file {e}
  • Unable to evaluate predicate on Bitcoin chainstate: {e}. e will have the following options:
    • Bitcoin RPC error: {e}
    • Bitcoin chainhook specification must include a field start_block in replay mode
    • unable to retrieve Bitcoin chain tip ({e})
    • Scan aborted (consecutive action errors >= 3): {e}. e will have some further reasons on why:
      • unable to build http client: {e}
      • unable to serialize payload {e}
      • unable to send request after several retries. most recent error: Trigger {url} failed with status {http_post_status}
      • unable to create file {file_name}: {e}
      • unable to retrieve current_dir: {e}
      • unable serialize bytes as utf8 string {e}
      • unable to open file {e}

@sabbyanandan
Copy link
Author

@MicaiahReid: Thanks.

@LakshmiLavanyaKasturi: Something to summarize in the Chainhook docs, perhaps under the error handling guide or similar.

@smcclellan smcclellan moved this from 🆕 New to 👀 In Review in DevTools Sep 5, 2023
MicaiahReid added a commit that referenced this issue Sep 14, 2023
### Description

This PR accomplishes a few things:
- renames/adds/removes predicate statuses according to the flowchart in
docs/images/predicate-status-flowchart/PredicateStatusFlowchart.png
 - adds more context to errors returned in Interrupted status
 - adds status data to `GET /v1/chainhooks` endpoint
- fixes bug in chainhook service where block streaming continued past
`end_block`

Fixes #396, fixes #324 

Also: 
Improves how we handle a restart of `chainhook service` while predicates
are scanning/streaming. Here are the cases we now handle:
1. Predicates that were in `scanning` status when Chainhook was
terminated will resume scanning starting from their
`last_evaluated_block_height`. *Note: because we only save predicate
status every 10 scans, we could end up re-emiting matches on a resetart*
2. Predicates that were in `new` status when Chainhook was terminated
will start scanning at the predicate's `start_block`
3. Predicates that were in `streaming` status will _return_ to a
`scanning` status, starting at `last_evaluated_block_height` to catch up
on the missed blocks. Note, the `number_of_blocks_to_scan` is set to 0
for this temporary catch-up, as it's difficult to compute the number of
remaining blocks in the context of this change
4. If predicates were passed in at startup, we also register those to
begin scanning, which previously didn't happen
5. We now allow passing in a predicate at startup _and_ registering
additional predicates with the predicate registration server. This means
that if you use the same startup predicate repeatedly, it will already
be saved in redis and _not_ be reloaded.

Fixes: #298, fixes #390, fixes #402, fixes #403
#### Breaking change?

The rename of `ScanningData`'s `number_of_blocks_sent` field could
technically be considered breaking, let's discuss.


### Checklist

- [x] All tests pass
- [x] Tests added in this PR (if applicable)
Test coverage before: 23.2%
Test coverage after: 37.72%
@github-project-automation github-project-automation bot moved this from 👀 In Review to ✅ Done in DevTools Sep 14, 2023
github-actions bot pushed a commit that referenced this issue Oct 10, 2023
## [1.1.0](v1.0.0...v1.1.0) (2023-10-10)

### Features

* allow matching with regex for stacks print_event ([#380](#380)) ([131809e](131809e)), closes [#348](#348)
* augment predicate status returned by GET/LIST endpoints ([#397](#397)) ([a100263](a100263)), closes [#396](#396) [#324](#324) [#390](#390) [#402](#402) [#403](#403)
* introduce "data_handler_tx" ([ee486f3](ee486f3))

### Bug Fixes

* build error ([85d4d91](85d4d91))
* build errors ([b9ff1aa](b9ff1aa))
* build errro ([be0c229](be0c229))
* bump retries and delays ([aff3690](aff3690))
* chainhook not being registered ([5a809c6](5a809c6))
* ensure that the parent block was previously received. else, fetch it ([2755266](2755266))
* migrate to finer zmq lib ([4eb5a07](4eb5a07))
* prevent panic when scanning from genesis block ([#408](#408)) ([1868a06](1868a06))
* remove event_handlers ([6fecfd2](6fecfd2))
* retrieve blocks until tip ([5213f5f](5213f5f))
* revisit approach ([67a34dc](67a34dc))
* use crossbeam channels ([ea33553](ea33553))
* workflow ([d434c93](d434c93))
@github-actions
Copy link

🎉 This issue has been resolved in version 1.1.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

MicaiahReid pushed a commit that referenced this issue Oct 10, 2023
## [1.1.0](v1.0.0...v1.1.0) (2023-10-10)

### Features

* allow matching with regex for stacks print_event ([#380](#380)) ([131809e](131809e)), closes [#348](#348)
* augment predicate status returned by GET/LIST endpoints ([#397](#397)) ([a100263](a100263)), closes [#396](#396) [#324](#324) [#390](#390) [#402](#402) [#403](#403)
* introduce "data_handler_tx" ([ee486f3](ee486f3))

### Bug Fixes

* build error ([85d4d91](85d4d91))
* build errors ([b9ff1aa](b9ff1aa))
* build errro ([be0c229](be0c229))
* bump retries and delays ([aff3690](aff3690))
* chainhook not being registered ([5a809c6](5a809c6))
* ensure that the parent block was previously received. else, fetch it ([2755266](2755266))
* migrate to finer zmq lib ([4eb5a07](4eb5a07))
* prevent panic when scanning from genesis block ([#408](#408)) ([1868a06](1868a06))
* remove event_handlers ([6fecfd2](6fecfd2))
* retrieve blocks until tip ([5213f5f](5213f5f))
* revisit approach ([67a34dc](67a34dc))
* use crossbeam channels ([ea33553](ea33553))
* workflow ([d434c93](d434c93))
vabanaerytk added a commit to vabanaerytk/chainhook that referenced this issue Aug 7, 2024
## [1.1.0](hirosystems/chainhook@v1.0.0...v1.1.0) (2023-10-10)

### Features

* allow matching with regex for stacks print_event ([#380](hirosystems/chainhook#380)) ([af0fcad](hirosystems/chainhook@af0fcad)), closes [#348](hirosystems/chainhook#348)
* augment predicate status returned by GET/LIST endpoints ([#397](hirosystems/chainhook#397)) ([0f32704](hirosystems/chainhook@0f32704)), closes [#396](hirosystems/chainhook#396) [#324](hirosystems/chainhook#324) [#390](hirosystems/chainhook#390) [#402](hirosystems/chainhook#402) [#403](hirosystems/chainhook#403)
* introduce "data_handler_tx" ([a7704d8](hirosystems/chainhook@a7704d8))

### Bug Fixes

* build error ([9d47f87](hirosystems/chainhook@9d47f87))
* build errors ([27b0ae4](hirosystems/chainhook@27b0ae4))
* build errro ([6363fad](hirosystems/chainhook@6363fad))
* bump retries and delays ([483663c](hirosystems/chainhook@483663c))
* chainhook not being registered ([7d2b156](hirosystems/chainhook@7d2b156))
* ensure that the parent block was previously received. else, fetch it ([79561c8](hirosystems/chainhook@79561c8))
* migrate to finer zmq lib ([2e55a20](hirosystems/chainhook@2e55a20))
* prevent panic when scanning from genesis block ([#408](hirosystems/chainhook#408)) ([4ab8dd8](hirosystems/chainhook@4ab8dd8))
* remove event_handlers ([e3664d2](hirosystems/chainhook@e3664d2))
* retrieve blocks until tip ([d21cacc](hirosystems/chainhook@d21cacc))
* revisit approach ([47d2ef6](hirosystems/chainhook@47d2ef6))
* use crossbeam channels ([50595ab](hirosystems/chainhook@50595ab))
* workflow ([84c4ec9](hirosystems/chainhook@84c4ec9))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request released
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants