Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support substreams V2 #14

Open
0237h opened this issue May 12, 2023 · 0 comments
Open

Support substreams V2 #14

0237h opened this issue May 12, 2023 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@0237h
Copy link
Collaborator

0237h commented May 12, 2023

With the new addition of substreams V2, a new gRPC service is required to use it.

The .proto definitions can be found here: https://github.com/streamingfast/substreams/tree/develop/proto/sf/substreams/rpc/v2

Problem

Right now the configuration tool will crash when attempting to create a stub for a V2-enabled endpoint (sf.substreams.rpc.v2.Stream):

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/user/Documents/substreams_firehose/substreams_firehose/config/__main__.py", line 50, in <module>
    main()
  File "/home/user/Documents/substreams_firehose/substreams_firehose/config/__main__.py", line 47, in main
    return ConfigApp().run()
           ^^^^^^^^^^^^^^^^^
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/apNPSApplication.py", line 30, in run
    return npyssafewrapper.wrapper(self.__remove_argument_call_main)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/npyssafewrapper.py", line 41, in wrapper
    wrapper_no_fork(call_function)
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/npyssafewrapper.py", line 97, in wrapper_no_fork
    return_code = call_function(_SCREEN)    
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/apNPSApplication.py", line 25, in __remove_argument_call_main
    return self.main()
           ^^^^^^^^^^^
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/apNPSApplicationManaged.py", line 172, in main
    self._THISFORM.edit()
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/fm_form_edit_loop.py", line 47, in edit
    self.edit_loop()
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/fm_form_edit_loop.py", line 38, in edit_loop
    self._widgets__[self.editw].edit()
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/wgwidget.py", line 458, in edit
    self._edit_loop()
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/wgwidget.py", line 474, in _edit_loop
    self.get_and_use_key_press()
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/wgwidget.py", line 610, in get_and_use_key_press
    self.handle_input(ch)
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/wgwidget.py", line 71, in handle_input
    self.handlers[_input](_input)
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/wgbutton.py", line 99, in h_toggle
    self.whenPressed()
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/fmActionFormV2.py", line 10, in whenPressed
    return self.parent._on_ok()
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/Documents/substreams_firehose/.venv/lib/python3.11/site-packages/npyscreen/fmActionFormV2.py", line 51, in _on_ok
    self.editing = self.on_ok()
                   ^^^^^^^^^^^^
  File "/home/user/Documents/substreams_firehose/substreams_firehose/config/ui/forms/stub_config_edit.py", line 220, in on_ok
    StubConfig.SUBSTREAMS_PACKAGE_OBJECT = Config.PROTO_MESSAGES_CLASSES[
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'sf.substreams.rpc.v2.Package'

This is due to the fact that the V2 still uses sf.substreams.v1 proto files. The tool is looking for those definitions in the sf.substreams.rpc.v2 folder since it's the prefix of the gRPC service.

Possible solutions

Differentiate service prefix with the actual .proto folder implementation

This will require an additional field in the stub configs to look for those definitions (not ideal).

Search for alternative implementations in case of KeyError

In the case above, one can look for other Package implementation under different packages. If no other implementations are found, throw. This can be tricky in case of duplicate or multiple implementations having the same name in different packages

Make special case for substreams V2

Detect the sf.substreams.rpc.v2 prefix and replace it with the sf.substreams.v1 when searching for proto classes (can be on KeyError only).

@0237h 0237h added the bug Something isn't working label May 12, 2023
@0237h 0237h self-assigned this May 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant