Have multiple output options after processing blocks

[0237h]

Problem

Currently, block processors holds the job of manipulating the data coming out of a gRPC stream, but the results are always set to be stored in a file (JSON formatted or not).

Proposition

It would be nice from the CLI as well as from a developer API perspective to be able to serve it to different mediums (a more general IO stream could be a start).

A non-exhaustive list of possible output formats:

• Files
  • JSONL
  • CSV
  • Quickbooks (?)
• Python
  • Pandas dataframe
  • Message queues
  • Collections

Note: the goal is not to simply pass the data stream to other advanced instances (like databases) as it would defeat the purpose of using the appropriate sinks for Firehose/Substreams. Rather, just provide common output formats for the end user.

PS: Interesting library to look for with the same sinks concepts design: Apache Flink