Block stream MVP

[netopyr]

Write the block stream as specified.

Unrelated, but good to be done before blockstreams

Preview Give feedback

1. Maintain SingleTransactionRecordBuilder in each SavePoint #13559
   4 of 4
   Block Streams Epic +2
   
2. Maintain each PendingRecord changes per SavePointStack #13447
   Block Streams +1
3. Add a feature flag for BlockStreams. Make the workflows capable of producing record stream/block stream/both based on the feature flag #14032
   Block Streams +1
   

StreamBuilders

Preview Give feedback

1. Implement IoBlockItemsBuilder with all TransactionInputs and TransactionOutputs #14349
   Block Streams +1
   
2. Block stream kick-off #13349
   Block Streams +1
   

   

POC Related

Preview Give feedback

1. Reactivate block stream prototype #13348
   Block Streams +1
   
2. Block stream kick-off #13349
   Block Streams +1
   

   

Protobufs

Preview Give feedback

1. Create a services branch to work off of #14031
   Block Streams +1
   
2. Work around PBJ issues so protobufs compile in services repo (e.g. not having nested proto definitions) #13897
   +0
   
3. Merge block-node hedera-protobufs branch into main

Writing Block To A File

Preview Give feedback

1. Create a block reader to read .blk files and output in a human-readable way #13921
   +0
   
2. Implement a basic block-writing facility using the given output types (StateChange, transaction outputs by service, etc.) In other words, create BlockStreamManager
3. Create FileBlockItemWriter, GrpcBlockItemWriter and a ConcurrentBlockItemWriter in the stream-manager-spike branch #14143
   Block Streams +1
   
4. Implement a BlockStream*Builder for integration in handle workflow #14121
   +0
   
5. Translate BlockItems with same TransactionID to equivalent SingleTransactionRecords #14438
   Block Streams +1
   
6. Have execute in HandleWorkflow return a wrapper with one or both streams of BlockItem/SingleTransactionRecord #14249
   Block Streams +1
7. Attach StateChanges to pending stream builders in Savepoint commit
8. Create an initial version of BlockStreamManager to produce Blocks #14247
   Block Streams +1
   
9. Implement BlockStreamManager #14354
   Block Streams +1
   
10. Write EventMetadata BlockItem's to the block stream
11. BlockItemWriter and Serial/Concurrent FileBlockItemWriter #14668
    Block Streams +1
    
12. Implement failInvalidStreamItems in HandleWorkflow for Block Streams #14919
    Block Streams +1
    

State Changes

Preview Give feedback

1. Define a first-pass API for getting state changes from platform #14138
   Block Streams +1
   
2. Create a basic sequence diagram that describes the flow of state changes for a single user transaction
3. Write a mock implementation for defined state changes API (placeholder for more tickets)
4. Implement collection of schema migrations state changes and pass said collection to the block stream writer #14506
   Block Streams +1
5. Determine areas where state is commited outside of the stack and try to use the stack.

Testing And Verification

Preview Give feedback

1. Use class from Translate BlockItems with same TransactionID to equivalent SingleTransactionRecords #14438 to verify block items
2. Create ConsensusSubmitMessageTranslator #14618
   Block Streams +1
   
3. Create UtilPrngTranslator #14619
   Block Streams +1
   
4. Create translators for contract operations #14645
   2 of 2
   Block Streams +1
   
5. Block Stream Txn Type Translators: XCreate #14926
   Block Streams +1
   
6. Create translators for token operations: TokenMint, TokenBurn, TokenWipe #14958
   Block Streams +1
   
7. Create or enhance translators for transaction record evmAddress (CryptoTransfer, EthereumCall) #14963
   Block Streams +1
   
8. Create translators for new node operations: NodeCreate, NodeUpdate, NodeDelete #14997
   Block Streams +1
   
9. Create translators for pending airdrop transactions (when possible)

Platform Dependencies

Preview Give feedback

1. API for StateChanges
2. Event metadata
3. State Proof data needed

Validators

Preview Give feedback

1. Add Validator that validates the state hash after applying state changes produces the same hash #14415
   Block Streams +1
   
2. Transfer List, Token Transfer list, etc.
3. Validator to validate block metadata and hash chain

Merge the stream-manager-spike branch to develop

Preview Give feedback

1. Open PR with STREAM_MODE in handleWorkflow with default set to RECORDS #14704
   +0
   
2. refactor: Convert RcDiff into usable class with cmd wrapper #14833
   +0
   
3. Open PR to develop with BlockItemWriter and FileBlockItemWriter #14940
   Block Streams +1
   
4. Separate the functionality around BlockStreamService for develop #14941
   Block Streams +1
   
5. Separate the functionality around StreamingTreeHasher and ConcurrentStreamingTreeHasher for develop #15027
   Block Streams +1
   
6. Separate the functionality around BlockStreamBuilder and PairedStreamBuilder for develop #15080
   Block Streams +1
   
7. Separate the functionality around BlockStreamManager for develop #15104
   Block Streams +1
   
8. Separate the functionality around state changes listeners for develop #15138
   Block Streams +1
   
9. Separate the functionality extending test-clients with block stream access for develop #15151
   +0
   
10. Separate the functionality around StateChangesValidator for develop #15147
    +0
    
11. Separate the functionality around TransactionRecordParityValidator for develop #15204
    +0
12. Separate the functionality around transaction translators for develop #15183
    Block Streams +1
    
13. Separate the functionality around BlockStreamBuilder for develop #15163
    Block Streams +1