Skip to content

Commit

Permalink
Sei remote sign hotfix: aggregate tcp chunks before unmarshal proto
Browse files Browse the repository at this point in the history
```go
// tendermint/cometbft proposal:
type Proposal struct {
	Type      SignedMsgType
	Height    int64
	Round     int32
	PolRound  int32
	BlockID   BlockID
	Timestamp time.Time
	Signature []byte
}
```

```go
// vs sei-tendermint proposal
type Proposal struct {
	Type            SignedMsgType
	Height          int64
	Round           int32
	PolRound        int32
	BlockID         BlockID
	Timestamp       time.Time
	Signature       []byte

        // this is a list, and can be very long...
	TxKeys          []*TxKey
	Evidence        *EvidenceList
	LastCommit      *Commit
	Header          Header
	ProposerAddress []byte
}
```

Since Proposal has TxKeys and other lists, Proposal has variable length
It is easily goes > 1024 bytes if block has big mount of txs. And it
is not a problem of canonical tendermint/cometbft implementations
since due to its message structure, it has a fixed max length < 1024 (DATA_MAX_SIZE)

sei-tendermint, when it connects to remote signer over tcp, sends
proposal divided by chunk of DATA_MAX_SIZE (1024) each, which kind of
fits the expectation of tmkms. However, tmkms never tries to
aggregate chunks. In fact, it is impossible for tmkms to implement
aggregation properly without knowing the length beforehand: which is
not provided by tendermint protocol.

There might be a confusioon also, because all implementations of
tendermint send lenght-delimited protobufs, and tmkms also reads with
a function "length delimited". However, it actually means that the
protobuf msg is prepended by it's length: so that when tmkms reads
1024 bytes it knows which zeroes are payload and which a need to be
cut. Another words, it has nothing to do with multi-chunk payload.

Which means that sei-tendermint just doesn't bother about tcp
remote signer, and it is impossible to make it work with tmkms without
rewriting both and adding this custom protocol of "aggregate chunks until
you get full message length".

--
This code implements aggregation by trying to unmarshal aggregated
message each time it gets a new chunk. I don't think it is a good idea
in a long run, however, the alternative would be to adjust both Sei
and tmkms, rolling out new length-aware protocol between them -- I'm
not sure how sufficient it is and definitely needs a
discussion. Current solution is compartable with both
cometbft/tendermint and sei-tendermint, however, way less efficient then
the original `read` implementation of tmkms.

P.S: Apart from custom length-aware protocol, there is another option:
implement grpc in tmkms, which seem to be supported by sei-tendermint.
  • Loading branch information
zarkone committed Apr 25, 2024
1 parent c8b7811 commit 5ab22bb
Showing 1 changed file with 31 additions and 10 deletions.
41 changes: 31 additions & 10 deletions src/rpc.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,9 @@ use crate::privval::SignableMsg;
use prost::Message as _;
use std::io::Read;
use tendermint::{chain, Proposal, Vote};
use tendermint_p2p::secret_connection::DATA_MAX_SIZE;
use tendermint_proto as proto;

// TODO(tarcieri): use `tendermint_p2p::secret_connection::DATA_MAX_SIZE`
// See informalsystems/tendermint-rs#1356
const DATA_MAX_SIZE: usize = 262144;

use crate::{
error::{Error, ErrorKind},
prelude::*,
Expand All @@ -31,12 +28,36 @@ pub enum Request {
impl Request {
/// Read a request from the given readable.
pub fn read(conn: &mut impl Read, expected_chain_id: &chain::Id) -> Result<Self, Error> {
let msg_bytes = read_msg(conn)?;

// Parse Protobuf-encoded request message
let msg = proto::privval::Message::decode_length_delimited(msg_bytes.as_ref())
.map_err(|e| format_err!(ErrorKind::ProtocolError, "malformed message packet: {}", e))?
.sum;
let mut msg_bytes: Vec<u8> = vec![];
let msg;

// fix for Sei: collect incoming bytes of Protobuf from incoming msg
loop {
let mut msg_chunk = read_msg(conn)?;
let chunk_len = msg_chunk.len();
msg_bytes.append(&mut msg_chunk);

// if we can decode it, great, break the loop
match proto::privval::Message::decode_length_delimited(msg_bytes.as_ref()) {
Ok(m) => {
msg = m.sum;
break;
}
Err(e) => {
// if chunk_len < DATA_MAX_SIZE (1024) we assume it was the end of the message and it is malformed
if chunk_len < DATA_MAX_SIZE {
return Err(format_err!(
ErrorKind::ProtocolError,
"malformed message packet: {}",
e
)
.into());
}
// otherwise, we go to start of the loop assuming next chunk(s)
// will fill the message
}
}
}

let (req, chain_id) = match msg {
Some(proto::privval::message::Sum::SignVoteRequest(
Expand Down

0 comments on commit 5ab22bb

Please sign in to comment.