hub->rollapp packets are not relayed by the flush mechanism

[danwt]

Overview

The relayer in general works in 2 ways:

1. event based - listens to events and by that knows how to relay messages
2. query based - evert flush interval queries both chains to see if any packets needs handling which were were missed by the event based processor

Problem:

Relayer flush doesn't work for packets sent from hub to rollapp if the rollapp has a long block time (e.g 1hr).
Part of the flush sequence is updating the chain's ibc light client. However, when trying to update the rollapp ibc client on the hub, and assuming no block took place in the last hour, the relayer tries to update it with the same height it's most likely have. This in turn fails (as you must have height + 1 to update ibc light client), which prevents from the pending packets to be relayed.

Example update client error (part of it):
Error sending messages <truncate-for-brevity> consensus height is greater than or equal to the current block height (1-5 >= 1-5): invalid height [cosmos/ibc-go/[email protected]/modules/core/03-connection/keeper/handshake.go:169] With gas wanted: '18446744073709551615' and gas used: '142375' : unknown request"}

Replicate:

1. Scenario 1

start a hub + rollapp with 1 hr rollapp blocktime
do all preconditions needed to be able to do a ibc transfer hub->rollapp 
kill relayer
do a ibc transfer hub -> rollapp
wait for event on hub to go out of date, so relayer will not see it
start relayer
# relayer will not be able to relay packet to RA, and ack to HUB, due to ??

What we want:

Make sure we can always relay even with low activity chains and when relayers come and go and might miss events

Solution ideas:

TODO

### Requirements

• Don't want busy loop unnecessarily
• Require as little as possible cost

Related issues:

• https://github.com/dymensionxyz/research/issues/173
• pull from upstream v2.5.1 #9
• Make the MaxClockDriftconfigurable #1
• Support on-demand block production cosmos/relayer#1332
• relayer doesn't transfer from hub to rollapp until new rollapp block is created #4
• 8b8e582 - our patch

Tools needed:

• Use relayer 2.5.2 at least https://github.com/cosmos/relayer/releases/tag/v2.5.2, will need to downstream our fork