[Deliverable] Enable testing of direct messages in unreliable networks

[chair28980]

Project board: https://github.com/orgs/waku-org/projects/25/views/1

Epics

• [Epic: Status] Enable testing of direct messages in unreliable networks go-waku#1234
• [Epic: QA] Enable testing of direct messages in unreliable networks #214

Output

• https://github.com/status-im/status-cli-tests
• https://status-im.github.io/status-cli-tests/

[plopezlpz]

Testing under unreliable networks of the following messages is running nightly here:

• 1:1 chat
• contact request/response
• community join request/response
• private group

Repo: https://github.com/status-im/status-cli-tests

[fryorcraken]

@plopezlpz Are we done with this?
I believe Vac/QA had some more test re suspend.
Note that communities on custom shards is tracked separately.

If Vac/QA is done then we can probably close this. @jm-clius @fbarbu15 please help clarify.

[fbarbu15]

@fryorcraken I talked with @plopezlpz and I'll finish the suspend tests this week.
Please let me know if there's any other scenarios that needs addressing as part of this ticket.
If not, let's open a new ticket for the remaining work (like communities on custom shards)
Thanks

[plopezlpz]

From this only missing is the pause use case: status-im/status-go#5531
For remaining I agree on opening a new ticket

@fryorcraken @fbarbu15

[fbarbu15]

Upon reviewing the QA tasks from the Waku Roadmap, the following items remain incomplete:

1. Direct Message Reliability

   • Review the current test coverage of chat functionalities in status-go and proceed with further coverage.
   • Review testing in status-go related to chat functionalities that rely on external systems (e.g., Waku fleet) and migrate them to an interoperable testing framework.

2. E2E reliability protocol

   • Review test coverage of multicast e2e reliability PoC
   • Upgrade test cases to ensure multicast e2e reliability protocol is tested with status-go CLI
   • Expand coverage of Status Communities features

3. Static Sharding - dedicated shards

   • Ensure coverage of existing scenarios with sharding, including focus on connection/disconnection and discovery.
   • Add coverage of community creation and message sending on dedicated shards, including opt-in message signing DoS protection.
   • Add coverage of shard discovery.

Regarding the Direct Message Reliability tasks, it appears there are two outstanding items that should be included in this milestone. Can we confirm if these tasks are still required from the QA team? And if yes, can we have some epics describing what needs to be done in a little more detail?

For the other milestones:

• E2E Reliability Protocol
• Static Sharding - Dedicated Shards

Who is responsible for creating the new GitHub tickets?
@fryorcraken @plopezlpz

[chair28980]

@fbarbu15 anyone can create requisite issues, I'm responsible for keeping things labeled and ensuring the parent/child issues are tracked appropriately.

These are the MIlestones we have for the above mentioned items:

• Direct Message Reliability: https://github.com/waku-org/pm/milestone/28
• End-to-end reliability protocol: https://github.com/waku-org/pm/milestone/29
• Static Sharding - dedicated shards: https://github.com/waku-org/pm/milestone/30
• all Milestones: https://github.com/waku-org/pm/milestones

[fbarbu15]

Thanks, I was referring more about who can help define the QA requirements in dedicated tickets, similar with what we had for Direct Message Reliability QA Task status-im/status-go#5144

[fbarbu15]

Hybernate tests are done: status-im/status-go#5531

[fryorcraken]

@fbarbu15 any preferences of were you'd like to see those epics created?

Review the current test coverage of chat functionalities in status-go and proceed with further coverage.

This item was included because of:

3. Static Sharding - dedicated shards
   Add coverage of community creation and message sending on dedicated shards, including opt-in message signing DoS protection.

The static sharding item seems necessary because Status-QA has reported issues with communities on dedicated shards ("everything break").

The role of this present item is to have a good base in terms of testing so that when tackling coverage for dedicated sharding, you don't start from scratch.

I expect here to review https://rfc.vac.dev/status/55/1to1-chat and https://rfc.vac.dev/status/56/communities and ensure that coverage is appropriate.

@plopezlpz @kaichaosun @cammellos @richard-ramos what are your thoughts on this matter?

Review testing in status-go related to chat functionalities that rely on external systems (e.g., Waku fleet) and migrate them to an interoperable testing framework.

My understanding is that a number of tests as part of status-go CI are executed against live fleet, leading to regular failure during the CI.
This was mention during a status test meeting at Montenegro. @cammellos and @igor-sirotin seemed to be the most vocal about it. (cc @iurimatias @jrainville).

The intent here was to spend dedicated time to stabilize status-go CI to help with development in general.
It is my opinion that CI should be self-contained to avoid failure due to reasons other than the PR changes, hence a proposal to move the test against external fleets to another framework, which can then be run nightly.

However, I have no development on status-go so best if I get some input from people tagged above.

[fbarbu15]

@fbarbu15 any preferences of were you'd like to see those epics created?

In github, similar with status-im/status-go#5144 works fine for me. But I'm open to other ways as well

[igor-sirotin]

My understanding is that a number of tests as part of status-go CI are executed against live fleet, leading to regular failure during the CI

Yes. I don't think there're many of them though.
Most messaging tests (protocol package) share a single Waku V1 node for several Messenger. These tests are not affected by the state of real fleets.

Here's a list of tests that I think use real fleet:

• TestRestartDiscoveryV5
• TestRendezvousDiscovery
• TestWakuV2Filter
• some more tests in wakuv2/waku_test.go

It is my opinion that CI should be self-contained to avoid failure due to reasons other than the PR changes

👍 👍 ❤️

[igor-sirotin]

Just found another test that connects to the outer world:

• TestPeerCount

[fryorcraken]

@plopezlpz Happy to close this?