refactor notification protocols

[turuslan]

Referenced issues

Closes #2261

Description of the Change

• peer count metric and telemetry count "/announce" streams, not connections.
• protocols are independent.
  • but "/announce" protocol controls "/grandpa" and "/transaction" protocols.
  • authority discovery controls "/validation" protocol.
• protocol error doesn't close connection to peer, only that protocol stream.
• protocols decide when to open streams.
• "/collation" and "/validation" protocols handle ViewUpdate messages, delegating other messages to observer.

Possible Drawbacks

• check peer reputation before connecting (how can peer reputation grow if there are no streams to raise reputation?)

[Lederstrumpf]

I can't yet confirm whether this PR addresses the stalling issue faced in #2261 since it OOMs too regularly (~ every 10 minutes) on this PR, which masks the stalling issue (OOM only happens every hour or so for me on current master, see #2276).

But it certainly improves consensus participation since the Kusama validator actually gets hundreds of era points now.

peer count metric and telemetry count "/announce" streams, not connections.

Note that according to telemetry, nodes on this branch all have 0 peers via this metric - I don't know whether this PR is not tallying "/announce" streams correctly, or whether that's the incorrect metric to use.

[kamilsa]

@Lederstrumpf interesting. Our node was running for more than a day with no crashes and no significance consensus stalls.

Can you please share the flags that you used to run the node?

[Lederstrumpf]

@Lederstrumpf interesting. Our node was running for more than a day with no crashes and no significance consensus stalls.

Can you please share the flags that you used to run the node?

Sure, it's the flags listed in the #2276 issue report, i.e.:

kagome --chain kusama -d [...] --validator --listen-addr [...] --public-addr [...] --name [...]  --rpc-port [...] --telemetry-url [...] --telemetry-url [...] --node-key-file [...]