update protocol detection doc

[wmorgan]

• Clean up and clarify various bits of language
• Add NATS to the list of protocols that need opaque ports
• Instruct that the resource should always be set on the Service resource

In the interests of simplicity, I kept the guidance around setting the annotation at the Service resource relatively simple.

This content probably needs to be moved into a separate Reference doc. I did not do that in this doc.

[cratelyn]

🌮 lovely work @wmorgan, thank you! i left some minor notes and suggestions, but broadly, i'm very happy to see these docs gain some additional clarity.

[kflynn]

Whoa, this is a big semantic change! Is this really true?

[olix0r]

Yes, when clients discover policies for a service clusterIP and port, we need to be able to know opaqueness at the service level. If the service is a headless, the service-level annotation will be ignored (because discovery cannot be for the clusterIP), but the general advice to annotate services is correct.

[kflynn]

Specifically, the old version of this doc says that you can set config.linkerd.io/opaque-ports on either the Service or the destination pods/namespace, but this version says that you must set it in both places. That's the semantic change I'm asking about.

It makes a lot of sense to me that there are cases that you'd want to annotate the Service; it's very surprising to me if we're changing to saying that you must annotate both the Service and the destination pods/namespace.

[olix0r]

It is also required to configure the workloads (via Server or annotation) to control inbound behavior when clients are unmeshed or do not target a clusterIP service.

[wmorgan]

Basically there are situations where one or the other suffice, and thus the "must" is not strictly necessary. But rather than describing the nuances of those situations I opted to just say, you must set it in both places. If you think that's overkill I'm not committed to the choice, so feel free to suggest other language and we can iterate.