update protocol detection doc

linkerd.io/content/2.16/features/protocol-detection.md

@@ -12,17 +12,17 @@ Linkerd is capable of proxying all TCP traffic, including TLS connections,
 WebSockets, and HTTP tunneling.
 
 In most cases, Linkerd can do this without configuration. To accomplish this,
-Linkerd performs *protocol detection* to determine whether traffic is HTTP or
-HTTP/2 (including gRPC). If Linkerd detects that a connection is HTTP or
-HTTP/2, Linkerd automatically provides HTTP-level metrics and routing.
-
-If Linkerd *cannot* determine that a connection is using HTTP or HTTP/2,
-Linkerd will proxy the connection as a plain TCP connection, applying
-[mTLS](../automatic-mtls/) and providing byte-level metrics as usual.
-
-(Note that HTTPS calls to or from meshed pods are treated as TCP, not as HTTP.
-Because the client initiates the TLS connection, Linkerd is not be able to
-decrypt the connection to observe the HTTP transactions.)
+Linkerd performs *protocol detection* to determine whether traffic is HTTP
+(including HTTP/2 and gRPC). If Linkerd detects that a connection is HTTP, it
+will automatically provide HTTP-level metrics and routing. If Linkerd *cannot*
+determine that a connection is using HTTP, Linkerd will proxy the connection as
+a plain TCP connection without HTTP metrics and routing. (In both cases,
+non-HTTP features such as [mutual TLS](../automatic-mtls/) and byte-level
+metrics are still applied.)
+
+Protocol detection can only happen if the HTTP traffic is unencrypted from the
+client. If the application itself initiates a TLS call, Linkerd will not be able
+to decrypt the connection, and will treat it as an opaque TCP connection.
 
 ## Configuring protocol detection
 
@@ -33,45 +33,53 @@ connections, you are likely running into a protocol detection timeout.
 This section will help you understand how to fix this.
 {{< /note >}}
 
-In some cases, Linkerd's protocol detection will time out because it doesn't see
-any bytes from the client. This situation is commonly encountered when using
-protocols where the server sends data before the client does (such as SMTP) or
-protocols that proactively establish connections without sending data (such as
-Memcache). In this case, the connection will proceed as a TCP connection after a
-10-second protocol detection delay.
+To do protocol detection, Linkerd waits for up to 10 seconds to see bytes sent
+from the client. Note that until the protocol has been determined, Linkerd
+cannot even establish a connection to the destination, since HTTP routing
+configuration may inform where this connection is established to.
+
+If Linkerd does not see enough data from the client within 10 seconds from
+connection establishment to determine the protocol, Linkerd will treat the
+connection as an opaque TCP connection and will proceed as normal, establishing
+the connection to the destination and proxying the data.
 
-To avoid this delay, you will need to provide some configuration for Linkerd.
-There are two basic mechanisms for configuring protocol detection: _opaque
-ports_ and _skip ports_:
+In practice, protocol detection timeouts typically happen when the application
+is using a protocol where the server sends data before the client does (such as
+SMTP) or a protocol that proactively establish connections without sending data
+(such as Memcache). In this case, everything will work, but Linkerd will
+introduce an unnecessary 10 second delay before connection establishment.
+
+To avoid this delay, you can provide some configuration for Linkerd. There are
+two basic mechanisms for configuring protocol detection: _opaque ports_ and
+_skip ports_:
 
 * Opaque ports instruct Linkerd to skip protocol detection and proxy the
  connection as a TCP stream
 * Skip ports bypass the proxy entirely.
 
-Opaque ports are generally preferred as they allow Linkerd to provide mTLS,
-TCP-level metrics, policy, etc. Skip ports circumvent Linkerd's ability to
-provide security features.
+Opaque ports are generally preferred as they allow Linkerd to still provide
+mTLS, TCP-level metrics, policy, etc. Skip ports, by contrast, create networking
+rules that avoid the proxy entirely, circumventing Linkerd's ability to provide
+security features.
 
 Linkerd maintains a default list of opaque ports that corresponds to the
 standard ports used by protocols that interact poorly with protocol detection.
-As of the 2.12 release, that list is: **25** (SMTP), **587** (SMTP), **3306**
-(MySQL), **4444** (Galera), **5432** (Postgres), **6379** (Redis), **9300**
-(ElasticSearch), and **11211** (Memcache).
 
 ## Protocols that may require configuration
 
 The following table contains common protocols that may require additional
 configuration.
 
-| Protocol | Standard port(s) | In default list? | Notes |
+| Protocol | Standard ports  | In default list? | Notes |
 |-----------------|------------------|------------------|-------|
 | SMTP | 25, 587 | Yes | |
 | MySQL | 3306 | Yes | |
-| MySQL with Galera | 3306, 4444, 4567, 4568 | Partially | Ports 4567 and 4568 are not in Linkerd's default set of opaque ports |
+| MySQL with Galera | 3306, 4444, 4567, 4568 | Partially | Ports 4567 and 4568 are not in Linkerd's default list of opaque ports |
 | PostgreSQL | 5432 | Yes | |
 | Redis | 6379 | Yes | |
 | ElasticSearch | 9300 | Yes | |
 | Memcache | 11211 | Yes | |
+| NATS | 4222, 6222, 8222 | No | |
 
 If you are using one of those protocols, follow this decision tree to determine
 which configuration you need to apply.
@@ -81,27 +89,32 @@ which configuration you need to apply.
 ## Marking ports as opaque
 
 You can use the `config.linkerd.io/opaque-ports` annotation to mark a port as
-opaque. Note that this annotation should be set on the _destination_, not on the
-source, of the traffic.
+opaque. Linkerd will skip protocol detection on opaque ports, and treat
+connections to them as TCP streams.
 
-This annotation can be set in a variety of ways:
+This annotation should be set on the _destination_, not on the source, of the
+traffic. This is true even if the destination is unmeshed, as it controls the
+behavior of meshed clients.
 
-1. On the workload itself, e.g. on the Deployment's Pod spec receiving the traffic.
-1. On the Service receiving the traffic.
-1. On a namespace (in which it will apply to all workloads in the namespace).
-1. In an [authorization policy](../server-policy/) `Server` object's
- `proxyProtocol` field, in which case it will apply to all pods targeted by that
- `Server`.
+This annotation *must* be set in two places:
 
-When set, Linkerd will skip protocol detection both on the client side and on
-the server side. Note that since this annotation informs the behavior of meshed
-_clients_, it can be applied to unmeshed workloads as well as meshed ones.
+1. On the Service receiving the traffic.
+2. On the workload itself (e.g. on the Deployment's Pod spec receiving the
+traffic), or on enclosing namespace, in which it will apply to all workloads in
+the namespace.
 
 {{< note >}}
 Multiple ports can be provided as a comma-delimited string. The values you
 provide will _replace_, not augment, the default list of opaque ports.
 {{< /note >}}
 
+{{< note >}}
+If you are using [authorization policies](../server-policy/), the `Server`'s
+`proxyProtocol` field which can be used to control protocol detection behavior
+and can be used instead of a Service annotation. Regardless, we suggest
+annotating the Service object for clarity.
+{{< /note >}}
+
 ## Marking ports as skip ports
 
 Sometimes it is necessary to bypass the proxy altogether. In this case, you can
@@ -128,12 +141,6 @@ Note that the default set of opaque ports can be configured at install
 time, e.g. by using `--set proxy.opaquePorts`. This may be helpful in
 conjunction with `enable-external-profiles`.
 
-{{< note >}}
-There was a bug in Linkerd 2.11.0 and 2.11.1 that prevented the opaque ports
-behavior of `enable-external-profiles` from working. This was fixed in Linkerd
-2.11.2.
-{{< /note >}}
-
 ## Using `NetworkPolicy` resources with opaque ports
 
 When a service has a port marked as opaque, any `NetworkPolicy` resources that