Initial draft for demonstrating how to validate Pubsub via canary

daprdocs/content/en/operations/troubleshooting/common_issues.md

@@ -273,3 +273,96 @@ kubectl config get-users
 ```
 
 You may learn more about webhooks [here](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/).
+
+## PubSub messages are not delivered to subscribers
+
+There are several reasons why a published message may not arrive at a subscriber. 
+
+Firstly, always check the dapr logs when dapr is initializing, most common infrastructure problems will appear during the initialisation phase, so it is important to check the logs for errors and warnings. 
+
+If there are no errors during the initialization phase, the **absence** of a log message similar to `app is subscribed to the following topics: [..]` is an important hint that there is a problem, as no subscribers are established to receive any published messages.
+
+If the PubSub component is newly provisioned (or you are first-time user of Dapr PubSub), it is recommended that the first step to take in diagnosing a delivery issue is to validate the PubSub component and its infrastructure is correctly configured.
+
+This can be easily done by publishing a Canary message from one Dapr sidecar to another Dapr sidecar via a PubSub Topic. 
+
+The Canary message will validate a happy-path exists between the sidecars. The happy-path is isolated from non-infrastructure factors which may cause a PubSub message to not deliver, i.e a serialization mismatch on the publisher App or subscriber App.
+
+Sending a canary can be as simple as a 3-step process as detailed below, but full instructions and options on sending Canary messages can be found here [Dapr Samples - PubSub Canary](https://github.com/dapr/samples) repository.
+
+### 3-step process for validating self-hosted dapr sidecars
+
+Given you have a PubSub component yaml in a folder called `components` and the pubsub component is named `my-pubsub`
+
+1. Run a dapr sidecar - This is the **Publisher**
+
+    ```bash
+    dapr run --resources-path components --dapr-http-port 3500
+    ```
+
+2. Run a second dapr sidecar - This is the **Subscriber**
+
+    For your convenience, a CLI argument is provided to automatically register a Canary subscriber against a given Topic : 
+
+    `--canary-subscriber {pubsub-name}{topic-name}`
+
+    ```bash
+    dapr run --resources-path components --canary-subscriber my-pubsub:my-topic
+    ```
+
+    The Dapr **Subscriber** will output a log message validating the Canary subcription 
+
+    ```bash
+    INFO[0000] canary subscription is enabled on the following topics: [my-topic] through pubsub=my-pubsub instance=90cccf2889fe scope=dapr.runtime.processor.pubsub type=log ver=edge
+    ```
+
+3. Via the **Publisher** sidecar, the canary message via HTTP/GRPC APIs or SDK, with a CloudEvent field `type` set to `dapr-canary`
+
+    Example using Publish API :
+
+    ```bash
+    curl http://localhost:3500/v1.0/publish/my-pubsub/my-topic?metadata.cloudevent.type=dapr.canary \
+      -H Content-type:application/json \
+      -X POST
+    ```
+
+    If successful, the Dapr **Subscriber** will output a log message of the Canary being received.
+
+    ```bash
+    INFO[0000] 🐦 Canary received! -- CloudEvent Id ded547f3-cdff-4a3f-b140-d3f4eaefc324 instance=90cccf2889fe scope=dapr.runtime.processor.pubsub type=log ver=edge
+    ```
+
+    If you do see the log, then you have successfully validated the PubSub infrastructure.
+
+    If you do not see the log, then there is likely a problem with the PubSub infrastructure which will need investigating.
+
+
+### EXTENDED CANARY DOCS -- THESE WILL BE MOVED OUT OF THE FAQ TO A SUITABLE WHEN THE IMPLEMENTATION IS STABLE
+
+
+Additional information about PubSub canary messages
+
+- In order to invoke the canary behavior on the subscriber, you MUST have a canary-enabled subscriber. This can be achieved by adding the `canary: true` property to ANY subscriber yaml :
+
+    ```yaml
+    apiVersion: dapr.io/v2alpha1
+    kind: Subscription
+    metadata:
+      name: order
+    spec:
+      topic: orders
+      routes:
+        default: /checkout
+      pubsubname: pubsub
+      canary: true
+    scopes:
+    - orderprocessing
+    ```
+
+- In order to send a canary message, the message MUST conform to the CloudEvent specifiction, and the `cloudevent.type` field MUST be set to  `dapr.canary`. 
+- [Dapr Raw events](https://docs.dapr.io/developing-applications/building-blocks/pubsub/pubsub-raw/#subscribing-to-raw-messages) are NOT supported for triggering canary behaviour.
+- Establishing an App Channel with your App (typically done by the `app-port` argument) is OPTIONAL for triggering canary behaviour.
+  - Given the App Channel IS NOT established, canary messages can still be sent successfully, providing the `cloudevent.type` field is set to  `dapr.canary` and the Subscriber yaml is configured with `canary: true`
+    - ANY message that is not authored to the specification above will be dropped.
+  - Given the App Chanel IS established, canary messages can still be sent successfully providing the `cloudevent.type` field is set to  `dapr.canary` and the Subscriber yaml is configured with `canary: true`.
+    -  ANY message that NOT authored to the specification above will be treated as a normal message and will be delivered to the App via the App Channel.