Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Outbox docs #3763

Merged
merged 28 commits into from
Oct 3, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
aee22ad
add outbox docs
yaron2 Sep 25, 2023
b8b1b41
Merge branch 'v1.12' into outbox-1
yaron2 Sep 25, 2023
e0124b5
update preview features
yaron2 Sep 25, 2023
466e0c9
Merge branch 'outbox-1' of github.com:yaron2/docs into outbox-1
yaron2 Sep 25, 2023
cfa77a4
Merge branch 'v1.12' into outbox-1
yaron2 Sep 26, 2023
53781e6
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Sep 26, 2023
ec43914
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Sep 26, 2023
fd46240
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Sep 26, 2023
0c67a66
updates
yaron2 Sep 26, 2023
6ddf0ad
changes
yaron2 Sep 26, 2023
7f4a20f
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Sep 26, 2023
8769743
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Sep 26, 2023
77f430e
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Sep 26, 2023
b10d665
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Sep 26, 2023
18a60af
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Sep 26, 2023
f1d4917
add demo
yaron2 Sep 26, 2023
8e85140
Merge branch 'outbox-1' of github.com:yaron2/docs into outbox-1
yaron2 Sep 26, 2023
f79b298
change redis to mysql
yaron2 Sep 26, 2023
0759b92
add outboxDiscardWhenMissingState clarification
yaron2 Sep 26, 2023
a2557f9
added diagram
yaron2 Sep 26, 2023
e88a195
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Oct 3, 2023
44fa3c9
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Oct 3, 2023
2aba97e
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Oct 3, 2023
0118ebc
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Oct 3, 2023
1487693
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Oct 3, 2023
53c33dc
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Oct 3, 2023
8c58c21
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Oct 3, 2023
646ebea
Update daprdocs/content/en/developing-applications/building-blocks/st…
yaron2 Oct 3, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,10 @@ For more information on message routing, read [Dapr pub/sub API reference]({{< r

Sometimes, messages can't be processed because of a variety of possible issues, such as erroneous conditions within the producer or consumer application or an unexpected state change that causes an issue with your application code. Dapr allows developers to set dead letter topics to deal with messages that cannot be delivered to an application. This feature is available on all pub/sub components and prevents consumer applications from endlessly retrying a failed message. For more information, read about [dead letter topics]({{< ref "pubsub-deadletter.md">}})

### Enabling the outbox pattern

Dapr enables developers to use the outbox pattern for achieving a single transaction across a transactional state store and any message broker. For more information, read [How to enable transactional outbox messaging]({{< ref howto-outbox.md >}})

### Namespace consumer groups

Dapr solves multi-tenancy at-scale with [namespaces for consumer groups]({{< ref howto-namespace >}}). Simply include the `"{namespace}"` value in your component metadata for consumer groups to allow multiple namespaces with applications of the same `app-id` to publish and subscribe to the same message broker.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
---
type: docs
title: "How-To: Enable the transactional outbox pattern"
linkTitle: "How-To: Enable the transactional outbox pattern"
weight: 400
description: "Commit a single transaction across a state store and pub/sub message broker"
---

The transactional outbox pattern is a well known design pattern for sending notifications regarding changes in an application's state. The transactional outbox pattern uses a single transaction that spans across the database and the message broker delivering the notification.

Developers are faced with many difficult technical challenges when trying to implement this pattern on their own, which often involves writing error-prone central coordination managers that, at most, support a combination of one or two databases and message brokers.

For example, you can use the outbox pattern to:
1. Write a new user record to an account database.
1. Send a notification message that the account was successfully created.

With Dapr's outbox support, you can notify subscribers when an application's state is created or updated when calling Dapr's [transactions API]({{< ref "state_api.md#state-transactions" >}}).

The diagram below is an overview of how the outbox feature works:

1) Service A saves/updates state to the state store using a transaction.
2) A message is written to the broker under the same transaction. When the message is successfully delivered to the message broker, the transaction completes, ensuring the state and message are transacted together.
3) The message broker delivers the message topic to any subscribers - in this case, Service B.

<img src="/images/state-management-outbox.png" width=800 alt="Diagram showing the steps of the outbox pattern">

## Requirements

The outbox feature can be used with using any [transactional state store]({{< ref supported-state-stores >}}) supported by Dapr. All [pub/sub brokers]({{< ref supported-pubsub >}}) are supported with the outbox feature.

{{% alert title="Note" color="primary" %}}
Message brokers that work with the competing consumer pattern (for example, [Apache Kafka]({{< ref setup-apache-kafka>}}) are encouraged to reduce the chances of duplicate events.
{{% /alert %}}

## Usage

To enable the outbox feature, add the following required and optional fields on a state store component:

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: mysql-outbox
spec:
type: state.mysql
version: v1
metadata:
- name: connectionString
value: "<CONNECTION STRING>"
- name: outboxPublishPubsub # Required
value: "mypubsub"
- name: outboxPublishTopic # Required
value: "newOrder"
- name: outboxPubsub # Optional
value: "myOutboxPubsub"
- name: outboxDiscardWhenMissingState #Optional. Defaults to false
value: false
```

### Metadata fields

| Name | Required | Default Value | Description |
| --------------------|-------------|---------------|------------------------------------------------------- |
| outboxPublishPubsub | Yes | N/A | Sets the name of the pub/sub component to deliver the notifications when publishing state changes
| outboxPublishTopic | Yes | N/A | Sets the topic that receives the state changes on the pub/sub configured with `outboxPublishPubsub`. The message body will be a state transaction item for an `insert` or `update` operation
| outboxPubsub | No | `outboxPublishPubsub` | Sets the pub/sub component used by Dapr to coordinate the state and pub/sub transactions. If not set, the pub/sub component configured with `outboxPublishPubsub` is used. This is useful if you want to separate the pub/sub component used to send the notification state changes from the one used to coordinate the transaction
| outboxDiscardWhenMissingState | No | `false` | By setting `outboxDiscardWhenMissingState` to `true`, Dapr discards the transaction if it cannot find the state in the database and does not retry. This setting can be useful if the state store data has been deleted for any reason before Dapr was able to deliver the message and you would like Dapr to drop the items from the pub/sub and stop retrying to fetch the state

### Combining outbox and non-outbox messages on the same state store

If you want to use the same state store for sending both outbox and non-outbox messages, simply define two state store components that connect to the same state store, where one has the outbox feature and the other does not.

#### MySQL state store without outbox

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: mysql
spec:
type: state.mysql
version: v1
metadata:
- name: connectionString
value: "<CONNECTION STRING>"
```

#### MySQL state store with outbox

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: mysql-outbox
spec:
type: state.mysql
version: v1
metadata:
- name: connectionString
value: "<CONNECTION STRING>"
- name: outboxPublishPubsub # Required
value: "mypubsub"
- name: outboxPublishTopic # Required
value: "newOrder"
```

## Demo

Watch [this video for an overview of the outbox pattern](https://youtu.be/rTovKpG0rhY?t=1338):

<div class="embed-responsive embed-responsive-16by9">
<iframe width="360" height="315" src="https://youtu.be/rTovKpG0rhY?t=1338" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,10 @@ Dapr enables states to be:

For more details read [How-To: Share state between applications]({{< ref howto-share-state.md >}}),

### Enabling the outbox pattern

Dapr enables developers to use the outbox pattern for achieving a single transaction across a transactional state store and any message broker. For more information, read [How to enable transactional outbox messaging]({{< ref howto-outbox.md >}})

### Querying state

There are two ways to query the state:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ For CLI there is no explicit opt-in, just the version that this was first made a
| **Cryptography** | Encrypt or decrypt data without having to manage secrets keys | N/A | [Cryptography concept]({{< ref "components-concept#cryptography" >}})| v1.11 |
| **Service invocation for non-Dapr endpoints** | Allow the invocation of non-Dapr endpoints by Dapr using the [Service invocation API]({{< ref service_invocation_api.md >}}). Read ["How-To: Invoke Non-Dapr Endpoints using HTTP"]({{< ref howto-invoke-non-dapr-endpoints.md >}}) for more information. | N/A | [Service invocation API]({{< ref service_invocation_api.md >}}) | v1.11 |
| **Actor State TTL** | Allow actors to save records to state stores with Time To Live (TTL) set to automatically clean up old data. In its current implementation, actor state with TTL may not be reflected correctly by clients, read [Actor State Transactions]({{< ref actors_api.md >}}) for more information. | `ActorStateTTL` | [Actor State Transactions]({{< ref actors_api.md >}}) | v1.11 |
| **Transactional Outbox** | Allows state operations for inserts and updates to be published to a configured pub/sub topic using a single transaction across the state store and the pub/sub | N/A | [Transactional Outbox Feature]({{< ref howto-outbox.md >}}) | v1.12 |

### Streaming for HTTP service invocation

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading