SDK generates different message format than dapr api for outbox pattern and message can't be consumed

[fabistb]

Expected Behavior

A published pub/sub message by the outbox pattern be able to be consumed using the standard dotnet sdk setup.
Also the message published by the sdk should have a similar format than the message published by the dapr api itself.

Actual Behavior

The message published by using the sdk and an the api can't be consumed by the application using the app.UseCloudEvents() middleware which returns a 415.

The message published als looks different depending on the utilization of the Dapr API or the dotnet sdk.

dotnet sdk:

{
"data": "{\"data\":\"{\\\"Key\\\":\\\"person1\\\",\\\"FirstName\\\":\\\"Max\\\",\\\"LastName\\\":\\\"Mustermann\\\",\\\"Age\\\":40}\",\
"datacontenttype\":\"text/plain\",\
"id\":\"aa21e718-d94e-47ca-8906-45df7dde620b\",\
"pubsubname\":\"messagebus\",\
"source\":\"main\",\
"specversion\":\"1.0\",\
"time\":\"2024-07-22T21:09:38+02:00\",\
"topic\":\"outbox-topic\",\
"traceid\":\"00-a13d4c515ef09fd1e409ffdb54f77f5d-d2ff4e5e6fa508a7-01\",\
"traceparent\":\"00-a13d4c515ef09fd1e409ffdb54f77f5d-d2ff4e5e6fa508a7-01\",\
"tracestate\":\"\",\
"type\":\"com.dapr.event.sent\"}"
}

dapr api:

{
"data": "{\"data\":\"map[age:35 firstName:Max key:person1 lastName:Mustermann]\",\
"datacontenttype\":\"text/plain\",\
"id\":\"bd2deacf-c9ef-496a-b613-edd509eb4a7e\",\
"pubsubname\":\"messagebus\",\
"source\":\"main\",\
"specversion\":\"1.0\",\
"time\":\"2024-07-22T21:05:02+02:00\",\
"topic\":\"outbox-topic\",\
"traceid\":\"00-9acbe9a5baa4127251ed30db1dae9d59-28539b42a5f2f32a-01\",\
"traceparent\":\"00-9acbe9a5baa4127251ed30db1dae9d59-28539b42a5f2f32a-01\",\
"tracestate\":\"\",\
"type\":\"com.dapr.event.sent\"}"
}

Steps to Reproduce the Problem

Small example app:
https://github.com/fabistb/vis-dapr/tree/feature/53-add-outbox-pattern-example/Main#outbox-pattern

Affected versions

Initially tested with dotnet sdk 1.13.1.
@alicejgibbons also tested with 1.14.0-rc02 and confirms that the issue still persists.

Release Note

RELEASE NOTE:
FIX: Outbox pattern for dotnet sdk

[alicejgibbons]

For repro I can confirm I tested with the Runtime RC 5 and v1.14.0-rc02 for dotnet sdk.

[antontroshin]

/assign

[m3nax]

I think the exception generated by my program when I use the dapr publish command depends on the different serialization of the data field.

Exception: System.Text.Json.JsonException: The JSON value could not be converted to EventRequest. Path: $ | LineNumber: 0 | BytePositionInLine: 5.

DAPR VERSION: 1.14.4
.NET SDK VERSION: 1.14.0

In my case when i publish an event from a .NET program it generates the following event:

{
	"data": {
		"machine": "s17",
		"text": "s15"
	},
	"datacontenttype": "application/json",
	"id": "3f1df251-ee89-49e2-9b60-a2141916494d",
	"pubsubname": "test-pub-sub",
	"source": "dapr-pub-sub-ui",
	"specversion": "1.0",
	"time": "2024-09-18T22:58:58+02:00",
	"topic": "tests",
	"traceid": "00-ed23265a9f2081ab922ba5480b9c4a43-9849ed44dbb8cc62-00",
	"traceparent": "00-ed23265a9f2081ab922ba5480b9c4a43-9849ed44dbb8cc62-00",
	"tracestate": "",
	"type": "com.dapr.event.sent"
}

But when i use the command: dapr publish --pubsub test-pub-sub --publish-app-id dapr-pub-sub-ui -t tests -d '{"machine": "s17", "text": "s15" }'
It generate:

{
	"data": "{machine: s17, text: s15 }",
	"datacontenttype": "application/json",
	"id": "5ab8faa6-8713-4a47-a661-731272de3caa",
	"pubsubname": "test-pub-sub",
	"source": "dapr-pub-sub-ui",
	"specversion": "1.0",
	"time": "2024-09-18T22:57:21+02:00",
	"topic": "tests",
	"traceid": "00-f16c74a03cb9e2faf9622a794c915bca-bdae99cceba9fc24-01",
	"traceparent": "00-f16c74a03cb9e2faf9622a794c915bca-bdae99cceba9fc24-01",
	"tracestate": "",
	"type": "com.dapr.event.sent"
}

Seem dapr doesn't respect the datacontenttype during the serialization of data field (Data deserialization spec v1.0.2) .
Correct?

[fabistb]

@m3nax I think the issue you describe here is something different since you are interacting with the Pub/Sub building block directly.
The OutBox Pattern, even though a message is published at the end belongs to the state store building block.

[m3nax]

@m3nax I think the issue you describe here is something different since you are interacting with the Pub/Sub building block directly. The OutBox Pattern, even though a message is published at the end belongs to the state store building block.

You are right, i will open an issue in dapr/dapr repo.

[alicejgibbons]

@fabistb can this be closed then?

[fabistb]

@alicejgibbons , I tested it again with the dapr 1.14.4 and dotnet-sdk 1.14.0.

It works for me now when calling the dapr endpoint via http directly.
However I have the same behavior as described above when using the dotnet-sdk.

Also setting the Content-Type and datacontenttype via the metadata parameter doesn't appear to have an impact.

[antontroshin]

@fabistb
I took another look at the example you've provided and tried to replicate it locally with Dapr 1.14.4
I used the same test request defined here https://github.com/fabistb/vis-dapr/blob/feature/53-add-outbox-pattern-example/Main/requests/outbox.http#L24
I managed to use the Outbox with a few specific metadata parameters successfully.
What worked for me is to modify the ChangeState (https://github.com/fabistb/vis-dapr/blob/feature/53-add-outbox-pattern-example/Main/Main/Services/OutboxPatternService.cs#L16) in the following manner:

public async Task ChangeState(OutboxDto body)  
{  
    var metadata = new Dictionary<string, string>  
    {  
        { "contentType", "application/json" }, // Content type of the StateTransactionRequest   
        { "datacontenttype", "application/json" } // Content type of the `data` field in cloudevent  
    };  
  
    await _daprClient.ExecuteStateTransactionAsync("outboxstatestore", new[]  
    {  
        new StateTransactionRequest(  
            body.Key,  
            ObjectToJsonByteArray(body),  
            StateOperationType.Upsert,  
            metadata: metadata // <- attach metadata  
        )  
    });  
}

The output was:

== APP == info: Main.Controllers.OutboxPatternController[0]
== APP ==       {"contentType":"application/json","data":"{\"Key\":\"person1\",\"FirstName\":\"Max\",\"LastName\":\"Mustermann\",\"Age\":40}","datacontenttype":"application/json","id":"outbox-b1305fe2-9dfe-4f75-8c3b-4f3c33ab94e8","pubsubname":"outbox-pubsub","source":"state-outbox-service-dotnet","specversion":"1.0","time":"2024-09-25T19:05:00-05:00","topic":"outbox-topic","traceid":"00-f0316ffcf47829eda52045116152d903-ee9e3b63ddc4a921-00","traceparent":"","tracestate":"","type":"com.dapr.event.sent"}

Then, I was able to successfully use app.UseCloudEvents() in Program.cs (https://github.com/fabistb/vis-dapr/blob/feature/53-add-outbox-pattern-example/Main/Main/Program.cs#L51)

The output was:

== APP == info: Main.Controllers.OutboxPatternController[0]
== APP ==       {"Key":"person1","FirstName":"Max","LastName":"Mustermann","Age":40}

Please try the same and update if this helps.

[fabistb]

@antontroshin thanks!
That resolved my issue with the Content-Type.

Unfortunately I think I think I managed to stumble in the next issue.
Could it be that the byte array which represents the content internally is converted into a string and pushed like that to the message broker?

In my example the controller method exceptions any kind of object.

    [HttpPost("outbox-topic")]
    [Topic("messagebus", "outbox-topic")]
    public async Task<IActionResult> ProcessStateInformation([FromBody] object message)
    {
        _logger.LogInformation(message.ToString());
        return Ok();
    }

That is for debug purposes fine but in the running scenario it should accept the OutboxDTO.
So it should look like this:

    [HttpPost("outbox-topic")]
    [Topic("messagebus", "outbox-topic")]
    public async Task<IActionResult> ProcessStateInformation([FromBody] OutboxDto message)
    {
        _logger.LogInformation(message.ToString());
        return Ok();
    }

If I change it like that it unfortunately fails with the following error.

Error processing Redis message 1727443936438-0: retriable error occurred: retriable error returned from 
app while processing pub/sub event outbox-ca91be8e-99a0-44cd-a5ca-017455c383d8, topic: 
outbox-topic, body: {"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1","title":"One 
or more validation errors occurred.","status":400,
"traceId":"00-348a53fe1c197efe2a52b01945f050e4-03fc0d6d3484f23f-01","errors":
{"$":["The JSON value could not be converted to Main.Models.OutboxDto. Path: 
$ | LineNumber: 0 | BytePositionInLine: 140."],"message":["The message field is required."]}}. 
status code returned: 400

The error doesn't make that much sense for me because the neither the DTO nor the request has a message field.

When I change the expected typ back object and compare the ValueKind displayed by my IDE from the Dapr API and the Dotnet-SDK they differ.

Dotnet-SDK:

** Dapr-API**:

If i now send the same string like the sdk generates directly to the Controller, I receive the same error as above.

### Post directly to pubsub endpoint
POST http://localhost:5070/api/v1.0/outbox/outbox-topic
Content-Type: application/json

"{"Key":"person1","FirstName":"Max","LastName":"Mustermann","Age":40}"

{
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "traceId": "00-d071d3ce5f055bfb5464624ae941554c-a01a9e7e42cb1716-00",
  "errors": {
    "$": [
      "The JSON value could not be converted to Main.Models.OutboxDto. Path: $ | LineNumber: 0 | BytePositionInLine: 3."
    ],
    "message": [
      "The message field is required."
    ]
  }
}

[antontroshin]

@fabistb This was fixed in the PR dapr/dapr#8062 opened following this issue and will be released in 1.14.5.

[fabistb]

@antontroshin great!
Thank you very much solving the issue.