az containerapp dapr enable --dapr-http-max-request-size 32 fails to increase max-request size.

[dale-personal]

Please provide us with the following information:

This issue is a: (mark with an x)

• [ x] bug report -> please search issues before submitting
• documentation issue or request
• regression (a behavior that used to work and stopped in a new release)

Issue description

Following the documentation found here:
https://learn.microsoft.com/en-us/cli/azure/containerapp/dapr?view=azure-cli-latest#az-containerapp-dapr-enable
The parameter to increase max request size does not increase the max request size.

Steps to reproduce

1. deploy a container app to azure container apps
2. run the cli command "az containerapp dapr enable --dapr-http-max-request-size 32" to enable dapr with the increased request size.
3. send a request more than the 4MB default dapr limit.
4. you will see a cryptic error message in the logs: System.Net.Http.HttpRequestException: An error occurred while sending the request.
   2022-09-21T12:26:38.938415931Z ---> System.IO.IOException: The response ended prematurely.
5. Running dapr locally you see a much more helpful error message: Dapr.DaprApiException: error invoke actor method: body size exceeds the given limit

Expected behavior [What you expected to happen.]
The container app has dapr enabled with the max request size increased.
When i run this command the output seems to indicate success:

{
"appId": "api",
"appPort": 80,
"appProtocol": "http",
"enableApiLogging": false,
"enabled": true,
"httpMaxRequestSize": 32,
"logLevel": "info"
}

Actual behavior [What actually happened.]
dapr is enabled, but the max request size is not increased.
When i look at the JSON for the container app overview in the portal, I only see:

"dapr": {
"enabled": true,
"appId": "api",
"appProtocol": "http",
"appPort": 80
}

[anthonychu]

@dale-personal When you view the container app json in the portal, can you please change the api version using the drop-down to 2022-06-01-preview and confirm the value? By default it uses an older version and doesn’t show the extra properties.

[dale-personal]

Ok, i do see the expected values when i change the API version:
"dapr": {
"enabled": true,
"appId": "api",
"appProtocol": "http",
"appPort": 80,
"httpMaxRequestSize": 32,
"logLevel": "info",
"enableApiLogging": false
}

I've done a little more digging and I believe the setting is working as expected for "normal" service calls.

I believe the bug is in the ActorProxy. When calling an actor proxy method that returns more that 4 MB of payload, I get the error below. I have set httpMaxRequestSize to 32 on the actor service, and as seen in the stack trace, it uses http, not grpc.

2022-09-23T15:51:35.732224614Z fail: Microsoft.AspNetCore.Server.Kestrel[13]
2022-09-23T15:51:35.732605324Z Connection id "0HMKTF7MOLASF", Request id "0HMKTF7MOLASF:00000003": An unhandled exception was thrown by the application.
2022-09-23T15:51:35.732650125Z System.Net.Http.HttpRequestException: An error occurred while sending the request.
2022-09-23T15:51:35.732657225Z ---> System.IO.IOException: The response ended prematurely.
2022-09-23T15:51:35.732663025Z at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-09-23T15:51:35.732668925Z --- End of inner exception stack trace ---
2022-09-23T15:51:35.732673825Z at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-09-23T15:51:35.732680425Z at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2022-09-23T15:51:35.732685926Z at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-09-23T15:51:35.732690826Z at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2022-09-23T15:51:35.732697626Z at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
2022-09-23T15:51:35.732704726Z at Dapr.Actors.DaprHttpInteractor.SendAsyncHandleSecurityExceptions(Func1 requestFunc, CancellationToken cancellationToken) 2022-09-23T15:51:35.732710326Z at Dapr.Actors.DaprHttpInteractor.SendAsyncHandleUnsuccessfulResponse(Func1 requestFunc, String relativeUri, CancellationToken cancellationToken)
2022-09-23T15:51:35.732715126Z at Dapr.Actors.DaprHttpInteractor.SendAsync(Func1 requestFunc, String relativeUri, CancellationToken cancellationToken) 2022-09-23T15:51:35.732722126Z at Dapr.Actors.DaprHttpInteractor.InvokeActorMethodWithRemotingAsync(ActorMessageSerializersManager serializersManager, IActorRequestMessage remotingRequestRequestMessage, CancellationToken cancellationToken) 2022-09-23T15:51:35.732726827Z at Dapr.Actors.Communication.Client.ActorRemotingClient.InvokeAsync(IActorRequestMessage remotingRequestMessage, CancellationToken cancellationToken) 2022-09-23T15:51:35.732740427Z at Dapr.Actors.Client.ActorProxy.InvokeMethodAsync(Int32 interfaceId, Int32 methodId, String methodName, IActorRequestMessageBody requestMsgBodyValue, CancellationToken cancellationToken) 2022-09-23T15:51:35.732745627Z at Dapr.Actors.Client.ActorProxy.ContinueWithResult[TRetval](Int32 interfaceId, Int32 methodId, Task1 task)

[kendallroden]

This change is still rolling out - thanks for your patience. We will investigate

[zhenqxuMSFT]

Hi @dale-personal , this looks like an issue in dapr actor. Could you also file a bug to dapr? https://github.com/dapr/dapr/issues

[berndverst]

This appears to be a DotNet SDK issue. I opened this tracking issue dapr/dotnet-sdk#978

This issue should not exist in the Python SDK (though Python currently does not use gRPC for actors while the DotNet SDK does).

[berndverst]

Please note that the receiving application (the actor host) must support receiving messages of the desired size. Some frameworks do not support such large messages by default and additional configuration may be required.

[yash-nisar]

I was not able to reproduce the issue with the dotnet-sdk. I was able to successfully send a >4MB payload through a remoting client.

for (int i = 0; i < 120000; i++)
{
    testData.Add(i, i);
}

and I used the following code to measure the total payload size:

var payloadSize = request.Headers.ToString().Length + (request.Content?.Headers.ContentLength ?? 0); 
Console.WriteLine(payloadSize/(1024.0 * 1024.0));

@dale-personal Do you mind sharing how did you set httpMaxRequestSize to 32 on the actor service ?

[dale-personal]

@yash-nisar

Above in the steps to reproduce:

2. run the cli command "az containerapp dapr enable --dapr-http-max-request-size 32" to enable dapr with the increased request size.

There is no way to set this specifically for the actor service, just the dapr runtime as far as i know. so the assumption is executing this commend set the max request size for the actor service as well (and it does work using dapr locally outside of the container environment)

[yash-nisar]

@dale-personal Would you be able to answer a few more questions if possible so we can troubleshoot what's going on.

1. How did you get the stack trace that you have posted above ? Were you using the dotnet-sdk ?
2. When you say "and it does work using dapr locally outside of the container environment", do you mean you are able to send >4MB payload using dapr locally ?

[dale-personal]

@yash-nisar

1. yes, i am using the dotnet-sdk. If i recall correctly, that stack trace came out of the container app logs.
2. What i meant by that is when i run locally setting the max-request-size to 32 (see: https://docs.dapr.io/operations/configuration/increase-request-size/), i am able to send requests larger than 4 MB. however, when i try the same thing using the AZ CLI (az containerapp dapr enable --dapr-http-max-request-size 32), i get the error originally described.

[simonjj]

Closing this issue due to it being inactive. If it still exists please reopen a new issue and point to this to preserve history.