Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error "Request Entity Too Large" #559

Closed
pditommaso opened this issue Jul 9, 2024 · 22 comments · Fixed by #658
Closed

Error "Request Entity Too Large" #559

pditommaso opened this issue Jul 9, 2024 · 22 comments · Fixed by #658
Assignees
@munishchouhan

Comments

@pditommaso
Copy link
Collaborator

Submitting a build request using a large Conda environment, the following error is thrown:

io.micronaut.http.client.exceptions.HttpClientResponseException: Client 'http://surrealv1:8100': Request Entity Too Large
	at io.micronaut.http.client.netty.DefaultHttpClient.handleStreamHttpError(DefaultHttpClient.java:1531)
	at io.micronaut.http.client.netty.DefaultHttpClient.lambda$streamRequestThroughChannel$47(DefaultHttpClient.java:1522)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:386)
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57)
	at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:814)
	at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:739)
	at reactor.core.publisher.FluxCreate$SerializedFluxSink.next(FluxCreate.java:161)
	at io.micronaut.http.client.netty.DefaultHttpClient.lambda$streamRequestThroughChannel0$48(DefaultHttpClient.java:1566)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
	at io.micronaut.http.client.netty.DefaultHttpClient$StreamStreamHttpResponseHandler.buildResponse(DefaultHttpClient.java:2343)
	at io.micronaut.http.client.netty.DefaultHttpClient$StreamStreamHttpResponseHandler.buildResponse(DefaultHttpClient.java:2331)
	at io.micronaut.http.client.netty.DefaultHttpClient$BaseHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2092)
	at io.micronaut.http.client.netty.DefaultHttpClient$BaseHttpResponseHandler.channelReadInstrumented(DefaultHttpClient.java:2010)
	at io.micronaut.http.client.netty.SimpleChannelInboundHandlerInstrumented.channelRead0(SimpleChannelInboundHandlerInstrumented.java:49)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.micronaut.http.netty.reactive.HandlerPublisher.channelRead(HandlerPublisher.java:325)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.micronaut.http.netty.stream.HttpStreamsHandler.channelRead(HttpStreamsHandler.java:280)
	at io.micronaut.http.netty.stream.HttpStreamsClientHandler.channelRead(HttpStreamsClientHandler.java:180)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)

The build raising this error is the following https://wave.seqera.io/view/builds/90d5550d6507c09c_1
@munishchouhan
Copy link
Member

This build id is no longer available.

@munishchouhan
Copy link
Member

I did ask the question in surrealdb discord channel, but no luck yet.
I will dig more into it.

@munishchouhan
Copy link
Member

I think it can be solved by increasing the size of micronaut.server.max-request-size, its default size is 10mb
https://micronaut-projects.github.io/micronaut-core/latest/guide/configurationreference.html#io.micronaut.http.server.HttpServerConfiguration

@munishchouhan
Copy link
Member

I will reproduce the issue and test it

@pditommaso
Copy link
Collaborator Author

Surely it's not a Micronaut *server* setting. It may be the client one

@munishchouhan
Copy link
Member

munishchouhan commented Jul 31, 2024 •
edited
Loading

I tried the same conda file to reproduce this error but there was no error in dev, it maybe because of the surreal updated version
I will try to reproduce it with the old one

@munishchouhan
Copy link
Member

finally able to reproduce the error, will work on fix now

@munishchouhan munishchouhan mentioned this issue Jul 31, 2024
Closed
2 tasks
@munishchouhan
Copy link
Member

opened issue with surreal team
surrealdb/surrealdb#4452

@pditommaso
Copy link
Collaborator Author

Have you tried the setting micronaut.http.client.max-content-length ?

https://micronaut-projects.github.io/micronaut-docs-mn2/2.0.3/guide/configurationreference.html

@munishchouhan
Copy link
Member

Have you tried the setting micronaut.http.client.max-content-length ?

https://micronaut-projects.github.io/micronaut-docs-mn2/2.0.3/guide/configurationreference.html

Not yet, thank you i will try now

@munishchouhan
Copy link
Member

Have you tried the setting micronaut.http.client.max-content-length ?

https://micronaut-projects.github.io/micronaut-docs-mn2/2.0.3/guide/configurationreference.html

still same error

@munishchouhan
Copy link
Member

I have tried the following options, but no luck:

1. micronaut.server.max-request-size
2. micronaut.http.services.*.max-content-length
3. micronaut.server.netty.max-initial-line-length
4. micronaut.http.client.max-content-length

@munishchouhan
Copy link
Member

munishchouhan commented Aug 1, 2024 •
edited
Loading

We have hit issue one year back also
#253 (comment)
it is indeed surrealdb limitation
we changed our strategy that time to send cves in different calls.
they did advice us to use java drivers, but there was a compatibility issue.
drivers works with Java 17.
They did mentioned to change the version to 8.
But that issue is still open
surrealdb/surrealdb.java#58

@munishchouhan
Copy link
Member

we are using java 17 in micronaut 4
so there we can switch to surreal drivers

@pditommaso
Copy link
Collaborator Author

Which is the current max allowed length? is the limitation coming from the protocol? http vs websocket?

@munishchouhan
Copy link
Member

current limit is 16kb
surrealdb/surrealdb#1545 (comment)
it can be overcome using websocket
surrealdb/surrealdb#1545 (comment)

@pditommaso
Copy link
Collaborator Author

Then let's add a truncation of content longer > 16k here for now

wave/src/main/groovy/io/seqera/wave/controller/ContainerController.groovy

Lines 307 to 309 in 5170386

final containerSpec = decodeBase64OrFail(req.containerFile, 'containerFile')
final condaContent = condaFileFromRequest(req)
final spackContent = spackFileFromRequest(req)

@munishchouhan
Copy link
Member

Then let's add a truncation of content longer > 16k here for now

wave/src/main/groovy/io/seqera/wave/controller/ContainerController.groovy

Lines 307 to 309 in 5170386

final containerSpec = decodeBase64OrFail(req.containerFile, 'containerFile')
final condaContent = condaFileFromRequest(req)
final spackContent = spackFileFromRequest(req)

We only need to truncate them when saving, so it is better to truncate here

wave/src/main/groovy/io/seqera/wave/service/persistence/impl/SurrealPersistenceService.groovy

Line 150 in 760f217

void saveContainerRequest(String token, WaveContainerRecord data) {

@pditommaso
Copy link
Collaborator Author

You are right

@munishchouhan munishchouhan linked a pull request Aug 1, 2024 that will close this issue
Fixes Error "Request Entity Too Large" #585
Closed
@pditommaso pditommaso mentioned this issue Sep 13, 2024
Merged
@pditommaso
Copy link
Collaborator Author

It would be better to solve this using Surreal SQL statement, instead of truncating the Conda env.

@munishchouhan
Copy link
Member

It would be better to solve this using Surreal SQL statement, instead of truncating the Conda env.

ok sure, i will work on it

@munishchouhan
Copy link
Member

Tried different ways to add it using SQL, the issue is paring of containerfile

{
    "code": 400,
    "details": "Request problems detected",
    "description": "There is a problem with your request. Refer to the documentation for further information.",
    "information": "There was a problem with the database: Parse error: Failed to parse query at line 6 column 29 expected an identifier\n  |\n6 | containerFile: \"FROM mambaorg/micromamba:1.5.8-lunar\n  |                ^ \n"
}

I changed it to the bytes which works, but then it will break the decoding when loading the objects from surrealdb

I will dig further to find better solutions

@munishchouhan munishchouhan mentioned this issue Sep 26, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@munishchouhan munishchouhan

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix entity too large seqeralabs/wave
Fixes Error "Request Entity Too Large" seqeralabs/wave
2 participants
@pditommaso @munishchouhan