Mongoose not always reading all data in TLS buffer

[kylex2]

Hi,
Sometimes we see that mongoose is not decrypting all incoming data and putting it into the c->recv buffer. When this happens it will throw mg_error with MG_MAX_RECV_SIZE. Data is being sent to mongoose using curl. Since the rtls buffer is not handled by us I wonder if this could be some internal mongoose behaviour causing this?

Thanks!

Mongoose version: 7.14 release
OpenSSL: 3.1.1

Log ending from working transfer:

7ded31 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 118, amount of bytes in c->recv: 0
7ded31 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3014, amount of bytes in c->recv: 0
7ded32 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 1088, amount of bytes in c->recv: 0
7ded32 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3984, amount of bytes in c->recv: 0
7ded32 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 610, amount of bytes in c->recv: 0
7ded32 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3506, amount of bytes in c->recv: 0
7ded33 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 132, amount of bytes in c->recv: 0
7ded33 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3028, amount of bytes in c->recv: 0
7ded34 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 1102, amount of bytes in c->recv: 0
7ded34 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3998, amount of bytes in c->recv: 0
7ded35 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 624, amount of bytes in c->recv: 0
7ded35 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3520, amount of bytes in c->recv: 0
7ded35 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 146, amount of bytes in c->recv: 0
7ded35 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 4490, amount of bytes in c->recv: 0
7ded36 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 0, amount of bytes in c->recv: 0
7ded36 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 0, amount of bytes in c->recv: 0
7ded36 2 mongoose_ipc.c:375:handle Uploaded 31031296 bytes
7df34e 2 mongoose_ipc.c:188:ipc_cb {"type": "system_status", "status": "SUCCESS"}

Log ending from failed transfer:

7b4c75 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3063786, amount of bytes in c->recv: 0
7b4c75 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3080192, amount of bytes in c->recv: 0
7b4c84 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3071978, amount of bytes in c->recv: 0
7b4c85 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3088384, amount of bytes in c->recv: 0
7b4c94 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3080170, amount of bytes in c->recv: 0
7b4c94 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3096576, amount of bytes in c->recv: 0
7b4ca3 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3088362, amount of bytes in c->recv: 0
7b4ca3 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3104768, amount of bytes in c->recv: 0
7b4cb3 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3096554, amount of bytes in c->recv: 0
7b4cb3 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3112960, amount of bytes in c->recv: 0
7b4cc2 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3104746, amount of bytes in c->recv: 0
7b4cc2 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3121152, amount of bytes in c->recv: 0
7b4cd3 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3112938, amount of bytes in c->recv: 0
7b4cd3 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3129344, amount of bytes in c->recv: 0
7b4ce2 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3121130, amount of bytes in c->recv: 0
7b4ce2 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3137536, amount of bytes in c->recv: 0
7b4cf2 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3129322, amount of bytes in c->recv: 0
7b4cf2 2 mongoose_ipc.c:368:handle amount of bytes in c->rtls: 3145728, amount of bytes in c->recv: 0
7b4cf2 1 event.c:31:mg_error            6 8 MG_MAX_RECV_SIZE
7b4cf2 2 mongoose_ipc.c:473:cb     Mongoose buffer is full
7b4cf2 2 mongoose_ipc.c:478:cb     IS CLOSING, 6, 0, 3145728

[scaprile]

You are exhausting the receive buffer, hence MG_MAX_RECV_SIZE
Buffering depends on TLS block size.
https://mongoose.ws/documentation/#build-options
#2779 (comment)

[kylex2]

Thanks for your reply. I was under the impression that we handle the data we get by consuming the c->recv on MV_EV_READ. The buffer that runs out is in c->rtls. Could this be related to #2668?

[scaprile]

No, 2668 is a different claim. Your issue seems to be the same as the one I linked to.
You can't consume data until it has been decrypted, and for some reason TLS is accumulating data (too large a block ?). The quick hack mentioned there also applies to you.

[kylex2]

I have now logged what seems to happen when it starts to fail.

[23880.207688] sock.c:286:read_conn              
[23880.207692] sock.c:287:read_conn              recv_raw returned 4344 bytes
[23880.207725] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.207739] sock.c:286:read_conn              
[23880.207743] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.207752] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.207767] sock.c:286:read_conn              
[23880.207770] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.207779] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.207791] sock.c:286:read_conn              
[23880.207794] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.207804] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.207835] sock.c:286:read_conn              
[23880.207839] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.207849] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.207887] sock.c:286:read_conn              
[23880.207891] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.207901] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.207920] sock.c:286:read_conn              
[23880.207924] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.207969] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.207974] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.208061] sock.c:286:read_conn              
[23880.208065] sock.c:287:read_conn              recv_raw returned 2896 bytes
[23880.208073] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.208077] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.208199] sock.c:286:read_conn              
[23880.208203] sock.c:287:read_conn              recv_raw returned 5792 bytes
[23880.208239] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.208266] sock.c:286:read_conn              
[23880.208269] sock.c:287:read_conn              recv_raw returned 2896 bytes
[23880.208295] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.208308] sock.c:286:read_conn              
[23880.208312] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.208321] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.208333] sock.c:286:read_conn              
[23880.208337] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.208346] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.208385] sock.c:286:read_conn              
[23880.208389] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.208436] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.208441] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.208510] sock.c:286:read_conn              
[23880.208514] sock.c:287:read_conn              recv_raw returned 4344 bytes
[23880.208521] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.208526] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.208644] sock.c:286:read_conn              
[23880.208648] sock.c:287:read_conn              recv_raw returned 4344 bytes
[23880.208682] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.208709] sock.c:286:read_conn              
[23880.208713] sock.c:287:read_conn              recv_raw returned 2896 bytes
[23880.208726] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.208763] sock.c:286:read_conn              
[23880.208767] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.208777] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.208800] sock.c:286:read_conn              
[23880.208803] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.208812] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.209549] sock.c:286:read_conn              
[23880.209557] sock.c:287:read_conn              recv_raw returned 16384 bytes
[23880.209638] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.209643] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.209722] sock.c:286:read_conn              
[23880.209726] sock.c:287:read_conn              recv_raw returned 968 bytes
[23880.209734] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.209738] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.209945] sock.c:286:read_conn              
[23880.209949] sock.c:287:read_conn              recv_raw returned 8192 bytes
[23880.210058] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.210063] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.210147] sock.c:286:read_conn              
[23880.210151] sock.c:287:read_conn              recv_raw returned 16406 bytes
[23880.210158] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.210163] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.210423] sock.c:286:read_conn              
[23880.210427] sock.c:287:read_conn              recv_raw returned 8192 bytes
[23880.210533] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.210538] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.210616] sock.c:286:read_conn              
[23880.210620] sock.c:287:read_conn              recv_raw returned 13570 bytes
[23880.210627] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.210632] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.210752] sock.c:286:read_conn              
[23880.210756] sock.c:287:read_conn              recv_raw returned 2836 bytes
[23880.210856] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.210861] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.210935] sock.c:286:read_conn              
[23880.210939] sock.c:287:read_conn              recv_raw returned 7300 bytes
[23880.210964] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.210968] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.211090] sock.c:286:read_conn              
[23880.211094] sock.c:287:read_conn              recv_raw returned 7240 bytes
[23880.211193] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.211198] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.211270] sock.c:286:read_conn              
[23880.211274] sock.c:287:read_conn              recv_raw returned 5792 bytes
[23880.211281] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.211286] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.211403] sock.c:286:read_conn              
[23880.211407] sock.c:287:read_conn              recv_raw returned 5792 bytes
[23880.211503] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.211508] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.211581] sock.c:286:read_conn              
[23880.211584] sock.c:287:read_conn              recv_raw returned 5792 bytes
[23880.211592] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.211596] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.211714] sock.c:286:read_conn              
[23880.211718] sock.c:287:read_conn              recv_raw returned 5792 bytes
[23880.211812] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.211817] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.211887] sock.c:286:read_conn              
[23880.211890] sock.c:287:read_conn              recv_raw returned 5792 bytes
[23880.211898] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.211902] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.212046] sock.c:286:read_conn              
[23880.212050] sock.c:287:read_conn              recv_raw returned 5792 bytes
[23880.212142] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.212147] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.212217] sock.c:286:read_conn              
[23880.212221] sock.c:287:read_conn              recv_raw returned 7240 bytes
[23880.212228] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.212233] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.212352] sock.c:286:read_conn              
[23880.212356] sock.c:287:read_conn              recv_raw returned 5792 bytes
[23880.212400] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.212418] sock.c:286:read_conn              
[23880.212422] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.212431] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.212443] sock.c:286:read_conn              
[23880.212447] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.212456] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.212467] sock.c:286:read_conn              
[23880.212471] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.212517] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.212522] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.212590] sock.c:286:read_conn              
[23880.212594] sock.c:287:read_conn              recv_raw returned 4344 bytes
[23880.212601] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.212606] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.212725] sock.c:286:read_conn              
[23880.212729] sock.c:287:read_conn              recv_raw returned 4344 bytes
[23880.212763] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.212790] sock.c:286:read_conn              
[23880.212812] sock.c:287:read_conn              recv_raw returned 2896 bytes
[23880.212826] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.212839] sock.c:286:read_conn              
[23880.212843] sock.c:287:read_conn              recv_raw returned 1448 bytes
[23880.212852] tls_openssl.c:243:mg_tls_recv     len: 8192, n: -1
[23880.213568] sock.c:286:read_conn              
[23880.213576] sock.c:287:read_conn              recv_raw returned 24616 bytes
[23880.213657] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.213662] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.213745] sock.c:286:read_conn              
[23880.213749] sock.c:287:read_conn              recv_raw returned 8688 bytes
[23880.213757] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.213762] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.213920] sock.c:286:read_conn              
[23880.213924] sock.c:287:read_conn              recv_raw returned 1838 bytes
[23880.214028] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.214033] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.214108] sock.c:286:read_conn              
[23880.214112] sock.c:287:read_conn              recv_raw returned 11194 bytes
[23880.214119] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.214124] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.214258] sock.c:286:read_conn              
[23880.214262] sock.c:287:read_conn              recv_raw returned 5212 bytes
[23880.214362] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.214367] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.214440] sock.c:286:read_conn              
[23880.214444] sock.c:287:read_conn              recv_raw returned 7820 bytes
[23880.214451] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.214455] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.215101] sock.c:286:read_conn              
[23880.215107] sock.c:287:read_conn              recv_raw returned 8586 bytes
[23880.215225] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.215230] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.215314] sock.c:286:read_conn              
[23880.215317] sock.c:287:read_conn              recv_raw returned 16406 bytes
[23880.215325] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.215329] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.215620] sock.c:286:read_conn              
[23880.215624] sock.c:287:read_conn              recv_raw returned 8192 bytes
[23880.215738] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.215743] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.215824] sock.c:286:read_conn              
[23880.215827] sock.c:287:read_conn              recv_raw returned 16048 bytes
[23880.215835] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.215839] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.215974] sock.c:286:read_conn              
[23880.215978] sock.c:287:read_conn              recv_raw returned 358 bytes
[23880.216559] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.216567] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.216675] sock.c:286:read_conn              
[23880.216679] sock.c:287:read_conn              recv_raw returned 16406 bytes
[23880.216687] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.216692] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192
[23880.216976] sock.c:286:read_conn              
[23880.216999] sock.c:287:read_conn              recv_raw returned 8192 bytes
[23880.217120] tls_openssl.c:243:mg_tls_recv     len: 8192, n: 8192
[23880.217125] sock.c:108:iolog                  running callback with ev: MG_EV_READ, c->recv.len: 8192

As you can see mongoose starts reading more data from the socket than it's decrypting and passing to the callback for consumption.
For TLS connections there doesn't seem to exist any code path for increasing the c->recv iobuf so max chunk size will be MG_IO_SIZE no matter how many bytes are actually needed.

[scaprile]

All I see is the same that I explained to the other OP and linked above.
Our file upload examples were not written for TLS, I don't know what you did, data is accumulating and exhausting the recv buffer or the rtls buffer, you have to handle smaller chunks or enlarge the buffer

[kylex2]

Thank you for reply. I guess I don't understand your explanation. Data is not exhausting the recv buffer. The recv buffer always stays at MG_IO_SIZE. It is the rtls buffer that is being exhausted because mongoose is filling it with larger chunks than what is used when decrypting into the recv buffer. Are you suggesting I should handle the c->rtls buffer manually?

[scaprile]

Data accumulates in c->rtls faster than it is consumed; even though the example is draining c->recv properly, eventually, if the file is large enough, c->rtls ends up triggering an OOM.
Please see our documentation, and follow the guidelines in our tutorials.
https://mongoose.ws/documentation/#build-options

MG_IO_SIZE 2048 Granularity of the send/recv IO buffer growth

Make it larger so buffers can be drained faster, basic comms theory: you either throttle the sender or buffer its data

[kylex2]

Yes, throttling the socket reads would work:

diff --git a/src/sock.c b/src/sock.c
index 66a23482..6ae40971 100644
--- a/src/sock.c
+++ b/src/sock.c
@@ -44,6 +44,8 @@
   (((errcode) < 0) && (errno == EPIPE || errno == ECONNRESET))
 #endif
 
+#define min(a,b) ((a) < (b) ? (a) : (b))
+
 union usa {
   struct sockaddr sa;
   struct sockaddr_in sin;
@@ -278,7 +280,7 @@ static void read_conn(struct mg_connection *c) {
     if (c->is_tls) {
       if (!ioalloc(c, &c->rtls)) return;
       n = recv_raw(c, (char *) &c->rtls.buf[c->rtls.len],
-                   c->rtls.size - c->rtls.len);
+                   min(c->rtls.size - c->rtls.len, len));
       if (n == MG_IO_ERR && c->rtls.len == 0) {
         // Close only if we have fully drained both raw (rtls) and TLS buffers
         c->is_closing = 1;

Also, increasing the c->recv would also work in my case:

diff --git a/src/sock.c b/src/sock.c
index 66a23482..ff60d411 100644
--- a/src/sock.c
+++ b/src/sock.c
@@ -91,6 +91,8 @@ static void setlocaddr(MG_SOCKET_TYPE fd, struct mg_addr *addr) {
   }
 }
 
+static bool ioalloc(struct mg_connection *c, struct mg_iobuf *io);
+
 static void iolog(struct mg_connection *c, char *buf, long n, bool r) {
   if (n == MG_IO_WAIT) {
     // Do nothing
@@ -104,7 +106,8 @@ static void iolog(struct mg_connection *c, char *buf, long n, bool r) {
     }
     if (r) {
       c->recv.len += (size_t) n;
-      mg_call(c, MG_EV_READ, &n);
+      if (!c->is_tls || ioalloc(c, &c->recv))
+        mg_call(c, MG_EV_READ, &n);
     } else {
       mg_iobuf_del(&c->send, 0, (size_t) n);
       // if (c->send.len == 0) mg_iobuf_resize(&c->send, 0);

A third way which would also make it work is the already rejected suggestion by @tubular-rheostat in #2668.

[kylex2]

Something like this maybe, it will throttle reads from the socket as long as there is decrypted data available for processing.

diff --git a/src/sock.c b/src/sock.c
index 66a23482..851f164f 100644
--- a/src/sock.c
+++ b/src/sock.c
@@ -277,6 +277,13 @@ static void read_conn(struct mg_connection *c) {
     long n = -1;
     if (c->is_tls) {
       if (!ioalloc(c, &c->rtls)) return;
+
+      if (mg_tls_pending(c)) {
+        n = mg_tls_recv(c, buf, len);
+        iolog(c, buf, n, true);
+        return;
+      }
+
       n = recv_raw(c, (char *) &c->rtls.buf[c->rtls.len],
                    c->rtls.size - c->rtls.len);
       if (n == MG_IO_ERR && c->rtls.len == 0) {

[kylex2]

This is same as above but more integrated. It de-couples the MG_IO_SIZE from TLS record size and makes sure to fill c->recv similar to non TLS connections making them look and behave the same from the callback point of view. Also c->rtls is used very lightly compared to reaching MG_MAX_RECV_SIZE before. Now it will be c->recv generating OOM just like it would on non TLS connections. Please let me know if you are interested at all. Thanks

diff --git a/src/sock.c b/src/sock.c
index 66a23482..f921f99f 100644
--- a/src/sock.c
+++ b/src/sock.c
@@ -274,16 +274,18 @@ static void read_conn(struct mg_connection *c) {
   if (ioalloc(c, &c->recv)) {
     char *buf = (char *) &c->recv.buf[c->recv.len];
     size_t len = c->recv.size - c->recv.len;
-    long n = -1;
+    long n = -1, n_pending;
     if (c->is_tls) {
       if (!ioalloc(c, &c->rtls)) return;
-      n = recv_raw(c, (char *) &c->rtls.buf[c->rtls.len],
-                   c->rtls.size - c->rtls.len);
+      n = n_pending = mg_tls_pending(c);
+      if (!n_pending)
+        n = recv_raw(c, (char *) &c->rtls.buf[c->rtls.len],
+                     c->rtls.size - c->rtls.len);
       if (n == MG_IO_ERR && c->rtls.len == 0) {
         // Close only if we have fully drained both raw (rtls) and TLS buffers
         c->is_closing = 1;
       } else {
-        if (n > 0) c->rtls.len += (size_t) n;
+        if (n > 0 && !n_pending) c->rtls.len += (size_t) n;
         if (c->is_tls_hs) mg_tls_handshake(c);
         n = c->is_tls_hs ? (long) MG_IO_WAIT : mg_tls_recv(c, buf, len);
       }

[scaprile]

AFAI'm concerned, and based on prior resolution of corner cases, and the fact that Mongoose not only runs over sockets nor OSs, no, we are not interested in your modifications. AFAI'm concerned, the answer is the one given and the fix to your issue is the one provided, coincident with what can be read in our docs.
Thank you.

[kylex2]

Ok, thanks for the clarification.