50Gb trasfer with Azure Blob back-end fails

[teekeino]

We are hitting a problem when trying to send a big file through SFTP with Azure Blob as back-end. This is the error we are seeing in the logs. Can't figure out if this is an Azure problem or a sftpgo problem. Help?

{"level":"warn","time":"2021-05-06T19:37:48.948","sender":"SFTP","connection_id":"SFTP_1a835fb878bc9e53e0335e665dad8c55e252e002a82d939fede3cb85ed1180cf_1","message":"Unexpected error for transfer, path: \"temp_50GB_file\", error: \"EOF\" bytes sent: 0, bytes received: 44429475840 transfer running since 1492855 ms"} {"level":"debug","time":"2021-05-06T19:37:48.949","sender":"Azure Blob SAS URL \"\"","connection_id":"1a835fb878bc9e53e0335e665dad8c55e252e002a82d939fede3cb85ed1180cf_1","message":"multipart upload error: -> github.com/Azure/azure-pipeline-go/pipeline.NewError, /home/runner/go/pkg/mod/github.com/!azure/[email protected]/pipeline/error.go:157\nHTTP request failed\n\nPut \"https://relexsftpstorage.blob.core.windows.net/sftp-dev-test/temp_50GB_file?blockid=YCkAAAAAAAA%3D&comp=block&se=2042-01-01T00%3A00%3A00Z&sig=bKVoczCpnP0iYA4Qpx6GMjFPmd6t1ZrqmWbv4HHvlUs%3D&sp=rwdl&sr=c&sv=2018-11-09&timeout=240\": context canceled\n"} {"level":"debug","time":"2021-05-06T19:37:49.014","sender":"Azure Blob SAS URL \"\"","connection_id":"1a835fb878bc9e53e0335e665dad8c55e252e002a82d939fede3cb85ed1180cf_1","message":"upload completed, path: \"temp_50GB_file\", readed bytes: 44429672448, err: -> github.com/Azure/azure-pipeline-go/pipeline.NewError, /home/runner/go/pkg/mod/github.com/!azure/[email protected]/pipeline/error.go:157\nHTTP request failed\n\nPut \"https://relexsftpstorage.blob.core.windows.net/sftp-dev-test/temp_50GB_file?blockid=YCkAAAAAAAA%3D&comp=block&se=2042-01-01T00%3A00%3A00Z&sig=bKVoczCpnP0iYA4Qpx6GMjFPmd6t1ZrqmWbv4HHvlUs%3D&sp=rwdl&sr=c&sv=2018-11-09&timeout=240\": context canceled\n"}

[drakkan]

Hi,

can you please post more logs? It is not clear why the EOF is generated, it could be a client side issue or a server disk space issue, can you also post the server side output of the df -h command? Do you have enough disk space on the server for the temporary file? Thank you

[drakkan]

The temporary file is wrote to an unliked file inside the user home dir, since the file is unliked it is not visible but it still requires disk space until the upload ends. If you want to upload a 50GB file you need 50GB of local disk space. You can change the temporary file location as explained above

[teekeino]

Hi, I tried to fix this by mounting Azure File Share as /root in the container. After the change the container wouldn't function at all. I could not log in from Azure portal or with SFTP.

I think I have now exhausted my options with sftpgo-in-azure-container-instances.

Final conclusion: ACI is not well suited for running SFTPgo - at least when aiming to use Azure Blob directly as a back-end. Trying out FileShare as a back-end was not an option in this case.

[drakkan]

Your final conclusion seems wrong, you have to mount your file shares in /srv/sftpgo and ensure that the sftpgo user can read and write there, after doing this it should work just fine.

I have never used ACI images but I think I can fix this for you, but not for free. Anyway if you follow the above instructions you should be able to fix the configuration yourself

[teekeino]

Indeed it was wrong! I mounted the file share to /srv/sftpgo and now big uploads work as expected. Thank you very much for the support!

[drakkan]

Glad it works for your use case, if you can please consider a sponsorship this will help enabling me to spend my time working on the project. Thank you