Immense amount of memory used by webdav #126
Comments
|
Thanks! @dsonck92, let me take a look at this, I can work on memory optimization for both WebDAV and API. |
|
Sounds good! This is currently a little set-back for me adopting this project currently, as it makes the cluster as unstable as nextcloud does. (This is also partly because I don't set memory limits, which I should probably do for these 2 services so at least they get OOM-killed if they step out of line, in the hope of slowly recovering to a working state). However I have full trust in that we can optimize this away, so I don't see this as a blocker. And the direct API uploads work fine, which is always a fallback. Extra thought: maybe it would be interesting to contribute to rclone a Voltashare backend, that can directly talk to the API. I understood rclone is also written in go, so perhaps creating a go library that abstracts away the API could give Voltashare the power that rclone can give: syncing, fs mounting, copying to/from any supported provider (e.g. migrate away from s3/nextcloud/google drive, etc) |
|
I'm very good at memory optimizations, it's one of my biggest strengths (partly because I'm coming from a native C/C++ background). So you can trust me on fixing this very soon! I'm setting this as my number 1 priority and I start working on it immediately. I kinda also have fun troubleshooting such issues. +1 for the idea of contributing a Voltaserve backend to rclone, it's always better to integrate and extend existing open source solutions. |
|
Yes, I can surely confirm there must be a memory leak somewhere :D
This is a idling instance if it helps you @bouassaba. |
|
Now let me explain the situation :) basically, we just hit the limitations of the WebDAV standard, this is a known thing, nothing new, that's why:
So what shall we do? We take the standard, which is WebDAV, and we think it differently. Turns out yes we can! So we are introducing a Fusion ⚛️ of WebDAV + S3 + Redis, and guess what? the result is stunning! 😎 I'm uploading 10 movies of a total of worth 12 GB storage, the memory usage of Here is the PR 🧑💻 #129 EDIT: In a nutshell, the idea is, during WebDAV's PUT operation, we bypass the traffic of microservice communication between |
|
@dsonck92 and @loboda4450 let me know your feedback about this :) |
|
This sounds really great, personally I really need to have efficient big file support, especially uploading, and reliable replacement as well. Nextcloud happens to accept the initial upload, but if it is changed, and the upload fails, it gets stuck with a lock file that needs to pass (or a manual file deletion on the server is required, removing history) EDIT: So it definitely sounds like Voltashare will shine here |
|
@dsonck92 regarding replacement, Voltaserve rocks there, it's smooth and automatically creates snapshots when you replace, whether via WebDAV or via the UI - yes you can replace via the UI too :) just right click the file and click upload on the context menu. |
|
@dsonck92 @loboda4450 as the new WebDAV is merged, I'm looking forward to your feedback ;) |
|
I will try it out, as the current typescript I'm still running, really doesn't like my rclone from nextcloud to voltaserve |
|
@bouassaba the hełm chart that I use must be fitted to new code, I will give my feedback asap. |
|
Currently, when trying to upload using rclone, it seems to get stuck: |
Let me do some tests with rclone and get back to you. |
|
That, is a good point, I think I haven't. As the pull policy isn't set to always yet |
|
Yes, I think it is functional now, I have to be careful as it also relies on my NextCloud install being functional |
|
Great to hear! so how is it the performance now? |
It managed to upload a 2GiB MOV file just fine, which also now previews on the web interface |
|
I do have to say that it still gains memory over time, but I think this will be squashed with the suggestions from the linters. There are a lot of missing EDIT: from what I remember, MinIO depending on how you upload in particular can leak, as it spawns a Go routine which is an anchor for the garbage collector to keep memory. |
|
To debug the above, the perf endpoints could be useful, which are go's internal profiling features. |
|
I will then deploy this on minikube to get more insights on memory usage (like history) - Docker compose is not helping me much. |
|
I did more heaving testing, I uploaded 55 GB worth of movies, continuously, without restarting the services, and the memory usage stays stable: I'm running all this bare Metal on an Apple M3, could it be some weirdness due to how the binaries behave in different architectures and operating systems? In my case Arm/macOS, and in your case I suppose x86/Linux? EDIT: I forgot to mention that I use Cyberduck as a WebDAV client. |
|
Yes, x86/Linux. Oh wait, there's one potential thing here. In Kubernetes, things written out locally might end up on tmpfs, which probably also contributes. Though I'm not too knowledgeable how this is reflected in |
|
@bouassaba well, now it actually stopped responding altogether, even to a simple |
|
No, there is no locking mechanism, everything is stateless. |
|
I'm wondering if it might be because it can't reach the api server somehow. Also, the gosec linter warned about webdav utilizing the |
|
@dsonck92 thanks for the log, yes it could be, let me have a look at what |
|
Yes, this is exactly what some linters will automate for us. In #140 I proposed a fix |
|
I will close this as resolved, because we did a good amount of testing for it. |
As I was playing with rclone to upload my existing Nextcloud synced folders to Voltaserve, I noticed the WebDav service is not very memory optimized. I attempted to upload my folder with several large (>5GiB) files, and noticed that the WebDav container ended up eating almost 21GiB (!). Uploading directly using the UI worked.
After a restart of the pod it went down to an acceptable 174MiB
I'm also noticing the API is still relatively heavy on memory, so it might not release all memory once uploads have been finished.
The text was updated successfully, but these errors were encountered: