Support for S3 Directory analysis for #3

[vjeeva]

Completely left that out!!! We need this. Trying to avoid hooking up S3 as a mount... My assumption already is that we are mounting the directory to be analyzed... Oops.

However, when moving to Kubernetes #7 it will be needed to have some type of network mount to decentralize the system. I already had Amazon EFS in mind for that (gonna need to add support for any other network mounts too eg SMB, etc).

This needs some thought. The issue is that the code assumes the backup paths are in the docker container /mount folder, which is mounted when the ui container is run. Can't do that with S3, and there are som solutions but many people advise against using S3 as a file system.

[vjeeva]

https://github.com/s3fs-fuse/s3fs-fuse. As long as it's read only I think should be fine. Just gotta ensure it IS or CAN BE SET read-only.

This way we can keep our dumps in S3 and keep EFS for Kubernetes decentralization only, reducing costs and unneeded replication.

[vjeeva]

Need to think about multiple buckets and multiple folders. They need to be mounted at the docker run level.

[vjeeva]

Creating mounts per s3 bucket. For local, just have a general mount point then the config will have to point from there to the backups per config. Rather than mount multiple things, have it in one directory. I could make it multiple like the s3 mounts but that would complicate the config file for local too.

[vjeeva]

Screw mounts. The software s3fs was too intensive, made the container too big. Rather just get an s3 ls written instead. Got a system for that written, generalized for any new transports.

[vjeeva]

Merged, but I didn't test with an s3 backup running yet. The directory listing works though!

Almost working, need to fix my own AWS access to test fully. Once done, will close this ticket.

[vjeeva]

Gonna take way longer than I thought. Closing, I'll reopen if it's an issue.