space usage / disk free / quota / etc.

[ThomasWaldmann]

This is trying to collect some related ideas and get feedback about these and maybe other options.

space usage (implementable via borgstore)

One obvious thing that can be done in relatively simple way is that the borgstore user uses .list() to get infos about all items and sums up the item sizes.

This will give the net space usage. The gross usage will be higher than that due to fs block size and other overheads.

This operation is relatively expensive if there are many items, but the borgstore user might have to iterate over all items for some other purposes anyway, so the space usage can be computed as a side effect of these other operations.

reserved allocated space (implementable via borgstore)

To avoid catastrophic "disk full" issues, borgstore users can just store some big items into the store. These can be freed in emergency situations to make free space available.

disk free (a la df / vfsstat)

This has some issues:

• vfsstat only implementable for local and sftp store
• vfsstat does not consider fs quotas
• df would require executing a shell command, parsing its output
• reported values are for the filesystem, not necessarily for the user (there might be other users using space, there might be usrquota/grpquota)

quota (as it could be implemented by borgstore)

quota use: see space usage

quota limit: this could only be implemented if there is a server side borgstore component enforcing this, that is not controlled by the client.

but as this would be implemented per borgstore, it is not necessarily very useful for storage providers if the user could create an arbitrary amount of borgstores.

quota (filesystem user/group quota)

Some filesystems on POSIX OSes like Linux support usrquota / grpquota.

quota use:

• querying usually requires executing a shell command and parsing its output, which is rather undesirable.
• there might be differences depending on the OS and tool.
• would account for fs overheads also

quota limit:

• admin can usually set soft and hard limits
• limits are per user (thus, for all borgstores of that user)

quota (cloud)

Not sure whether / how they implement that.

[ThomasWaldmann]

@m3nu can you add something from your perspective please?

[AfroThundr3007730]

Re: Reserved space

As long as this is configurable and only on user request, this could be useful. The last thing I'd want is for my repos (sometimes many of them) to become bloated with an unnecessary reservation object if I'm managing free space properly with another method. This is especially relevant if I'm paying for storage space or the network bandwidth to sync the repo elsewhere.

I'm assuming this is something borg would implement internally vice a user just adding a dummy archive for this purpose. Something like MIN(fixed_size, % of repo_size) would work, I'd think, or let the user set the size.

Re: Quotas

I know python has the FsQuota module for managing quotas on many *nix filesystems, but I'm not sure if that would be useful for Windows systems. Although shutil.disk_usage works on Windows, I'm not sure if it accounts for quotas.

Edit: There is GetDiskFreeSpaceEx, which takes quotas into account on Windows.

I know that in many cases, querying for free space as an unprivileged user returns a value taking the user quota into account, so perhaps we could just trust that in most cases, or decide based on the underlying filesystem, if known.

[ThomasWaldmann]

borg2 (master branch) has now borg repo-space <size> for that.

using it is optional and how much space (besides the granularity) is your choice.

[ThomasWaldmann]

Using python quota packages: thanks for the hints.

But that's only useful if we have server-side code running for borg (thus: currently only for file: (local) and ssh: (ssh-tunnel to remote borg, which then uses file: on remote side). But at least better than having to code it ourselves.