Tumbleweed Pool delete quota error

[phillxnet]

When deleting a freshly imported Pool with Shares in Tumblweed (newer kernel & btrfs), for each Share on the Pool we receive a blocker error.

I.e.

Our regular warning of:

Delete Pool (rock-pool2)
Pool and all associated Shares, Snapshots, NFS/Samba/SFTP exports, and Replicas, will be deleted.
Are you sure? DATA WILL BE LOST!

plus:

Rockstor managed Shares:

test_share01 (5 GB)
test_share02 (1 GB

Pool Delete attempt one:

Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 40, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/pool.py", line 793, in delete
    remove_share(so.pool, so.subvol_name, so.pqgroup, force=force)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1196, in remove_share
    qgroup_destroy(qgroup, root_pool_mnt)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1501, in qgroup_destroy
    return run_command([BTRFS, "qgroup", "destroy", qid, mnt_pt], log=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/src/rockstor/system/osi.py", line 289, in run_command
    raise CommandException(cmd, out, err, rc)
system.exceptions.CommandException: Error running a command. cmd = /usr/sbin/btrfs qgroup destroy 0/258 /mnt2/rock-pool2. rc = 1. stdout = ['']. stderr = ['ERROR: unable to destroy quota group: Device or resource busy', '']

But the associated Share will now have been removed regardless. A subsequent error for every other Share will then follow on each subsequent Pool delete attempt:

Pool Delete attempt two:

Regular warning plus this time:

Rockstor managed Shares:

test_share02 (1 GB)

Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 40, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/pool.py", line 793, in delete
    remove_share(so.pool, so.subvol_name, so.pqgroup, force=force)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1196, in remove_share
    qgroup_destroy(qgroup, root_pool_mnt)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1501, in qgroup_destroy
    return run_command([BTRFS, "qgroup", "destroy", qid, mnt_pt], log=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/src/rockstor/system/osi.py", line 289, in run_command
    raise CommandException(cmd, out, err, rc)
system.exceptions.CommandException: Error running a command. cmd = /usr/sbin/btrfs qgroup destroy 0/260 /mnt2/rock-pool2. rc = 1. stdout = ['']. stderr = ['ERROR: unable to destroy quota group: Device or resource busy', '']

Third Pool delete attempt:

Rockstor managed Shares:

I.e. no shares now. And the Pool delete succeeds.

[phillxnet]

@Hooverdan96 I'm wondering if this may be another angle on your observation of our possibly not tending to quota group deletes properly: which may now have surfaced with newer kernels/filesystem subsystems, i.e. as per your comments in the following replication issue:

• btrfs send/receive - Failure to destroy quota group can prevent promoting oldest snapshot during replication btrfs send/receive - Failure to destroy quota group can prevent promoting oldest snapshot during replication #2901

e.g. from that issue:

['ERROR: unable to destroy quota group: Device or resource busy', '']

[Hooverdan96]

Do you think, it might be a timing issue, i.e. one should wait until other processes finish before deleting the quota group? Or the sequence of deletion might need to be changed?

[phillxnet]

@Hooverdan96 I don't think it is a waiting thing. But agreed that we likely need to change the sequence, or rather add a quota delete to it. In the test cases there was almost no data/quota load. So that may be relevant. However this same reproducer test did not exhibit this issue on a 15.6 host. So very likely a change in behaviour (more strict) in TW than in Leap 15.6 that we need to adapt to some how.

Once the new installers are out this should be trivial to reproduce - especially give this was near out-of-the-box behaviour. But it was directly after a prior Pool import.