Reduce memory footprint of culling P2P rechunking

[hendrikmakait]

Previously, culling appeared to have an exponentially-growing memory footprint, this PR fixes that.

• Tests added / passed
• Passes pre-commit run --all-files

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    25 files  ±0      25 suites  ±0   10h 15m 38s ⏱️ - 6m 8s
 4 122 tests ±0   4 002 ✅  - 2    111 💤 ±0   9 ❌ +2 
47 615 runs  ±0  45 498 ✅  - 3  2 107 💤 ±0  10 ❌ +3 

For more details on these failures, see this check.

Results for commit 4db28ae. ± Comparison against base commit 4aeed40.

♻️ This comment has been updated with latest results.

[jrbourbeau]

Thanks @hendrikmakait, I can confirm this drastically reduces the memory required to kick off some of the large ERA5 rechunking jobs I've been running lately. (@dcherian you may also find this one interesting)

[dcherian]

you guys are killing it :)

[hendrikmakait]

Benchmarking on Coiled showed no impact on performance (https://github.com/coiled/benchmarks/actions/runs/10628454825). Note that we do not measure client-side memory.

[phofl]

thanks!