Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve graph submission time for P2P rechunking by avoiding unpack recursion into indices #8672

Merged

Conversation

fjetter
Copy link
Member

@fjetter fjetter commented Jun 5, 2024

Closes dask/dask#11162

This is a bit annoying but it is very effective. It reduces the time to unpack the graph tasks from about 8min down to 2s

cc @dcherian @hendrikmakait

@dcherian
Copy link

dcherian commented Jun 5, 2024

hehe cool

Copy link
Contributor

github-actions bot commented Jun 5, 2024

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    29 files  ± 0      29 suites  ±0   11h 10m 42s ⏱️ + 10m 29s
 4 057 tests + 1   3 953 ✅  -  3     97 💤 ±0  7 ❌ +4 
55 883 runs  +14  53 712 ✅ +10  2 163 💤 ±0  8 ❌ +4 

For more details on these failures, see this check.

Results for commit a571fc4. ± Comparison against base commit 7cbfc4d.

@hendrikmakait
Copy link
Member

Thanks, @fjetter!

@hendrikmakait hendrikmakait merged commit 5708bdf into dask:main Jun 5, 2024
30 of 35 checks passed
@fjetter fjetter deleted the ensure_rechunk_partial_index_not_unpacked branch June 5, 2024 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

P2P rechunking of ERA-5 from spatial to temporal dimension is failing hard
3 participants