Tolerate empty vectors of shuffle intermediates #5163

Job	Run time
Additional Builds and Concurrency Tests	14m 5s
Basic Checks	11m 58s
Measure coverage	9m 23s
Release builds and tests	8m 51s
miri	3m 10s
sanitize (address)	19m 35s
Slow tests	12m 0s
sanitize (address, multi-threading)	19m 3s
sanitize (leak)	8m 14s
sanitize (leak, multi-threading)	8m 12s
	1h 54m 31s

Provide feedback