reduce cost of large variant matrix

[minrk]

Description

when variant matrix is large and mostly unused (as in conda-forge), the length of input_variants may be several thousand (13,824 in the case of petsc4py) when only a few are actually used.

This causes get_loop_vars and metadata.copy() to become very expensive and dominate render time.

This reduction cuts time spent in render_recipe for petsc4py from over 2 minutes to 40 seconds to produce 72 actual variants:

before:

after:

(result is unchanged)

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

[codspeed-hq]

CodSpeed Performance Report

Merging #5392 will improve performances by ×2.6

_{Comparing minrk:reduce_variants (311e48b) with main (433f048)}

Summary

⚡ 1 improvements
✅ 3 untouched benchmarks

Benchmarks breakdown

	Benchmark	main	minrk:reduce_variants	Change
⚡	test_render_recipe	64.7 s	25.1 s	×2.6

[minrk]

need to investigate why the conda-build tests produce different results with actual runs of conda-build, all of which seem to produce the right variants.

[minrk]

seems to be the exclusion of get_used_vars doesn't actually cover all used variables. And some tests appear to expect matrix entries for explicitly unused variables (bzip2 in test_setting_condarc_vars_with_env_var_expansion), which makes me think that perhaps this should be a conda-smithy thing and not a conda-build thing.

[beeankha]

pre-commit.ci autofix

[minrk]

config.copy on the line before already does exactly this, no need to do it twice

[minrk]

get_loop_vars is far cheaper if we pass a subset to consider instead of computing the (usually quite small) intersection after looping over all variables across all variants.

[minrk]

I've taken a different approach that doesn't modify the variants list at all, so shouldn't have any consequences besides performance. Instead of reducing the actual variants list, I've reduced the cost of the two dominant operations on the large variant list:

• defer copying metadata.config.variants for each top_level variant until after it's been filtered, so far fewer variant dicts are copied (each variant should be copied exactly once, I believe, rather than once per top_level loop)
• in get_used_loop_vars, compute used_vars first and pass it to get_loop_vars so only the used subset are considered, rather than comparing all keys across all variants every time.

The first changes the variants copy from O(top level variants * input_variants) to O(top_level_variants * per_top_level_variants), which is the same as O(input_variants). So the reduction is equal to the number of top level variants; in the case of petsc4py on linux-64, that's 72 * 13,824 = 995,328, reduced to 72 * 192 = 13,824, a reduction of 98.6%. In the case of conda-forge, the real used number of variants is 72, so there's still a further 99.5% that the original proposal reduced by, but I don't know how to do that safely while maintaining all the unspecified guarantees in conda-build.

The second changes the get_loop_vars call from O(variant keys * input variants) to O(used vars * input variants), in the case of petsc4py that's ~365 -> 14, a reduction of 96%:

The savings aren't quite what they are for actually reducing the variants list because there are still some operations on the full list but it still cuts render time in half.

[minrk]

I'm afraid I don't understand the mac failures or how they could be related to this PR. The same tests pass just fine on my mac. Hopefully something transient?

[beckermr]

Those mac failures do look unrelated. Let's try to rerun before we dig into them.

[beckermr]

@mbargull Can you take a look at this one? I don't see any obvious problems but this work gets a bit into the guts of the code.

[minrk]

Anyone have a chance to look at this?

[beckermr]

@isuruf @beeankha @kenodegard Can we merge this one for the 24.9 release?