Add [pinned, unpinnned] matrix to ci_eval.yaml.

[ScottTodd]

Progress on #760. Depends on #765.

This changes ci_eval.yml (sharktank perplexity tests for llama) from testing:

1. iree perplexity (unpinned iree)
2. torch perplexity (unpinned iree-turbine)

to testing:

1. iree perplexity (unpinned iree)
2. iree perplexity (pinned iree)
3. torch perplexity (unpinned iree-turbine)

The pinned versions are shared across the project. We could individually update pins per test workflow or package, but I'd much rather keep the pins shared by default and leave fragmenting as an escape hatch as needed. I can update the other scheduled workflows to match this style if the changes here look good.

Triggered a test run here: https://github.com/nod-ai/shark-ai/actions/runs/12642858744.

[archana-ramalingam]

Progress on #760. Depends on #765.

This changes ci_eval.yml (sharktank perplexity tests for llama) from testing:

1. iree perplexity (unpinned iree)
2. torch perplexity (unpinned iree-turbine)

to testing:

1. iree perplexity (unpinned iree)
2. iree perplexity (pinned iree)
3. torch perplexity (unpinned iree-turbine)

The pinned versions are shared across the project. We could individually update pins per test workflow or package, but I'd much rather keep the pins shared by default and leave fragmenting as an escape hatch as needed. I can update the other scheduled workflows to match this style if the changes here look good.

Triggered a test run here: https://github.com/nod-ai/shark-ai/actions/runs/12642858744.

ci_eval.yaml perplexity test is intended to run on larger number of prompts with all attention versions of all Llama models, resulting in up to 6 tests in the future. Depending on the model size, each may take anywhere between 1 to 12 hours to complete. Given this is scheduled after IREE nightly release at 3 am PST, this might block dev resources in the AM. We might need to run them in dedicated CI machines, if we want to choose the matrix route.
We can do unpinned for schedule and pinned for pull_request/ push

[ScottTodd]

If we need more runners, I think we have plenty of extra machines that we can plug in. At this point, with multiple workflows unstable across unpinned versions, my main priority is getting back to a state where we know what works on which versions.

When we have more tests that take longer, we can

• add more hardware runners to take jobs
• shard tests across runners
• make the tests faster

This matrix approach is a bit inflexible (all variants run whenever the workflow is triggered, individual jobs can't be run in isolation, canceling the job cancels all variants, et c.) but it is simple to understand and add to more workflows.

[marbre]

Not sure about where to run those / the runner resources but overall looks good to me.