Run CMSSW validation of the 2024 attention model

[jpata]

Following the PF validation documentation in https://github.com/jpata/particleflow/blob/main/mlpf/data_cms/README.md, rerun the CMSSW validation on:

• the ACAT 2022 model: https://huggingface.co/jpata/particleflow/tree/main/cms/acat2022_20221004_model40M
• the new model from https://huggingface.co/jpata/particleflow/tree/main/cms/2024_05_16_attn_model21M, Full CMS pytorch training in May 2024 #316
  and compare the results.

We can use the high-stats PF validation to produce plots like this: https://jpata.web.cern.ch/jpata/mlpf/results/acat2022_20221004_model40M/ (old model from 2022).

Currently there is a blocker that the PF validation dataset is apparently only on FNAL disk and not accessible.
I'm in the process of transferring it temporarily to T2_EE_Estonia to make this validation possible.
https://cms-rucio-webui.cern.ch/rule?rule_id=24d4a7bf78ae4816a17136c8d33e1395

[jpata]

Looks like older relvals are agressively deleted these days so the usual PF validation workflows don't run any more.

Luckily I have these still at T2_EE_Estonia:

/store/relval/CMSSW_12_3_0_pre6/RelValQCD_FlatPt_15_3000HS_14/GEN-SIM-DIGI-RAW/PU_123X_mcRun3_2021_realistic_v11-v1/10000

which I think should work.

[jpata]

I redid the validation on top of the sample using the ACAT2022 model and updated the recipe in
#322.

The next step is to put in the new pytorch->ONNX model: #216.

Then the validation can be redone and compared to the ACAT2022 model.

[jpata]

The new model required a newer onnxruntime, I had to update the CMSSW validation recipe in #323.

[jpata]

The jobs with the new model and CMSSW_14_1_0 are now running on CPU at T2_EE_Estonia.

[jpata]

The validations are available at https://indico.cern.ch/event/1426959/#2-cms-status-and-plans, some relevant code changes were done in #330.