Update 141X data GTs in autoCond #46824

perrotta · 2024-11-28T14:55:27Z

Update online GTs in autoCond after the end of the 2024 data taking (to be used for tests and development):

run3_hlt: 141X_dataRun3_HLT_frozen_v2
- Same as 140X_dataRun3_HLT_v2 with snapshot at 2024-06-13 14:22:43 (UTC)
- Remove not-needed records from the HLT GT, see https://cms-talk.web.cern.ch/t/call-for-hlt-gt-record-information-by-subsystems/48795
- Add records for GEM dead and masked strips, see https://cms-talk.web.cern.ch/t/gem-request-for-including-gts-gemmaskedstrips-gemdeadstrips/55154
- Diff with respect to previous version _v1 is here (unfrozen)
run3_data_express : 141X_dataRun3_Express_frozen_v4
- Same as 141X_dataRun3_Express_v4 but with snapshot at 2024-11-28 13:23:29 (UTC)
- Add records for GEM dead and masked strips, see https://cms-talk.web.cern.ch/t/gem-request-for-including-gts-gemmaskedstrips-gemdeadstrips/55154
- Diff with respect to previous version _v3 is here (unfrozen)
run3_data_prompt : 141X_dataRun3_Prompt_frozen_v4
- Same as 141X_dataRun3_Prompt_v4 but with snapshot at 2024-11-28 13:26:44 (UTC)
- Add records for GEM dead and masked strips, see https://cms-talk.web.cern.ch/t/gem-request-for-including-gts-gemmaskedstrips-gemdeadstrips/55154
- Diff with respect to previous version _v3 is here (unfrozen)

Also the offline data GT needed to be fixed:

run3_data : 141X_dataRun3_v5
- New offline HCAL response correction production tag with fixes for era 2024C and 2024E, see https://cms-talk.web.cern.ch/t/call-for-conditions-re-reco-of-2024-eras-c-d-e/42331/25
- Add records for GEM dead and masked strips, see https://cms-talk.web.cern.ch/t/gem-request-for-including-gts-gemmaskedstrips-gemdeadstrips/55154
- Diff with respect to previous version _v4 is here

Moreover, to avoid crashes in the unit tests because of the removed (now unused) records in the HLT GT, the commit 6198154 comments out the scal_dqm_sourceclient-live_cfg.py test from the ones run in the unit tests of DQM/Integration: as correctly reported in a comment here below, the SCAL FED has been removed from DAQ since the end of Run 2.

PR validation:

The removal of the not-needed records from the HLT GT was verified online as documented in CMSALCAFAST-94
Rely on these PR tests for a wider coverage

cmsbuild · 2024-11-28T14:55:51Z

cms-bot internal usage

cmsbuild · 2024-11-28T14:57:50Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46824/42829

cmsbuild · 2024-11-28T14:58:10Z

A new Pull Request was created by @perrotta for master.

It involves the following packages:

Configuration/AlCa (alca)

@atpathak, @cmsbuild, @consuegs, @perrotta can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @missirol, @mmusich, @rsreds, @tocheng, @yuanchao this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

perrotta · 2024-11-28T15:04:42Z

please test

cmsbuild · 2024-11-28T16:48:55Z

-1

Failed Tests: UnitTests
Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f6165/43137/summary.html
COMMIT: 6198154
CMSSW: CMSSW_15_0_X_2024-11-28-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46824/43137/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test TestDQMOnlineClient-scal_dqm_sourceclient had ERRORS

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 46
DQMHistoTests: Total histograms compared: 3484682
DQMHistoTests: Total failures: 438
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3484224
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
Checked 202 log files, 172 edm output root files, 46 DQM output files
TriggerResults: no differences found

perrotta · 2024-11-28T16:51:22Z

please test with #46742

perrotta · 2024-11-28T16:52:07Z

abort
(#46742 has nothing to do with the removal of the unused records...)

cmsbuild · 2024-11-28T21:23:19Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46824/42833

There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/Integration/python/clients/scal_dqm_sourceclient-live_cfg.py modified in PR(s): Remove the configuration of the legacy CUDA workflows #46076

cmsbuild · 2024-11-28T23:12:04Z

Pull request #46824 was updated. @antoniovagnerini, @atpathak, @cmsbuild, @consuegs, @perrotta, @rseidita can you please check and sign again.

perrotta · 2024-11-28T23:19:25Z

please test

cmsbuild · 2024-11-29T02:12:16Z

-1

Failed Tests: UnitTests
Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f6165/43146/summary.html
COMMIT: b0b9c55
CMSSW: CMSSW_15_0_X_2024-11-28-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46824/43146/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test TestDQMOnlineClient-scal_dqm_sourceclient had ERRORS

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 46
DQMHistoTests: Total histograms compared: 3484682
DQMHistoTests: Total failures: 493
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3484169
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
Checked 202 log files, 172 edm output root files, 46 DQM output files
TriggerResults: no differences found

mmusich · 2024-11-29T07:17:12Z

DQM/Integration/python/clients/scal_dqm_sourceclient-live_cfg.py

@@ -76,7 +76,7 @@
 process.dqmmodules = cms.Sequence(process.dqmEnv + process.dqmSaver )#+ process.dqmSaverPB)
 process.evfDQMmodulesPath = cms.Path(
                              process.l1GtUnpack*


Hi, just a couple of quick comments:

from the exception message [1] log here , actually (also) l1GtUnpack needs to be changed.

the SCAL FED has been removed from DAQ since the end of Run 2. Do we still need to support this client? The fact that CMSALCAFAST-94 succeeded without these changes is proof that this client is not run anymore in live data-taking.

[1]

----- Begin Fatal Exception 29-Nov-2024 00:46:25 CET----------------------- An exception of category 'NoRecord' occurred while [0] Processing Event run: 355380 lumi: 19 event: 26716348 stream: 0 [1] Running path 'evfDQMmodulesPath' [2] Calling method for module L1GlobalTriggerRawToDigi/'l1GtUnpack' Exception Message: No "L1MuTriggerScalesRcd" record found in the EventSetup. Please add an ESSource or ESProducer that delivers such a record. ----- End Fatal Exception -------------------------------------------------

Thank you @mmusich
Indeed, removing that test at all would be a much simpler solution: if you also support it, I'll do so. (Now I am traveling, I will do it as soon as I will be able to)

removing that test at all would be a much simpler solution:

rather than removing the test, I would suggest to remove completely the configuration file of the client.

Yeah, that would make sense. But I'd let it be done in the proper way by @cms-sw/l1-l2 (or by whoever else wants to do it).
For the sake of just having the updated GTs running without crashes in the CMSSW unit tests system let just comment out here the obsolete one.

@cms-sw/dqm-l2 can you comment on a possible removal of the client from the release already in this PR?

cmsbuild · 2024-11-30T08:08:19Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46824/42849

cmsbuild · 2024-11-30T08:08:43Z

Pull request #46824 was updated. @antoniovagnerini, @atpathak, @cmsbuild, @consuegs, @perrotta, @rseidita can you please check and sign again.

perrotta · 2024-11-30T08:14:19Z

please test

cmsbuild · 2024-11-30T09:53:03Z

+1

Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f6165/43164/summary.html
COMMIT: 23b9148
CMSSW: CMSSW_15_0_X_2024-11-29-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46824/43164/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 46
DQMHistoTests: Total histograms compared: 3484682
DQMHistoTests: Total failures: 491
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3484171
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
Checked 202 log files, 172 edm output root files, 46 DQM output files
TriggerResults: found differences in 1 / 44 workflows

perrotta · 2024-11-30T11:54:28Z

+alca

No changes were expected because of the GT updates (HCAL response corrections in the offline GT were only updated with a small fix for eras 2024C and 2024E; all other GT updates are kind of technical), and no changes were observed in the test outputs

Update 141X data GTs in autoCond

6198154

cmsbuild added this to the CMSSW_15_0_X milestone Nov 28, 2024

cmsbuild added alca-pending pending-signatures tests-pending orp-pending code-checks-pending labels Nov 28, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 28, 2024

cmsbuild added tests-started and removed tests-pending labels Nov 28, 2024

cmsbuild added tests-rejected and removed tests-started labels Nov 28, 2024

cmsbuild added requires-external tests-started and removed tests-rejected labels Nov 28, 2024

cmsbuild added tests-pending dqm-pending code-checks-pending and removed tests-started requires-external code-checks-approved labels Nov 28, 2024

cmsbuild added the code-checks-approved label Nov 28, 2024

cmsbuild added tests-started and removed tests-pending labels Nov 28, 2024

cmsbuild added tests-rejected and removed tests-started labels Nov 29, 2024

mmusich reviewed Nov 29, 2024

View reviewed changes

Remove obsolete SCAL test

23b9148

perrotta force-pushed the updateDataGtInAutocond branch from b0b9c55 to 23b9148 Compare November 30, 2024 08:05

cmsbuild added tests-pending code-checks-pending and removed tests-rejected code-checks-approved labels Nov 30, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 30, 2024

cmsbuild added tests-started and removed tests-pending labels Nov 30, 2024

cmsbuild added tests-approved and removed tests-started labels Nov 30, 2024

cmsbuild added alca-approved and removed alca-pending labels Nov 30, 2024

This was referenced Nov 30, 2024

Update data GT in autoCond - CMSSW_14_2_X #46831

Open

Update offline data GT in autoCond - CMSSW_14_1_X #46832

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update 141X data GTs in autoCond #46824

Update 141X data GTs in autoCond #46824

perrotta commented Nov 28, 2024 •

edited

Loading

cmsbuild commented Nov 28, 2024 •

edited

Loading

cmsbuild commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

perrotta commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

perrotta commented Nov 28, 2024

perrotta commented Nov 28, 2024 •

edited

Loading

cmsbuild commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

perrotta commented Nov 28, 2024

cmsbuild commented Nov 29, 2024

mmusich Nov 29, 2024

perrotta Nov 29, 2024

mmusich Nov 29, 2024

perrotta Nov 30, 2024

mmusich Nov 30, 2024

cmsbuild commented Nov 30, 2024

cmsbuild commented Nov 30, 2024

perrotta commented Nov 30, 2024

cmsbuild commented Nov 30, 2024

perrotta commented Nov 30, 2024

Update 141X data GTs in autoCond #46824

Are you sure you want to change the base?

Update 141X data GTs in autoCond #46824

Conversation

perrotta commented Nov 28, 2024 • edited Loading

PR validation:

cmsbuild commented Nov 28, 2024 • edited Loading

cmsbuild commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

perrotta commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

Unit Tests

Comparison Summary

perrotta commented Nov 28, 2024

perrotta commented Nov 28, 2024 • edited Loading

cmsbuild commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

perrotta commented Nov 28, 2024

cmsbuild commented Nov 29, 2024

Unit Tests

Comparison Summary

mmusich Nov 29, 2024

Choose a reason for hiding this comment

perrotta Nov 29, 2024

Choose a reason for hiding this comment

mmusich Nov 29, 2024

Choose a reason for hiding this comment

perrotta Nov 30, 2024

Choose a reason for hiding this comment

mmusich Nov 30, 2024

Choose a reason for hiding this comment

cmsbuild commented Nov 30, 2024

cmsbuild commented Nov 30, 2024

perrotta commented Nov 30, 2024

cmsbuild commented Nov 30, 2024

Comparison Summary

perrotta commented Nov 30, 2024

perrotta commented Nov 28, 2024 •

edited

Loading

cmsbuild commented Nov 28, 2024 •

edited

Loading

perrotta commented Nov 28, 2024 •

edited

Loading