Fix norm uncertainties

[davidwalter2]

Previously norm uncertainties were added via "addLnNSystematic" but not propagated into the sideband regions for the fake estimation.

• The LnN systematics are now removed and norm uncertainties are instead added as shape uncertainties (that was already the case before once converted to hdf5 tensor). Now the same interface is used, via "addSystematic". I validated that it is equivalent for the Z, some differences are expected for the W.

• The luminosity histogram for the W is removed as well and this variation is added in the same way with "addSystematic" in setupCombine on the fly.

  • To maintain consistency with the unblinded results, the mirroring is done in the linear scale as it was done before (and not in log scale that would be more consistent with the way the luminosity uncertainty is defined for the Z i.e. as LnN uncertainty)

Fixing the combined W and Z fit in the CI

[bendavid]

Since I believe this is the first PR which changes the nominal result we should think about whether we want to make a branch or such before merging it.

[davidwalter2]

Since I believe this is the first PR which changes the nominal result we should think about whether we want to make a branch or such before merging it.

A branch for where we keep the nominal result unchanged or a branch where changes go?

[bendavid]

for right this minute (until we finish with the supplemental plots with helicity xsecs etc) it would be better to leave the nominal analysis unchanged. The easiest way is probably to set propagateToFakes=False for these particular systematics such that the old behaviour is preserved

[bendavid]

Is it understood why there are still numerical differences with respect to the reference?

The constructed shape uncertainties should be really exactly equivalent to the lnN no?

[davidwalter2]

Is it understood why there are still numerical differences with respect to the reference?

The constructed shape uncertainties should be really exactly equivalent to the lnN no?

Yes there are small numerical differences because previously a histogram was filled in the histmaker with the event weights scaled up and down according to the luminosity uncertainty. Now the nominal histogram is used and scaled up and mirrored at the setupCombine.py step.

[davidwalter2]

Can this be merged?

[kdlong]

Did you test the impact on the result with high stats? If we're going to do additional studies based on reviewer comments, we probably don't want the central value to change.

[bendavid]

I also don't really understand why this change should induce even visible numerical differences, since the shape variation should just be a simple scaling of the histogram.

[davidwalter2]

Here are the results from running the workflow dispatch:
https://cmsmwbot.web.cern.ch/WMassAnalysis/PRValidation/PR241114_fixNormUnc/2024_12_04/
compared to scheduled build:
https://cmsmwbot.web.cern.ch/WMassAnalysis/PRValidation/ScheduledBuilds/2024_12_03_2cda170/
So there are a few cases in the impact plots where the ordering or the last digit flips.
What do you think?

[davidwalter2]

I was able to trace back the small differences. In the W case previously there was a histogram provided with the lumi varied up and down by multiplying the event weight w_up = w * factor and w_down = w / factor. Then the variations were symmetrized by "average" in the log space.
In this PR the up histogram is mirrored in the log space directly (i.e. as lnN systematic as it is already the case for the Z) leading to a different result.
To maintain same results with the unblinded results I added the old logic for the W and the new for the Z here:
https://github.com/davidwalter2/WRemnants/blob/241114_fixNormUnc/scripts/combine/setupCombine.py#L1622-L1677
In the future when we decide to give up maintaining the unblinded results we can simply remove the first block and tread the lumi uncertainty consistently as lnN systematic everywhere.

The PR gives now consistent results with the reference CI and is from my side ready to be merged.
This PR: https://cmsmwbot.web.cern.ch/WMassAnalysis/PRValidation/PR562/2025_01_03/
Reference: https://cmsmwbot.web.cern.ch/WMassAnalysis/PRValidation/ReferenceRuns/2025_01_02_97f5005/