LLR Workflow

Contact: [email protected]

Summary

The following contains useful notes for the running of the code when the SKIMS are already available. The process is: BigNTuples → SKIMS → Histo files → plots → limits → needed stuff in general

• BigNTuples → are the NTuples coming from CERN and containing everything
• SKIMS → they are small NTuples where only the stuff that is needed for our analysis is saved (step needed just so we do not have to deal with a big amount of memory from the BigNTuples dimension)
• Histo files → .root files containing the histograms that are then used for the plotting, limit extraction, etc.

In the following the workflow is roughly explained with a bit of detail:

Prepare Big Ntuples

• See here
• Outdatet but quicker
• Uses CRAB and lxplus
• Branch 106X_HH_UL, use NtupleProducer/test/ folder
• Make sure the isMC flag is the same in NtupleProducer/test/submitAllDatasetOnCrab_LLR.py and NtupleProducer/test/analyzer_LLR.py.
• Datasets stored under NtupleProducer/test/datasets_UL18.txt
• Submission outputs store under /dpm/in2p3.fr/home/cms/trivcat/store/user/bfontana/HHNtuples_res/

Crab Submission

• Run with python2 submitAllDatasetOnCrab_LLR.py
• Grafana (Sign in below with CERN’s credentials)
• crab submit / crab submit -d <folder> / crab status

Prepare SKIM NTuples

To launch the production of skimmed samples we use:

bash scripts/submit_skims_UL18.sh -t <some_tag> --user bfontana

this calls scripts/makeListOnStorage.py (input files definition) and scripts/skimNtuple.py (skimming job submission) per sample. The configuration file is config/skims_UL18.cfg file (you may need to change some of its parameters).

One can add the option -n to avoid producing input files lists if one is sure they are up-to-date. All options can be inspected with -h. One can comment out some samples in the script to avoid skimming over all the samples. Two text files are created (unless empty) automatically: goodfiles.txt and badfile.txt. There the list of “good” or “bad/corrupted” files is stored, defined according to scripts/check_outputs.py (called within scripts/skimNtuple.py). The list of “good” files is used in subsequent analysis steps. This additional step avoids crashes due to corrupted skimmed files. Note: Some logic (a simple lock, transparent to the user) was implemented to ensure no race condition happens when jobs write to the same text files.

Job resubmission

To resubmit jobs assigned to the badfiles.txt, the user must simply issue the following command:

bash scripts/submit_skims_UL18.sh -t <some_tag> --resubmit

where the tag must be the same used in the original submission. New log files are named appropriately, so to make it clear they correspond to resubmitted jobs. Old logs are not lost. The user can resubmit the jobs as many times as needed, and the most recent badfiles.txt (with a slightly different name) is picked as input.

When the skims are prepared we have to run the calculation of the systematics on them. The process is exactly the same, but this time we run:

source scripts/submit_syst<year>.sh

Fill cfg of the samples

We fill the file config/sampleCfg_*.cfg with the paths to the various directories containing the SKIM NTuples.

• The most recent file is config/sampleCfg_UL18.cfg

Fill cfg of selections

We fill the selectionCfg_*.cfg with all the selection criteria that are decided (the weights are set in this config file). The currently most up-to-date files are:

• config/selectionCfg_ETau_UL18_template.cfg
• config/selectionCfg_MuTau_UL18_template.cfg
• config/selectionCfg_TauTau_UL18_template.cfg

Fill main cfg

We fill the mainCfg_*.cfg file in which we have to specify:

1. the samples to be included (here we have to use the aliases defined inside the samplesCfg_*.cfg file). The samples to be specified are data, signal and background
2. the variables to be plotted (these variables will have to match the variables that we are trying to plot inside the makeFinalPlots.sh) [e.g. tauH_pt ]
3. the selections to be plotted (these selections will have to match the selections that we are trying to plot inside the makeFinalPlots.sh/py) [i.e. baseline, s1b1jresolved, s2b0jresolved, sboostedLL]
4. in the [pp_QCD] section, the regions to be used for the QCD estimation with the ABCD method

The currently most up-to-date files are:

• config/mainCfg_ETau_UL18.cfg
• config/mainCfg_MuTau_UL18.cfg
• config/mainCfg_TauTau_UL18.cfg

HIstogram production

To launch the production of the histograms for all channels (example):

for i in "E" "Mu" "Tau"; do python scripts/submitHistoFiller.py --cfg config/mainCfg_${i}Tau_UL18.cfg --njobs 20 --tag <tag>; done

the number of jobs is by default set to 10. The --tag option can take any value; it is used solely for book keeping, and it will be used during the plotting step. More jobs means more output file and quicker individual jobs (up to the availability of the cluster); 20 jobs tends to complete in around 10/15 minutes. The data used corresponds to what was defined on the sampleCfg_*.cfg file.

The command will create a folder under /data_CMS/cms/${USER}/HHresonant_hist/<tag>/ in which the histograms (in ROOT format), logs and copies of used configuration files are stored. To modify the output folder use the --outdir option. If no errors are present all the logs will end with:

@@ ... saving completed, closing output file
... exiting

Each job produces a ROOT file containing histograms that must be merged (see the next step).

Check the jobs

To check if the jobs are still running or if they are done or if they broke for some reason launch one of the following:

condor_q # built-in solution, see manual
/opt/exp_soft/cms/t3/t3stat # wrapper at LLR
/opt/exp_soft/cms/t3/t3stat -q # gives only the queue

this will give a live output of the machine carrying out the jobs. Statuses:

• R: running
• Q: queueing (also known as “Idle” state)

If a job has some issue (for instance, it goes to “Hold” state) you can kill it with:

condor_rm <code_name_of_job>  # cancel signal job
condor_rm -name llrt3condor <username> # cancels all jobs under username

Combine histograms

for i in "E" "Mu" "Tau"; do python scripts/combineFillerOutputs.py --tag <tag>; done
# rm outPlotter_*.root 

where --tag is the same as the tag used in the previous step. You can use --dir in case the outputs of the previous step were not stored in the default path.

The above command will merge all the outPlotter_*.root files in a single one and will then create the analyzedOutPlotter.root file in which the histograms have been analysed for the post-production of the plots.

Final plots

To make the final plots we have to use makeFinalPlots.sh:

for i in "E" "Mu" "Tau"; do bash scripts/makeFinalPlots.sh -t <tag> -c ${i}Tau -s baseline --no-sig; done

where -t points again to the same tag as before, and -c (channel) can be “EleTau”, “MuTau” or “TauTau”. The options --no-data or --no-sig can be added to remove the corresponding contributions from the final plots. Type -h to see all available options.

Some variables are hard-coded, like the kind of selection we are using [i.e. baseline, s1b1jresolved, s2b0jresolved, sboostedLL] and which variables to plot (they match the variables that were specified in the mainCfg_*.cfg).

Many varables are also hard-coded in makeFinalPlots.py.

The plots are copied from their local storage to https://${EOS_USER}}.web.cern.ch/${EOS_USER}/HH_Plots/${TAG}/${CHANNEL}/${BASELINE}/. With some minor html/php definitions you will manage to see the plots in your browser.

Limits extraction with datacards

The limit extraction is obtained via a maximum likelihood fit. This paper summarizes well the statistical techniques employed. The most up to date limit extraction is done with this repository. The combine tool is used; you can find its documentation here.

1. make_res_cards.sh

Calls write_res_card.py, generates the datacards per channel/category/mass point. ABCD regions for QCD estimate generated separately

2. make_workspace_res.sh

Combines ABCD regions datacards, and generates workspaces for each channel/category/mass point

1) combine_res_categories.sh

Combines datacards from the 4 categories, and generates workspaces for each channel/mass point

2) combine_res_channels.sh

Combines datacards from the 3 channels, and generates workspaces for each category/mass point

3) combine_res_all.sh

Combines all datacards for the period, and generates workspaces for each mass point

3. get_limits_res.sh

Runs combine for asymptotic limits for all channel/category/mass point separately, stores result in a log file for easy limit plotting.

4. plotSimple_resMass.py