Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

episode7: fix content for REANA 0.9.1 #31

Merged
merged 1 commit into from
Oct 12, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 37 additions & 16 deletions _episodes/07-higgstotautau-parallel.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,10 @@ example via the scatter-gather paradigm introduced in the previous episode.
The overall ``reana.yaml`` looks like:

```yaml
version: 0.6.0
inputs:
files:
- steps.yaml
- workflow.yaml
parameters:
files:
- root://eospublic.cern.ch//eos/root-eos/HiggsTauTauReduced/GluGluToHToTauTau.root
Expand Down Expand Up @@ -65,6 +67,7 @@ outputs:
files:
- fit/fit.png
```
{: .source}

Note that we define input files and cross sections and short names as an array. It is this array
that we shall be scattering around.
Expand All @@ -87,6 +90,7 @@ The skimming step definition looks like:
parameters: [input_file, cross_section]
step: {$ref: 'steps.yaml#/skim'}
```
{: .source}

where the step is defined as:

Expand All @@ -105,6 +109,7 @@ skim:
publish:
skimmed_file: '{output_file}'
```
{: .source}

Note the scatter paradigm that will cause nine parallel jobs for each input dataset file.

Expand All @@ -126,6 +131,7 @@ The histograms can be produced as follows:
parameters: [input_file, output_names]
step: {$ref: 'steps.yaml#/histogram'}
```
{: .source}

with:

Expand All @@ -147,6 +153,7 @@ histogram:
publish:
histogram_file: '{output_dir}/*.root'
```
{: .source}

## HiggsToTauTau merging

Expand All @@ -162,6 +169,7 @@ Gather time! How do we merge scattered results?
output_file: '{workdir}/merged.root'
step: {$ref: 'steps.yaml#/merge'}
```
{: .source}

with:

Expand All @@ -180,6 +188,7 @@ merge:
publish:
merged_file: '{output_file}'
```
{: .source}

## HiggsToTauTau fitting

Expand All @@ -195,6 +204,7 @@ The fit can be performed as follows:
fit_outputs: '{workdir}'
step: {$ref: 'steps.yaml#/fit'}
```
{: .source}

with:

Expand All @@ -213,6 +223,7 @@ fit:
publish:
fit_results: '{fit_outputs}/fit.png'
```
{: .source}

## HiggsToTauTau plotting

Expand All @@ -226,6 +237,8 @@ Challenge time! Add plotting step to the workflow.

> ## Solution
>
> The addition to the worklow specification is:
>
> ```yaml
> - name: plot
> dependencies: [merge]
Expand All @@ -236,8 +249,9 @@ Challenge time! Add plotting step to the workflow.
> plot_outputs: '{workdir}'
> step: {$ref: 'steps.yaml#/plot'}
> ```
> {: .source}
>
> with:
> The step is being defined as:
>
> ```yaml
> plot:
Expand All @@ -254,33 +268,29 @@ Challenge time! Add plotting step to the workflow.
> publish:
> fitting_plot: '{plot_outputs}'
> ```
>
> {: .source}
{: .solution}

## Full workflow

Assembling the previous stages visually, the full workflow looks like:

<img src="{{ page.root }}/fig/awesome-analysis-yadage-parallel/workflow.png" />

## Running full workflow

We are now ready to run the example of REANA cloud.
We are now ready to assemble the previous stages together and run the example on the REANA cloud.

> ## Exercise
>
> Run HiggsToTauTau parallel workflow on REANA cloud. How many job does the workflow have? How
> much faster it is executed when compared to the simple Serial version?
> Write and run the HiggsToTauTau parallel workflow on REANA cloud. How many job does the workflow
> have? How much faster it is executed when compared to the simple Serial version?
>
{: .challenge}

> ## Solution
>
> reana.yaml:
> The REANA specification file `reana.yaml` looks as follows:
>
> ```yaml
> version: 0.6.0
> inputs:
> files:
> - steps.yaml
> - workflow.yaml
> parameters:
> files:
> - root://eospublic.cern.ch//eos/root-eos/HiggsTauTauReduced/GluGluToHToTauTau.root
Expand Down Expand Up @@ -319,8 +329,9 @@ We are now ready to run the example of REANA cloud.
> files:
> - fit/fit.png
> ```
> {: .source}
>
> workflow.yaml:
> The workflow definition file `workflow.yaml` is:
>
> ```yaml
> stages:
Expand Down Expand Up @@ -378,7 +389,7 @@ We are now ready to run the example of REANA cloud.
> step: {$ref: 'steps.yaml#/plot'}
> ```
>
> steps.yaml:
> The workflow steps defined in `steps.yaml` are:
>
> ```yaml
> skim:
Expand Down Expand Up @@ -456,4 +467,14 @@ We are now ready to run the example of REANA cloud.
> ```
{: .solution}

## Results

The computational graph of the workflow looks like:

<img src="{{ page.root }}/fig/awesome-analysis-yadage-parallel/workflow.png" />

The workflow produces the following fit:

<img src="{{ page.root }}/fig/awesome-analysis-yadage-parallel/fit.png" />

{% include links.md %}
Binary file added fig/awesome-analysis-yadage-parallel/fit.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 4 additions & 2 deletions fig/awesome-analysis-yadage-parallel/reana.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
version: 0.6.0
inputs:
files:
- steps.yaml
- workflow.yaml
parameters:
files:
- root://eospublic.cern.ch//eos/root-eos/HiggsTauTauReduced/GluGluToHToTauTau.root
Expand Down Expand Up @@ -36,4 +38,4 @@ workflow:
file: workflow.yaml
outputs:
files:
- outputs/statanalysis/fitresults/pre.png
- fit/fit.png
8 changes: 4 additions & 4 deletions fig/awesome-analysis-yadage-parallel/steps.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ skim:
process:
process_type: 'interpolated-script-cmd'
script: |
./skim {input_file} {output_file} {cross_section} 11467.0 0.1
./skim {input_file} {output_file} {cross_section} 11467.0 0.1
environment:
environment_type: 'docker-encapsulated'
image: gitlab-registry.cern.ch/awesome-workshop/awesome-analysis-eventselection-stage3
Expand All @@ -11,7 +11,7 @@ skim:
publisher_type: interpolated-pub
publish:
skimmed_file: '{output_file}'

histogram:
process:
process_type: 'interpolated-script-cmd'
Expand Down Expand Up @@ -42,7 +42,7 @@ merge:
publisher_type: interpolated-pub
publish:
merged_file: '{output_file}'

fit:
process:
process_type: 'interpolated-script-cmd'
Expand All @@ -56,7 +56,7 @@ fit:
publisher_type: interpolated-pub
publish:
fit_results: '{fit_outputs}/fit.png'

plot:
process:
process_type: 'interpolated-script-cmd'
Expand Down