Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hyperloop documentation additions for requesting long train #264

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/hyperloop/operatordocumentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,15 @@ There are a number of settings that you can decide on when composing a train:
<img src="../images/cloneTrain.png" width="70%">
</div>

### <a name="request-long-train"></a>Request Long Train

* <a name="longTrain"></a>The _Request long train_ tab allows users to request a long train after the train ran on a short dataset. Short datasets are subsets of a big dataset (_set up in the Dataset settings_). First, a train run needs to be **Done on a smaller short dataset** before being run on a bigger dataset.
* When requesting a long train, it is possible to request standard derived data from a short train with slim derived data.

<div align="center">
<img src="../images/longTrainDerivedType.png" width="70%">
</div>

* Any user who is part of the analysis can request a long train. Approval from the participating analyses PWGs conveners is required in order to submit a long train. Train operators or admins can also approve a long train, but it is usually done by the PWG.

<div align="center">
Expand Down
8 changes: 8 additions & 0 deletions docs/hyperloop/userdocumentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

When opening a page in Hyperloop which has not been visited before, a guided tour will explain key concepts. These tours provide an interactive learning experience for Hyperloop, easily activated with a single click. They are ideal for beginners and for refreshing knowledge.

Where appropriate, when one tour ends, the next will begin to explain the next section of Hyperloop. Tours can be exited at any time. Once closed, they will not automatically begin on future page visits.

Check failure on line 10 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces

Remove the trailing spaces at the end of the line.

<div align="center">
<img src="../images/JoyrideWelcome.png" width="35%">
Expand Down Expand Up @@ -85,7 +85,7 @@
<div align="center">
<img src="../images/wagonShortcuts.png" width="80%">
</div>

Check failure on line 88 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces

Remove the trailing spaces at the end of the line.
## <a name="wagon-settings"></a> Wagon Settings

* <a name="wagonsettings"></a>In _Wagon settings_ you can modify the wagon name, work flow name, and select wagon's dependencies. The dependencies offered are wagons from the same _Analysis_ or from [_Service wagons_](#servicewagons).
Expand All @@ -93,7 +93,7 @@
<div align="center">
<img src="../images/wagonSettings.png" width="70%">
</div>

Check failure on line 96 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces

Remove the trailing spaces at the end of the line.
## <a name="wagon-configuration"></a> Wagon Configuration

* <a name="wagonconfiguration"></a>In _Configuration_ the wagon configuration corresponding to the workflow will be available in the _Base_. The configuration is divided per _Task_, hence if you need to add a new parameter, you will need add it in the following order: task, parameter and value.
Expand All @@ -120,26 +120,26 @@

* In order to update the base and subwagon configuration with the latest version of the workflow, click on the button `↻ sync` in _Configuration_. By synchronizing the configuration, the parameters which no longer belong to the workflow will be removed, and the values of the wagon's _Base_ will be updated as well if they have not been modified by the user.

## <a name="wagon-derived-data"></a> Derived data

Check failure on line 123 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces

Remove the trailing spaces at the end of the line.

* <a name="wagonderived"></a>In _Derived Data_ the tables which are produced by the task are displayed. If activated, these are saved to the output if the train is run as a derived data production. The produced derived data can be made available by the operators and serve as input for subsequent trains.

Check failure on line 125 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces

Remove the trailing spaces at the end of the line.

Check failure on line 126 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces

Remove the trailing spaces at the end of the line.
### <a name="deriveddatatypes"></a> Derived data types
* At the moment, there are two types of derived data specifications:
* Standard derived data (marked with 🗂️)- if the wagon is used in a train, this will produce derived data to be used for further analysis. The results will not be merged across runs and can be used as input for future train runs. Note that standard derived data trains do not submit automatically and may need additional approval. If in doubt, please seek advise before enabling derived data tables in your wagon configuration.
* Slim derived data (marked with green bordered 🗂️) - similarly to the standard derived data case, if used in a train, this will produce derived data to be used for further analysis. This is reserved for derived data of small output size. The results will be merged across runs and are not available to use in future train runs. The data will be automatically deleted after a preset period of time. You can mark a wagon for running as slim derived data by checking `Ready for slim derived data`.

Check failure on line 131 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces

Remove the trailing spaces at the end of the line.
* For wagons set as ready for slim derived data, two more fields need to be correctly set:
* Max DF size - This sets the maximal dataframe size in the merging step. Has to be 0 for not-self contained derived data (which need parent file access).
* Max derived file size - Sets the size limit for the output file size of the derived data file. This is an expert parameter which usually does not have to be changed. Only change this value if the processing in subsequent trains takes so long that the jobs fail. If set to 0 a good value will be automatically determined.

Check failure on line 135 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces

Remove the trailing spaces at the end of the line.
* In order to update the derived data configuration with the latest version of the workflow, click on the button `↻ sync` in _Derived data_. By synchronizing the derived data, the tables which no longer belong to the workflow will be removed, and the values of the tables will be updated.

<div align="center">
<img src="../images/derivedDataEx.png" width="70%">
</div>

## <a name="wagon-test-statistics"></a> Test Statistics

Check failure on line 142 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces

Remove the trailing spaces at the end of the line.

* <a name="wagonteststatistics"></a>_Test Statistics_ contains three graphs that display different metrics following the tests this wagon was part of. The first graph plots the _PSS Memory_ corresponding to each test run. The second one diplays the _CPU Time_, _Wall time_ and _Throughput_ along the test runs for this wagon. Finally, the third graph shows the _Output size_ at each test run.

Expand Down Expand Up @@ -288,7 +288,7 @@
</div>

* If you only want to see the top 10 graph with the highest average, check the Show top 10 largest box.

Check failure on line 291 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces

Remove the trailing spaces at the end of the line.
* To produce this type of performance graphs for a local O2 execution, follow the instructions [here](#producing-performance-graphs-for-a-local-o2-execution).

* Whenever a wagon configuration is changed, if there are enabled wagons (including wagons that depend on it), then the test is automatically reset and a new test is launched. However, if the enabled wagon was already composed in a train, the train will run with the wagons and dataset configuration of the time at which the train was created.
Expand Down Expand Up @@ -492,8 +492,16 @@
<img src="../images/mergedOutput1.png" width="90%">
</div>


### <a name="request-long-train"></a> Request Long Train

* <a name="longTrain"></a>The _Request long train_ tab allows users to request a long train after the train ran on a short dataset. Short datasets are subsets of a big dataset (_set up in the Dataset settings_). First, a train run needs to be **Done on a smaller short dataset** before being run on a bigger dataset.
* Any user who is part of the analysis can request a long train. Approval from the participating analyses PWGs conveners is required in order to submit a long train.
<div align="center">
<img src="../images/longTrainDerivedType.png" width="70%">
</div>

* When requesting a long train, it is possible to request standard derived data from a short train with slim derived data by changing the derived data setting as shown above.

<div align="center">
<img src="../images/requestLongTrain.png" width="70%">
Expand Down
Binary file added docs/images/longTrainDerivedType.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading