Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename linked datasets to short datasets in Hyperloop documentation #253

Merged
merged 3 commits into from
Apr 18, 2024
Merged

Rename linked datasets to short datasets in Hyperloop documentation #253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f586011
Update userdocumentation.md
nicolaspoffley Apr 17, 2024
8ba7015
Update operatordocumentation.md
nicolaspoffley Apr 17, 2024
d735c32
Update notifications.md
nicolaspoffley Apr 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/hyperloop/notifications.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
* Notifies the user that an update has been made to one of the datasets included in his or her analyses. The notification message contains a link to the history of the dataset that has been modified, as well as a link to the analysis this dataset is included in. The update here indicates that:
* The list of runs to be excluded in a production has been modified
* The dataset has been staged or unstaged to a target
* A linked dataset had been added or removed
* A short dataset had been added or removed
* The automatic composition settings have changed, e.g. the schedule

<div align="center">
Expand Down Expand Up @@ -74,9 +74,9 @@
<img src="../images/mergelistUpdate.png" width="90%">
</div>

## <a name="linkedDataset"></a>Linked datasets
## <a name="linkedDataset"></a>Short datasets

* When a dataset has linked datasets, it cannot be part of an analysis. If a dataset was part of an analysis, and then the dataset is modified to have linked datasets, then the dataset is removed from the analysis. If there were enabled wagons, they are disabled.
* When a dataset has short datasets, it cannot be part of an analysis. If a dataset was part of an analysis, and then the dataset is modified to have short datasets, then the dataset is removed from the analysis. If there were enabled wagons, they are disabled.

## <a name="infoWagonDisabled"></a>Disabled wagon

Expand Down Expand Up @@ -107,7 +107,7 @@
<div align="center">
<img src="../images/wagonDisabled.png" width="90%">
</div>

Check failure on line 110 in docs/hyperloop/notifications.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces 

Remove the trailing spaces at the end of the line.
* The wagon is disabled if the workflow name has been changed in the meantime. This is fixed by updating the workflow name in the wagon configuration.

<div align="center">
Expand Down
8 changes: 4 additions & 4 deletions docs/hyperloop/operatordocumentation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,8 @@ There are a number of settings that you can decide on when composing a train:

### <a name="stagedsubmission"></a>Staged Submission

* Linked datasets are subsets of a big dataset
* First, a train run needs to be _Done_ on a smaller linked dataset before being run on a bigger dataset
* Short datasets are subsets of a big dataset
* First, a train run needs to be _Done_ on a smaller short dataset before being run on a bigger dataset
* Any user who is part of the analysis can **request a long train**, by going on the train result to the _Request long train_ tab
* Choose the dataset, target and wagons you want to include and click the _Compose and request approval_ button.
* Approval from the participating analyses PWGs conveners is required in order to submit a long train
Expand Down Expand Up @@ -189,7 +189,7 @@ There are a number of settings that you can decide on when composing a train:
<img src="../images/cloneTrain.png" width="70%">
</div>

* <a name="longTrain"></a>The _Request long train_ tab allows users to request a long train after the train ran on a linked dataset. Linked datasets are subsets of a big dataset (_set up in the Dataset settings_). First, a train run needs to be **Done on a smaller linked dataset** before being run on a bigger dataset.
* <a name="longTrain"></a>The _Request long train_ tab allows users to request a long train after the train ran on a short dataset. Short datasets are subsets of a big dataset (_set up in the Dataset settings_). First, a train run needs to be **Done on a smaller short dataset** before being run on a bigger dataset.
* Any user who is part of the analysis can request a long train. Approval from the participating analyses PWGs conveners is required in order to submit a long train. Train operators or admins can also approve a long train, but it is usually done by the PWG.

<div align="center">
Expand Down Expand Up @@ -246,7 +246,7 @@ There are a number of settings that you can decide on when composing a train:
<img src="../images/editDatasetOptions.png" width="70%">
</div>

* <a name="datasetoptions"></a>In the **Options** box, you can add linked datasets to the current dataset, which will be used for the [**staged submission**](#stagedsubmission). Enabling _Run final merging over all runs in this dataset_ will merge all the runs of all the productions during the final merging.
* <a name="datasetoptions"></a>In the **Options** box, you can add short datasets to the current dataset, which will be used for the [**staged submission**](#stagedsubmission). Enabling _Run final merging over all runs in this dataset_ will merge all the runs of all the productions during the final merging.
* <a name="datasetstaging"></a>In the **Analysis Facility Staging**, the operator is able to stage or unstage the data to the available targets displayed in the dropdown.
* Choose a target from the dropdown list, and you will be notified of the amount of data required for the staging process. Confirm by clicking OK. The staging process will start once clicking the _Save all changes_ button at the top right of the page.
* You can hover over the staging percentage to get a detailed status of the staging progress. By clicking on the percentage, you will be lead to a new tab where you can see the elaborate overview of the transfer requests on MonALISA.
Expand Down
2 changes: 1 addition & 1 deletion docs/hyperloop/userdocumentation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

When opening a page in Hyperloop which has not been visited before, a guided tour will explain key concepts. These tours provide an interactive learning experience for Hyperloop, easily activated with a single click. They are ideal for beginners and for refreshing knowledge.

Where appropriate, when one tour ends, the next will begin to explain the next section of Hyperloop. Tours can be exited at any time. Once closed, they will not automatically begin on future page visits.

Check failure on line 10 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces 

Remove the trailing spaces at the end of the line.

<div align="center">
<img src="../images/JoyrideWelcome.png" width="35%">
Expand Down Expand Up @@ -85,7 +85,7 @@
<div align="center">
<img src="../images/wagonShortcuts.png" width="80%">
</div>

Check failure on line 88 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces 

Remove the trailing spaces at the end of the line.
## <a name="wagon-settings"></a> Wagon Settings

* <a name="wagonsettings"></a>In _Wagon settings_ you can modify the wagon name, work flow name, and select wagon's dependencies. The dependencies offered are wagons from the same _Analysis_ or from [_Service wagons_](#servicewagons).
Expand All @@ -93,7 +93,7 @@
<div align="center">
<img src="../images/wagonSettings.png" width="70%">
</div>

Check failure on line 96 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces 

Remove the trailing spaces at the end of the line.
## <a name="wagon-configuration"></a> Wagon Configuration

* <a name="wagonconfiguration"></a>In _Configuration_ the wagon configuration corresponding to the workflow will be available in the _Base_. The configuration is divided per _Task_, hence if you need to add a new parameter, you will need add it in the following order: task, parameter and value.
Expand All @@ -120,26 +120,26 @@

* In order to update the base and subwagon configuration with the latest version of the workflow, click on the button `↻ sync` in _Configuration_. By synchronizing the configuration, the parameters which no longer belong to the workflow will be removed, and the values of the wagon's _Base_ will be updated as well if they have not been modified by the user.

## <a name="wagon-derived-data"></a> Derived data

Check failure on line 123 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces 

Remove the trailing spaces at the end of the line.

* <a name="wagonderived"></a>In _Derived Data_ the tables which are produced by the task are displayed. If activated, these are saved to the output if the train is run as a derived data production. The produced derived data can be made available by the operators and serve as input for subsequent trains.

Check failure on line 125 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces 

Remove the trailing spaces at the end of the line.

Check failure on line 126 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces 

Remove the trailing spaces at the end of the line.
### <a name="deriveddatatypes"></a> Derived data types
* At the moment, there are two types of derived data specifications:
* Standard derived data (marked with 🗂️)- if the wagon is used in a train, this will produce derived data to be used for further analysis. The results will not be merged across runs and can be used as input for future train runs. Note that standard derived data trains do not submit automatically and may need additional approval. If in doubt, please seek advise before enabling derived data tables in your wagon configuration.
* Slim derived data (marked with green bordered 🗂️) - similarly to the standard derived data case, if used in a train, this will produce derived data to be used for further analysis. This is reserved for derived data of small output size. The results will be merged across runs and are not available to use in future train runs. The data will be automatically deleted after a preset period of time. You can mark a wagon for running as slim derived data by checking `Ready for slim derived data`.

Check failure on line 131 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces 

Remove the trailing spaces at the end of the line.
* For wagons set as ready for slim derived data, two more fields need to be correctly set:
* Max DF size - This sets the maximal dataframe size in the merging step. Has to be 0 for not-self contained derived data (which need parent file access).
* Max derived file size - Sets the size limit for the output file size of the derived data file. This is an expert parameter which usually does not have to be changed. Only change this value if the processing in subsequent trains takes so long that the jobs fail. If set to 0 a good value will be automatically determined.

Check failure on line 135 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces 

Remove the trailing spaces at the end of the line.
* In order to update the derived data configuration with the latest version of the workflow, click on the button `↻ sync` in _Derived data_. By synchronizing the derived data, the tables which no longer belong to the workflow will be removed, and the values of the tables will be updated.

<div align="center">
<img src="../images/derivedDataEx.png" width="70%">
</div>

## <a name="wagon-test-statistics"></a> Test Statistics

Check failure on line 142 in docs/hyperloop/userdocumentation.md

View workflow job for this annotation

GitHub Actions / PR formatting / whitespace

Trailing spaces 

Remove the trailing spaces at the end of the line.

* <a name="wagonteststatistics"></a>_Test Statistics_ contains three graphs that display different metrics following the tests this wagon was part of. The first graph plots the _PSS Memory_ corresponding to each test run. The second one diplays the _CPU Time_, _Wall time_ and _Throughput_ along the test runs for this wagon. Finally, the third graph shows the _Output size_ at each test run.

Expand Down Expand Up @@ -492,7 +492,7 @@
<img src="../images/mergedOutput1.png" width="90%">
</div>

* <a name="longTrain"></a>The _Request long train_ tab allows users to request a long train after the train ran on a linked dataset. Linked datasets are subsets of a big dataset (_set up in the Dataset settings_). First, a train run needs to be **Done on a smaller linked dataset** before being run on a bigger dataset.
* <a name="longTrain"></a>The _Request long train_ tab allows users to request a long train after the train ran on a short dataset. Short datasets are subsets of a big dataset (_set up in the Dataset settings_). First, a train run needs to be **Done on a smaller short dataset** before being run on a bigger dataset.
* Any user who is part of the analysis can request a long train. Approval from the participating analyses PWGs conveners is required in order to submit a long train.

<div align="center">
Expand Down
Loading