Skip to content

Commit

Permalink
some rewording, fixed formatting of a note
Browse files Browse the repository at this point in the history
  • Loading branch information
robertoostenveld committed Jan 17, 2024
1 parent 0490568 commit 51f8650
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 15 deletions.
16 changes: 8 additions & 8 deletions development/releasing.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,11 @@ redirect_from:

Daily releases of FieldTrip are made available on the download server. The corresponding commits (i.e., point in time in the git repository) are tagged, such that the releases are also [available from GitHub](https://github.com/fieldtrip/fieldtrip/releases). New FieldTrip versions are released only if all tests pass. So, on some days there will not be a release.

## Quality control in FieldTrip
## Releasing the code

Every evening one of the FieldTrip [automation scripts](https://github.com/fieldtrip/automation) is executed as a cron job. It checks whether there are new commits on the [release branch](https://github.com/fieldtrip/fieldtrip/tree/release), and if so, it makes a new zip file of fieldtrip, fieldtrip-lite, and some of the modules, and copies those to the download server. It also tags the corresponding commit on the release branch on GitHub with ""YYYYMMDD", consistent with the naming of the zip files.

## Quality control prior to every release

The [fieldtrip/test](https://github.com/fieldtrip/fieldtrip/tree/master/test) directory contains many test scripts (technically they are functions). We use these initially to reproduce and identifying the cause of a bug, or for designing new end-user scripts that matches some new functionality that we want to implement. Once a reported bug has been fixed and/or the new functionality has been implemented, we keep the test scripts for [regression testing](https://en.wikipedia.org/wiki/Regression_testing).

Expand All @@ -19,14 +23,10 @@ If you suspect a problem with the FieldTrip code, the best way to resolve it is

The [master branch](https://github.com/fieldtrip/fieldtrip/tree/master) is tested _every evening_ by executing all test functions using the [dashboard scripts](https://github.com/fieldtrip/dashboard) that are running as a cron job on the [DCCN compute cluster](https://dccn-hpc-wiki.readthedocs.io). If the complete test batch passes, the changes on the master branch are automatically merged into the [release branch](https://github.com/fieldtrip/fieldtrip/tree/release). If there is a problems with one of the test scripts, an email is sent to the main developers and the updates are _not_ merged from master into release.

## Releasing the code

Every evening one of the FieldTrip [automation scripts](https://github.com/fieldtrip/automation) is executed as a cron job. It checks whether there are new commits on the [release branch](https://github.com/fieldtrip/fieldtrip/tree/release), and if so, it makes a new zip file of fieldtrip, fieldtrip-lite, and some of the modules, and copies those to the download server. It also tags the corresponding commit on the release branch on GitHub with ""YYYYMMDD", consistent with the naming of the zip files.

## Testing guidelines to external contributors
## Guidelines to contributors

{% include markup/end %}
When you [contribute](/development/contribute) to FieldTrip, it is recommended that you [test your changes](/development/testing). If the necessary test scripts pass, you can submit a [pull request](https://github.com/fieldtrip/fieldtrip/pulls).
{% include markup/info %}
When you [contribute](/development/contribute) to FieldTrip, it is recommended that you [test your changes](/development/testing). If the necessary test scripts pass, you can submit a [pull request](https://github.com/fieldtrip/fieldtrip/pulls).
{% include markup/end %}

When you [add a new FieldTrip function](/development/testing#adding-a-new-test-script), it is good practice to also include a short test script. This helps the maintainers to evaluate your contribution, and in the future to ensure that future changes to code elsewhere do not break your contribution. When you [modify existing code](/development/testing), you should check for existing [test scripts](/development/testing#running-existing-tests) and run them locally prior to sending a pull-request.
12 changes: 5 additions & 7 deletions development/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,9 @@ redirect_from:

# Testing

## Why is testing important?
To make sure everything works correctly, we use [regression testing](https://en.wikipedia.org/wiki/Regression_testing). This way, we can be confident that when we add, modify, or remove a function, we won't break the existing ones. Testing also helps us find and fix any issues early on, ensuring that FieldTrip functions smoothly for all users. We use nightly testing as part of the [release](/development/releasing) procedure.

FieldTrip is a toolbox with many functions, designed to be compatible with each other. This means that one function often relies on the output of another function. Functions are categorized as [high, low-level, or private](https://www.fieldtriptoolbox.org/development/architecture/#high-level-low-level-and-private-functions), with high-level functions depending on low-level and private functions.

To make sure everything works correctly, we use [regression testing](https://en.wikipedia.org/wiki/Regression_testing). This way, we can be confident that when we add, modify, or remove a function, we won't break the existing ones. Testing also helps us find and fix any issues early on, ensuring that FieldTrip functions smoothly for all users. We use nightly testing as part of the [release](/development/release) procedure.
FieldTrip is a toolbox with many functions, designed to be compatible with each other. This means that one function often relies on the output of another function. Functions are categorized as [high, low-level, or private](https://www.fieldtriptoolbox.org/development/architecture/#high-level-low-level-and-private-functions), with high-level functions depending on low-level and private functions. The regression testing mainly focusses on high- and low-level functions that users can call from their analysis scripts.

## How are the tests organized in FieldTrip?

Expand Down Expand Up @@ -56,7 +54,7 @@ This helps to select an appropriate subset of tests to run based on:
1. **WALLTIME**: The duration that a test needs to run. This duration is usually more than the actual duration needed since it also includes the time that MATLAB itself takes to start (which is about 30-60 seconds) and the time that it takes to load the test data.
2. **MEM**: MEM stands for memory, and it represents the amount of memory required for a test to run.
3. **DATA**: The external data that the test requires to run. More specifically, `DATA no` means that the test doesn't need any external data to run. `DATA public` means it needs data available from the [download server](https://download.fieldtriptoolbox.org/). `DATA private` means that it needs data that are not publicly accessible but only to people working in the DCCN.
4. **DEPENDENCY**: The dependencies, i.e. high- or low-level FieldTrip functions to which the test script is particularly sensitive. There are three types of dependencies within the FieldTrip codebase: self-dependencies (where a function relies on itself), direct dependencies (comprising both [high-level and some low-level FieldTrip functions](https://www.fieldtriptoolbox.org/development/architecture/#high-level-low-level-and-private-functions) called directly within a test script), and indirect dependencies (involving [some low-level functions and all the private FieldTrip functions](https://www.fieldtriptoolbox.org/development/architecture/#high-level-low-level-and-private-functions) that, while they are not directly called within a test script, still play a role in the execution process). For example, in FieldTrip you can run:
4. **DEPENDENCY**: The dependencies, i.e. high- or low-level FieldTrip functions to which the test script is particularly sensitive. There are three types of dependencies within the FieldTrip codebase: self-dependencies (where a function relies on itself), direct dependencies (comprising both [high-level and some low-level FieldTrip functions](https://www.fieldtriptoolbox.org/development/architecture/#high-level-low-level-and-private-functions) called directly within a test script), and indirect dependencies (involving [some low-level functions and all the private FieldTrip functions](https://www.fieldtriptoolbox.org/development/architecture/#high-level-low-level-and-private-functions) that, while they are not directly called within a test script, still play a role in the execution process). For example, in FieldTrip you can run:

ft_test find_dependency test_bug46

Expand Down Expand Up @@ -131,8 +129,8 @@ If you are an external contributor outside the DCCN network, you can select test

To quickly find potential errors, you may want to run the short and small tests first. You can sort the tests with increasing memory and walltime:

% Convert the memory from string to numbers to allow sorting.
filtered_test.mem = str2double(filtered_test.mem);
% Convert the memory from string to numbers to allow sorting.
filtered_test.mem = str2double(filtered_test.mem);

filtered_test = sortrows(filtered_test, 'mem', 'ascend');
filtered_test = sortrows(filtered_test, 'walltime', 'ascend');
Expand Down

0 comments on commit 51f8650

Please sign in to comment.