[enh] Make all process compatible with raw files

[Edouard2laire]

This PAr aims at improving the way files are handled. The objective is to keep the type of the files. So if the input is raw, the output is raw. If the input is epoched data, so is the output.

@rcassani, could you check what i am doing wrong? the output file is not registered to the database when creating a new raw file?

it would be nice if it was simpler from the API point of view to save/load either epoch data or raw files as from the process point of view, we don't really care if the file we are processing is raw or epoch. Currently, it leads to a lot of code duplication and case handling to make sure each process can work on either raw or epoched data...

[rcassani]

@rcassani, could you check what i am doing wrong? the output file is not registered to the database when creating a new raw file?

Done 👍

it would be nice if it was simpler from the API point of view to save/load either epoch data or raw files as from the process point of view, we don't really care if the file we are processing is raw or epoch. Currently, it leads to a lot of code duplication and case handling to make sure each process can work on either raw or epoched data...

If there are cases where only processing happens, the process could be written as Category Filter, the the data will be already loaded as the field A in sInput for data and raw. Only when run with data it will offer the option of over writing the input file

• Process Category Filter and Filter2
• Simple example in process_absolute

[Edouard2laire]

Thank you for the comment. The file are now registered in the database. However, i am not able to open it

***************************************************************************
** Error: Line 83:  in_bst_data (line 83)
** Cannot load recordings file: "/Users/edelaire1/Documents/bst_db/TutorielNIRSTORM2023/data/Subject01/@rawsub-01_task-tapping_run-01_dOD/data_0raw_OD_231102_1717.mat".
**  load
** Unable to read MAT-file /Users/edelaire1/Documents/bst_db/TutorielNIRSTORM2023/data/Subject01/@rawsub-01_task-tapping_run-01_dOD/data_0raw_OD_231102_1717.mat. File might be corrupt.
** 
** Call stack:
** >in_bst_data.m at 83
** >bst_memory.m>LoadDataFile at 689
** >bst_memory.m at 73
** >view_timeseries.m at 107
** >tree_callbacks.m at 261
** >bst_call.m at 28
** >panel_protocols.m>CreatePanel/protocolTreeClicked_Callback at 125
** >bst_call.m at 28
** >panel_protocols.m>@(h,ev)bst_call(@protocolTreeClicked_Callback,h,ev) at 75
** 
***************************************************************************

it seems that the file is created but matlab cannot open it

[Edouard2laire]

@rcassani When using the filter type of process, is there an easy way to have access to the events (so we could remove those lines https://github.com/Nirstorm/nirstorm/pull/219/files#diff-1ab9716e183e1a953da0f256a555bfac4adad214f253515ee0e3c0534b9ad0d9L106-L113) and not to have to load the entire file a second time?

[rcassani]

Unfortunately, it is necessary to read the events from the file.

It can be faster by reading only that field for imported data:

sDataIn = in_bst_data(sInputs.FileName, 'Events');
events = sDataIn.Events;

For raw files it should be fast to read F (like in the pointed lines), as it is only metadata of the raw file, not the recordings.

[Edouard2laire]

Unfortunately, it is necessary to read the events from the file.

It can be faster by reading only that field for imported data:

sDataIn = in_bst_data(sInputs.FileName, 'Events');
events = sDataIn.Events;

For raw files it should be fast to read F (like in the pointed lines), as it is only metadata of the raw file, not the recordings.

Thank you. it seems that we can make minimal change to bst_process to have the event. would that be ok for you ? (see brainstorm-tools/brainstorm3#655)

Thank you for the comment. The file are now registered in the database. However, i am not able to open it

***************************************************************************
** Error: Line 83:  in_bst_data (line 83)
** Cannot load recordings file: "/Users/edelaire1/Documents/bst_db/TutorielNIRSTORM2023/data/Subject01/@rawsub-01_task-tapping_run-01_dOD/data_0raw_OD_231102_1717.mat".
**  load
** Unable to read MAT-file /Users/edelaire1/Documents/bst_db/TutorielNIRSTORM2023/data/Subject01/@rawsub-01_task-tapping_run-01_dOD/data_0raw_OD_231102_1717.mat. File might be corrupt.
** 
** Call stack:
** >in_bst_data.m at 83
** >bst_memory.m>LoadDataFile at 689
** >bst_memory.m at 73
** >view_timeseries.m at 107
** >tree_callbacks.m at 261
** >bst_call.m at 28
** >panel_protocols.m>CreatePanel/protocolTreeClicked_Callback at 125
** >bst_call.m at 28
** >panel_protocols.m>@(h,ev)bst_call(@protocolTreeClicked_Callback,h,ev) at 75
** 
***************************************************************************

it seems that the file is created but matlab cannot open it

Do you have any idea about that issue? Edit: it seems to be working :)

[Edouard2laire]

List of processes to adapt for continuous recording:

Preprocessing

• process_nst_deglitch.m -> Convert to filter
• process_nst_detrend.m -> Convert to filter
• process_nst_detect_bad.m
• process_nst_detect_bad_CV.m
• process_nst_iir_filter.m -> Convert to filter
• process_nst_motion_correction.m -> Convert to filter
• process_nst_remove_ssc.m -> Convert to filter

MBLL

• process_nst_dOD.m
• process_nst_mbll.m
• process_nst_mbll_dOD.m

Global

• Improve the history for the filter process

[rcassani]

Hi @Edouard2laire, is my input already required for this PR?

[Edouard2laire]

Hello,

I think this is ready to be merged. Let me know if you want to review it or if I should merge it as is.

One issue I would like us to check is brainstorm-tools/brainstorm3#656 as it is often important in nirs to check what the preprocessing steps are doing to the data so we would open the raw data and after motion correction to be sure the motion correction step worked well and it makes it necessary to have two continuous files open at the same time.

[Edouard2laire]

Accidentally closed the PR. Reopening here #223