-Usually, you will do the first step (the session setup) once at the beginning of your script -to construct a fixed session header that can be used in subsequent requests in your script. +[**Step 2: Install Python if needed**](step-2-install-python-if-needed-anchor) . +
-(setting-up-a-session-from-matlab-anchor)= -### Setting up a session from MATLAB +[**Step 3: Connect Python to MATLAB**](step-3-connect-python-to-matlab-anchor). +If already connected, skip to Step 4. + -The goal of the session setup is to construct a header that can be used in subsequent web requests. -The first step is to call the [**getHostOptions.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/getHostOptions.m). +[**Step 4: Install HEDtools**](step-4-install-hedtools-anchor) +```` -(setting)= -`````{admonition} Establish a session. -:class: tip -```matlab +(step-1-find-python-anchor)= +#### Step 1: Find Python -host = 'https://hedtools.ucsd.edu/hed'; -[servicesUrl, options] = getHostOptions(host) +Follow these steps until you find a Python executable that is version 3.7 or greater. +If you can't locate one, you will need to install it. -``` -````` -The `host` should be set to the URL of the webserver that you are using. -The call to `getHostOptions`, only needs to be done once at the beginning of your session. -The `servicesURL` and the `options` can be used for all of your subsequent requests. +````{Admonition} Does MATLAB already have a good version of Python you can use? -This function constructs the services URL from the host URL. -The function also makes a service request to obtain a CSRF token and a cookie using -[**getSessionInfo**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/getSessionInfo.m). -The function then constructs a header and calls the MATLAB `weboptions` function -to get an options object suitable for use with the MATLAB `webwrite` function -use in all of our examples. +In your MATLAB command window execute the following function: -The `getHostOptions` does all the setup for using the services. -As indicated by the code below, all communication is done in JSON. -However, as demonstrated below, the MATLAB `webwrite` function -takes a MATLAB `struct` as its `request` parameter and internally -converts to the format specified in the header before making the request. +```matlab +pyenv +``` +The following example response shows that MATLAB is using Python version 3.9 +with executable located at `C:\Program Files\Python39\pythonw.exe`. -(gethostoptions-source-anchor)= -`````{admonition} Source for getHostOptions. -:class: tip ```matlab -function [servicesUrl, options] = getHostOptions(host) - csrfUrl = [host '/services']; - servicesUrl = [host '/services_submit']; - [cookie, csrftoken] = getSessionInfo(csrfUrl); - header = ["Content-Type" "application/json"; ... - "Accept" "application/json"; ... - "X-CSRFToken" csrftoken; "Cookie" cookie]; + PythonEnvironment with properties: - options = weboptions('MediaType', 'application/json', ... - 'Timeout', 120, 'HeaderFields', header); + Version: "3.9" + Executable: "C:\Program Files\Python39\pythonw.exe" + Library: "C:\Program Files\Python39\python39.dll" + Home: "C:\Program Files\Python39" + Status: NotLoaded + ExecutionMode: InProcess ``` -````` +```` -The `Timeout` parameter indicates how many seconds MATLAB will wait for a response -before returning as a failed operation. -The `timeout` value of 120 seconds is sufficient for most situations. -However, this can be adjusted upward or downward to suit the user. -The `HeaderFields` sets the parameters of HTTP request. +If MATLAB has already knows about a suitable Python version that is at least 3.7, +you are ready to go to [**Step 4: Install HEDTools**](step-4-install-hedtools-anchor). +Keep track of the location of the Python executable. -(creating-a-request-structure-anchor)= -### Creating a request structure +If the `pyenv` did not indicate a suitable Python version, you will need to +find the Python on your system (if there is one), or install your own. -The request structure is a MATLAB `struct` which must have a `service` field and can have an -arbitrary number of fields depending on which service is being requested. +There are several likely places to look for Python on your system. -The simplest service is `get_services`, -which returns a string containing information about the available services. -This service requires no additional parameters. -In this and other examples, we assume that `getHostOptions` has been called to retrieve -`servicesUrl` and `options` for use in the session. +**For Linux users**: +>Likely places for system-space installation are `/bin`, `/local/bin`, `/usr/bin`, `/usr/local/bin`, or `/opt/bin`. User-space installations are usually your home directory in a subdirectory such as `~/bin` +or `~/.local/bin`. -`````{admonition} Request a list of available HED web services. -:class: tip -```matlab +**For Windows users**: +> Likely places for system-space installation are `C:\`, `C:\Python`, or `C:\Program Files`. +User-space installations default to your personal account +in `C:\Users\yourname\AppData\Local\Programs\Python\python39` where `yourname` is your Windows account name +and `python39` will be the particular version (in this case Python 3.9). -request = struct('service', 'get_services'); -response = webwrite(servicesUrl, request, options); -response = jsondecode(response); +If you don't have any success finding a Python executable, +you will need to install Python as described in +[**Step 2: Install Python if needed**](step-2-install-python-if-needed-anchor). + +Otherwise, you can skip to [**Step 3:Connect Python to MATLAB**](step-3-connect-python-to-matlab-anchor). +```{warning} +**You need to keep track of the path to your Python executable for Step 3.** ``` -````` -The MATLAB `webwrite` returns a JSON structure as specified in the `options`. -The MATLAB `jsondecode` function returns a MATLAB `struct` whose format is explained -below in [**Decoding a service response**](decoding-a-service-response-anchor). +(step-2-install-python-if-needed-anchor)= +#### Step 2: Install Python if needed +If you don't have Python on your system, you will need to install it. +Go to [**Python downloads**](https://www.python.org/downloads/) and pick the correct installer +for your operating system and version. -Except for `get_services`, all other services are of the form *target_command* -where *target* is the primary type of data acted on (events, schema, sidecar, spreadsheet, or string). -The possible values for *command* depend on the value of *target*. -For example `sidecar_validate` requests that a JSON sidecar be validated. +Depending on your OS and the installer options you selected, +Python may be installed in your user space or in system space for all users. +- You should keep track of the directory that Python was installed in. +- You may want to add the location of the Python executable to your PATH. + (Most installers give you that option as part of the installation.) -The `get_services` command provides information about the HED services that -are available and the parameters required. -The `get_services` entry for `sidecar_validate` is the following: +#### Installing in a virtual environment -`````{admonition} The get_services entry for sidecar_validate. -:class: tip -```text -sidecar_validate: - Description: Validate a BIDS JSON sidecar (in string form) and return errors. - Parameters: - json_string - schema_string or schema_url or schema_version - check_for_warnings - Returns: A list of errors if any. -``` -````` +https://www.mathworks.com/support/search.html/answers/1750425-python-virtual-environments-with-python-interface.html?fq%5B%5D=asset_type_name:answer&page=1 +(step-3-connect-python-to-matlab-anchor)= +#### Step 3: Connect Python to Matlab -The *Parameters* section indicates the fields in addition to the `service` -that are needed in the request structure. -For example, `sidecar_validate` requires a HED schema. -One possibility is to read a schema into a string and provide this information in `schema_string`. -Another possibility is to provide a URL for the schema. -The most-commonly used option is to use `schema_version` to indicate one of the supported -versions available in the -[**hedxml**](https://github.com/hed-standard/hed-specification/tree/master/hedxml) directory of the -[**hed-schemas**](https://github.com/hed-standard/hed-schemas) repository on GitHub. -(create-request-sidecar-validate-anchor)= -`````{admonition} Create a request for the sidecar_validate web service. -:class: tip +C:\Users\username\AppData\Local\Programs\Python\python -m venv C:\Users\username\py38 + +Setting the Python version uses the MATLAB `pyenv` function with the `'Version'` argument +as illustrated by the following example. + +````{admonition} Example MATLAB function call connect MATLAB to Python. + ```matlab -jsonText = fileread('../../datasets/eeg_ds003645s_hed_demo/task-FacePerception_events.json'); -request = struct('service', 'sidecar_validate', ... - 'schema_version', '8.2.0', ... - 'json_string', jsonText, ... - 'check_for_warnings', 'on'); +>> pyenv('Version', 'C:\Program Files\Python39\python.exe') ``` -````` -This example reads the JSON sidecar to be validated as a character array into the variable `jsonText` -and makes a request for validation using `HED8.2.0.xml`. +```` -The request indicates that validation warnings as well as errors should be included in the response. -If you wish to exclude warnings, use `off` instead of `on` as the `check_for_warnings` field value. +Be sure to substitute the path of the Python that you have found. +Notice that the executable listed in Step 1 was `pythonw.exe`, but we have used `python.exe` here +to indicate the command line version. -The [**demoSidecarServices.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/demoSidecarServices.m) -function shows complete examples of the various HED services for JSON sidecars. +Use the MATLAB `pyenv` function again without arguments to check that your installation is as expected. -(making-a-service-request-anchor)= -### Making a service request - -The HED services all use the MATLAB `webwrite` to make HED web service requests. -The following call uses the -[**sidecar_validate request**](create-request-sidecar-validate-anchor) -from the previous section. +````{Admonition} Example response for pyenv all with no arguments after setting environment. -`````{admonition} Request the sidecar validation service. -:class: tip ```matlab -response = webwrite(servicesUrl, request, options); -response = jsondecode(response); -outputReport(response, 'Example: validate a JSON sidecar'); -``` -````` + PythonEnvironment with properties: -The [**outputReport.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/outputReport.m)
-MATLAB script outputs the response in readable form with a user-provided table.
+ Version: "3.9"
+ Executable: "C:\Program Files\Python39\python.exe"
+ Library: "C:\Program Files\Python39\python39.dll"
+ Home: "C:\Program Files\Python39"
+ Status: NotLoaded
+ ExecutionMode: InProcess
+```
+````
-If the web server is down or times out during a request,
-the MATLAB `web_write` function throws an exception,
-and the script terminates without setting the response.
+(step-4-install-hedtools-anchor)=
+#### Step 4: Install HEDTools
-If the connection completes successfully, the response will set.
-The next section explains the response structure in more detail.
+The general-purpose package manager for Python is called `pip`.
+By default, `pip` retrieves packages to be installed from the [**PyPI**](https://pypi.org)
+package repository. You will need to use the version of `pip` that corresponds
+to the version of Python that is connected to MATLAB.
+This may not be the default `pip` used from the command line.
-(decoding-a-service-response-anchor)=
-### Decoding a service response
+````{admonition} Command to install hedtools in MATLAB.
+To install the latest released version of `hedtools` type a pip command such as the
+following in your MATLAB command window.
-All HED web services return a response consisting of a JSON dictionary with
-4 keys as summarized in this table.
+```matlab
+system('"C:\Program Files\Python39\Scripts\pip" install hedtools')
+```
+Use the full path of the pip associated
+with the Python that you are using with MATLAB
+````
-| Field name | Meaning |
-| ---------- | ------- |
-| service | Name of the requested service. |
-| results | Results of the operation. |
-| error_type | Type of error if the service failed. |
-| error_msg | Explanation of the message if the service failed. |
+Giving the full path to `pip` corresponding to the Python installation that MATLAB
+is using ensures that MATLAB knows about `HEDtools`.
+(The version of MATLAB that Python is using may not be the same as the Python in the system PATH.)
-The `jsondecode` function translates the JSON dictionary into a MATLAB structure.
+Also watch the resulting messages in the command window to make sure that HEDtools
+was successfully installed.
+In the case of the above example, the Python being used is in system space,
+which requires administrator privileges.
-The `error_type` indicates whether the service request completed successfully
-and was able to get results.
-The `error_type` **does NOT** indicate the nature of the results
-(for example whether a JSON sidecar was valid or not),
-but rather whether the server was able to complete the request without raising an exception.
-A failure `error_type` is highly unusual and indicates some type of
-unexpected internal web service error.
-Errors of this type should be reported using the
-[**GitHub hed-python issues**](https://github.com/hed-standard/hed-python/issues) mechanism.
+The first line of the output was:
-The `results` structure has the actual results of the service request.
+```matlab
+ Defaulting to user installation because normal site-packages is not writeable
+```
-| Field name | Meaning |
-| ---------- | ------- |
-| command | Command executed in response to the service request. |
-| command_target | Type of data on which the command was executed. |
-| data | Data returned by the service (either processed result or a list of errors). |
-| msg_category | Success or warning depending on the result of processing the service. |
-| msg | Explanation of the output of the service. |
-| output_display_name | (Optional) File name for saving return data. |
-| schema_version | (Optional) Version of the HED schema used in the processing. |
+On Windows these packages will be found in a `site-packages` directory such as:
-The `results` structure will always have `command`, `command_target`
-fields indicating what operation was performed on what type of data.
+```text
+`C:\Users\username\AppData\Roaming\Python\Python39\site-packages`
+```
-The `msg_category` will be `success` or `warning` depending on whether there were errors.
-The contents of the `data` field will contain different information depending on the `msg_category`.
-For example, if a sidecar had validation errors,
-`results.msg_category` will be `warning` and the `results.data` value
-should be interpreted as a list of errors.
-If the sidecar had no errors, `results.data` will be an empty string.
+On Linux these packages might be found in directory such as:
-(eeglab-integration-anchor)=
-## EEGLAB plug-in integration
-EEGLAB is the most widely used EEG software environment for analysis of
-human electrophysiological (and related) data.
-EEGLAB combines graphical and command-line user interfaces,
-making it friendly for both beginners who may who prefer a visual,
-and automated way of analyzing data as well as experts,
-who can easily customize, extend, and automate the EEGLAB tool environment by writing new
-EEGLAB-compatible scripts and functions.
+```text
+/home/username/.local/lib/python3.9/site-packages/
+```
-HED is fully integrated into EEGLAB via the *HEDTools* plug-in, allowing users to annotate their
-EEGLAB STUDY and datasets with HED,
-as well as enabling HED-based data manipulation and processing.
-### Installing *HEDTools*
+```{warning}
+If your system had a Python 2 installed at some point, your Python 3
+executable might be named `python3` rather than `python`.
-*HEDTools* EEGLAB plug-in can be installed using one of the following ways:
+Similarly, the `pip` package manager might be named `pip3` instead of `pip`.
-#### Method 1: EEGLAB Extension Manager:
+```
-Launch EEGLAB. From the main GUI select:
+The following MATLAB statement can be used to test that everything was installed correctly.
-> **File --> Manage EEGLAB extension**
+````{Admonition} Test that everything is installed.
-The extension manager GUI will pop up.
+```matlab
+pyrun("from hed import _version as vr; print(f'Using HEDTOOLS version: {str(vr.get_versions())}')")
+```
+If everything installed correctly, the output will be something like
-From this GUI look for and select the plug-in *HEDTools* from the main window,
-then click into the *Install/Update* button to install the plug-in.
+```matlab
+Using HEDTOOLS version: {'date': '2022-06-20T14:40:24-0500', 'dirty': False, 'error': None, 'full-revisionid': 'c4ecd1834cd31a05ebad3e97dc57e537550da044', 'version': '0.1.0'}
+```
+````
-#### Method 2: Download and unzip
+(matlab-wrappers-for-HED-tools-anchor)=
+### MATLAB wrappers for HEDTools
-Download the zip file with the content of the plug-in *HEDTools* either from
-the EEGLAB [**plug-ins summary page**](https://sccn.ucsd.edu/eeglab/plugin_uploader/plugin_list_all.php).
+The [**hedtools_wrappers**](https://github.com/hed-standard/hed-examples/tree/main/hedcode/matlab_scripts/hedtools_wrappers) directory in the
+[**hed-examples**](https://github.com/hed-standard/hed-examples) GitHub repository
+contains MATLAB wrapper functions for calling various commonly used HED tools.
-Unzip file into the folder *../eeglab/plugins* and restart the *eeglab* function in a MATLAB session.
+#### Direct calls to HEDTools
-(annotating-datasets-anchor)=
-### Annotating datasets
+Wrapper functions are provided to some of the more commonly used
+functions in the HEDTools suite.
-We will start by adding HED annotations to the EEGLAB tutorial dataset.
+The following example shows the MATLAB wrapper function
+[**validateHedInBids.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/hedtools_wrappers/validateHedInBids.m),
+which contains the underlying calls to HEDTools Python BIDs validation.
-After installing the *HEDTools* open the EEGLAB main window and load the dataset by selecting the menu item:
-> **File --> Load existing dataset** .
+````{admonition} A MATLAB wrapper function for validating HED in a BIDS dataset.
+:class: tip
-Selecting the tutorial dataset under your eeglab installation _eeglab/sample_data/eeglab_data.set_.
+```matlab
+function issueString = validateHedInBids(dataPath)
+ py.importlib.import_module('hed');
+ bids = py.hed.tools.BidsDataset(dataPath);
+ issues = bids.validate();
+ issueString = string(py.hed.get_printable_issue_string(issues));
-Read a description of the dataset and of its included event codes by selecting:
+```
-> **Edit --> About this dataset**:
+Example MATLAB calling code for this function:
-
+```matlab
+dataPath = 'H:\datasets\eeg_ds003645s_hed';
+issueString = validateHedInBids(dataPath);
+if isempty(issueString)
+ fprintf('Dataset %s has no HED validation errors\n', dataPath);
+else
+ fprintf('Validation errors for dataset %s:\n%s\n', dataPath, issueString);
+end
-The description gives a general idea of the codes found in the event structure.
-Yet, inquisitive researchers interested in the nature of the stimuli
-(e.g., color and exact location of the squares on the screen)
-would have to look up the referenced paper for details.
+```
+````
+In above example assumes that the BIDS dataset was located at `H:\datasets\eeg_ds003645s_hed`.
+We tested it with the [**eeg_ds003645s_hed**](https://github.com/hed-standard/hed-examples/tree/main/datasets/eeg_ds003645s_hed) available on GitHub.
+You can download and use this test data or set `dataPath` to the root directory of your own dataset.
-Our goal in using HED tags is to describe the experimental events that are recorded
-in the *EEG.event* data structure in sufficient detail that anyone using the dataset
-in the future will not need to find and read a separate, detailed description of the dataset
-or study to understand the recorded experimental events.
-As demonstrated below, such annotation will allow us to extract epochs using meaningful
-HED tags instead of the alphanumeric codes often associated with shared EEG data.
+#### Calls to HED remodeling tools
-#### Launching EEGLAB HEDTools
-To add and view HED tags for the dataset, from EEGLAB menu, select:
+Many of the most useful HEDTools functions are packaged in the
+HED remodeling tool suite.
+These tools allow operations such as creating summaries, validating the dataset, and transforming event files to be run on an entire dataset.
-> **Edit --> Add/Edit event HED tags**.
+The following example illustrates a call that creates a summary of the
+experimental conditions for a HED-tagged dataset.
-_HEDTools_ will extract information from the *EEG.event* structure,
-automatically detecting the event structure fields and their unique values.
+````{admonition} A MATLAB wrapper function for a remodeling operation to create a summary.
+:class: tip
-The _HEDTools_ ignore the fields the event structure fields *.latency*, *.epoch*, and *.urevent*.
+```matlab
+function runRemodel(remodel_args)
+ py.importlib.import_module('hed');
+ py.hed.tools.remodeling.cli.run_remodel.main(remodel_args);
+```
-A window will appear asking you to verify/select categorical fields:
+Example MATLAB calling code for this function:
-
+```matlab
-Here both *position* and *type* are categorical fields.
-*HEDTools* automatically selects fields with less than 20 unique values to be categorical,
-but the user can modify which values are chosen.
+dataPath = 'G:\ds003645';
+remodelFile = 'G:\summarize_hed_types_rmdl.json';
+remodel_args = {dataPath, remodelFile, '-b', '-x', 'stimuli', 'derivatives'};
+runRemodel(remodel_args);
-CTagger (for 'Community Tagger') is a graphical user interface (GUI) built to facilitate
-the process of adding HED tags to recorded events in existing datasets.
-Clicking *Continue* brings up the *CTagger* interface:
+```
+````
-
+The command line arguments to the various remodeling functions are
+given in a cell array, rather than a regular MATLAB array.
+For the remodeling operations, first and second operation must be the dataset root
+directory and the remodeling file name, respectively.
+In this example, dataset `ds003645` has been downloaded from [**openNeuro**](https://openneuro.org) to the `G:\` drive.
+The remodeling file used in this example can be found at
+See [**File remodeling quickstart**](FileRemodelingQuickstart.md)
+and [**File remodeling tools**](FileRemodelingTools.md) for
+additional information.
+The wrapper functions are available on GitHub in
+the [**hedtools_wrappers**](https://github.com/hed-standard/hed-examples/tree/develop/src/matlab_scripts/hedtools_wrappers) directory.
-The CTagger GUI is organized using a split window strategy.
-The left window shows the items to be tagged,
-and the right window shows the current HED tags associated with the selected item.
-The *Show HED schema* button brings up a browser for the HED vocabulary.
-Through the CTagger GUI, users can explore the HED schema,
-quickly look up and add tags (or tag groups) to the desired event codes,
-and use import/export features to reuse tags on from other data recordings in the same study.
+### MATLAB functions for Python
-The process of tagging is simply choosing tags from the available vocabulary
-(using the HED schema browser) and associating these tags with each event code.
+The following table lists the relevant MATLAB functions that are available.
+You should refer to the help facility for your version of MATLAB to get the details of what is
+supported for your version of MATLAB.
-Once familiar with HED and the vocabulary, most users just type the tags directly in the tag window
-shown on the right.
-
-CTagger is used as part of the HEDTools plug-in in this tutorial,
-but it can also be used as a standalone application.
-
-Instructions on downloading and using the standalone version of CTagger,
-as well as step-by-step guide on how to add HED annotation with CTagger,
-can be found at in [**Tagging with CTagger**](./CTaggerGuiTaggingTool.md).
+| MATLAB command | Purpose |
+| -------------- | --------|
+| `pyenv` | Setup your Python environment in MATLAB.
+ options = weboptions('MediaType', 'application/json', ...
+ 'Timeout', 120, 'HeaderFields', header);
+```
+`````
+The `Timeout` parameter indicates how many seconds MATLAB will wait for a response
+before returning as a failed operation.
+The `timeout` value of 120 seconds is sufficient for most situations.
+However, this can be adjusted upward or downward to suit the user.
+The `HeaderFields` sets the parameters of HTTP request.
-This will bring up a window to specify the options for extracting data epochs:
+(creating-a-request-structure-anchor)=
+### Creating a request structure
-
+The request structure is a MATLAB `struct` which must have a `service` field and can have an
+arbitrary number of fields depending on which service is being requested.
-The *pop_epochhed* menu is almost identical to the EEGLAB *pop_epoch* menu with the
-exceptions of the first input field (**Time-locking HED tag(s)**) and the second
-input field (**Exclusive HED tag(s)**).
+The simplest service is `get_services`,
+which returns a string containing information about the available services.
+This service requires no additional parameters.
+In this and other examples, we assume that `getHostOptions` has been called to retrieve
+`servicesUrl` and `options` for use in the session.
-Instead of passing in or selecting from a group of unique event types,
-the user passes in a comma separated list of HED tags.
-For each event all HED tags in this list must be found for a data epoch to be generated.
-Clicking the adjacent button (with the label …) will open a search tool to help you select
-HED tags retrieved from the dataset.
+`````{admonition} Request a list of available HED web services.
+:class: tip
+```matlab
-
+request = struct('service', 'get_services');
+response = webwrite(servicesUrl, request, options);
+response = jsondecode(response);
-When you type something in the search bar, the dialog displays a list below containing possible matches.
-Pressing the "up" and "down" arrows on the keyboard while the cursor is in the search bar moves to the
-next or previous tag in the list.
+```
+`````
-Pressing "Enter" selects the current tag in the list and adds the tag to the search bar.
-You can continue search and add tags after adding a comma after each tag.
-When done, click the **Ok** button to return to the main epoching menu.
+The MATLAB `webwrite` returns a JSON structure as specified in the `options`.
+The MATLAB `jsondecode` function returns a MATLAB `struct` whose format is explained
+below in [**Decoding a service response**](decoding-a-service-response-anchor).
-(python-hedtools-in-matlab-anchor)=
-## Python HEDTools in MATLAB
-+The HED services all use the MATLAB `webwrite` to make HED web service requests. +The following call uses the +[**sidecar_validate request**](create-request-sidecar-validate-anchor) +from the previous section. -[**Step 2: Install Python if needed**](step-2-install-python-if-needed-anchor) . -
+`````{admonition} Request the sidecar validation service. +:class: tip +```matlab +response = webwrite(servicesUrl, request, options); +response = jsondecode(response); +outputReport(response, 'Example: validate a JSON sidecar'); +``` +````` -[**Step 3: Connect Python to MATLAB**](step-3-connect-python-to-matlab-anchor). -If already connected, skip to Step 4. - +The [**outputReport.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/outputReport.m)
+MATLAB script outputs the response in readable form with a user-provided table.
-[**Step 4: Install HEDtools**](step-4-install-hedtools-anchor)
-````
+If the web server is down or times out during a request,
+the MATLAB `web_write` function throws an exception,
+and the script terminates without setting the response.
-(step-1-find-python-anchor)=
-#### Step 1: Find Python
+If the connection completes successfully, the response will set.
+The next section explains the response structure in more detail.
-Follow these steps until you find a Python executable that is version 3.7 or greater.
-If you can't locate one, you will need to install it.
+(decoding-a-service-response-anchor)=
+### Decoding a service response
-````{Admonition} Does MATLAB already have a good version of Python you can use?
+All HED web services return a response consisting of a JSON dictionary with
+4 keys as summarized in this table.
-In your MATLAB command window execute the following function:
+| Field name | Meaning |
+| ---------- | ------- |
+| service | Name of the requested service. |
+| results | Results of the operation. |
+| error_type | Type of error if the service failed. |
+| error_msg | Explanation of the message if the service failed. |
-```matlab
-pyenv
-```
-The following example response shows that MATLAB is using Python version 3.9
-with executable located at `C:\Program Files\Python39\pythonw.exe`.
+The `jsondecode` function translates the JSON dictionary into a MATLAB structure.
-```matlab
- PythonEnvironment with properties:
+The `error_type` indicates whether the service request completed successfully
+and was able to get results.
+The `error_type` **does NOT** indicate the nature of the results
+(for example whether a JSON sidecar was valid or not),
+but rather whether the server was able to complete the request without raising an exception.
+A failure `error_type` is highly unusual and indicates some type of
+unexpected internal web service error.
+Errors of this type should be reported using the
+[**GitHub hed-python issues**](https://github.com/hed-standard/hed-python/issues) mechanism.
- Version: "3.9"
- Executable: "C:\Program Files\Python39\pythonw.exe"
- Library: "C:\Program Files\Python39\python39.dll"
- Home: "C:\Program Files\Python39"
- Status: NotLoaded
- ExecutionMode: InProcess
-```
-````
+The `results` structure has the actual results of the service request.
-If MATLAB has already knows about a suitable Python version that is at least 3.7,
-you are ready to go to [**Step 4: Install HEDTools**](step-4-install-hedtools-anchor).
-Keep track of the location of the Python executable.
+| Field name | Meaning |
+| ---------- | ------- |
+| command | Command executed in response to the service request. |
+| command_target | Type of data on which the command was executed. |
+| data | Data returned by the service (either processed result or a list of errors). |
+| msg_category | Success or warning depending on the result of processing the service. |
+| msg | Explanation of the output of the service. |
+| output_display_name | (Optional) File name for saving return data. |
+| schema_version | (Optional) Version of the HED schema used in the processing. |
-If the `pyenv` did not indicate a suitable Python version, you will need to
-find the Python on your system (if there is one), or install your own.
+The `results` structure will always have `command`, `command_target`
+fields indicating what operation was performed on what type of data.
-There are several likely places to look for Python on your system.
+The `msg_category` will be `success` or `warning` depending on whether there were errors.
+The contents of the `data` field will contain different information depending on the `msg_category`.
+For example, if a sidecar had validation errors,
+`results.msg_category` will be `warning` and the `results.data` value
+should be interpreted as a list of errors.
+If the sidecar had no errors, `results.data` will be an empty string.
-**For Linux users**:
+(eeglab-integration-anchor)=
+## EEGLAB plug-in integration
+EEGLAB is the most widely used EEG software environment for analysis of
+human electrophysiological (and related) data.
+EEGLAB combines graphical and command-line user interfaces,
+making it friendly for both beginners who may who prefer a visual,
+and automated way of analyzing data as well as experts,
+who can easily customize, extend, and automate the EEGLAB tool environment by writing new
+EEGLAB-compatible scripts and functions.
->Likely places for system-space installation are `/bin`, `/local/bin`, `/usr/bin`, `/usr/local/bin`, or `/opt/bin`. User-space installations are usually your home directory in a subdirectory such as `~/bin`
-or `~/.local/bin`.
+HED is fully integrated into EEGLAB via the *HEDTools* plug-in, allowing users to annotate their
+EEGLAB STUDY and datasets with HED,
+as well as enabling HED-based data manipulation and processing.
-**For Windows users**:
-> Likely places for system-space installation are `C:\`, `C:\Python`, or `C:\Program Files`.
-User-space installations default to your personal account
-in `C:\Users\yourname\AppData\Local\Programs\Python\python39` where `yourname` is your Windows account name
-and `python39` will be the particular version (in this case Python 3.9).
+### Installing *HEDTools*
-If you don't have any success finding a Python executable,
-you will need to install Python as described in
-[**Step 2: Install Python if needed**](step-2-install-python-if-needed-anchor).
+*HEDTools* EEGLAB plug-in can be installed using one of the following ways:
-Otherwise, you can skip to [**Step 3:Connect Python to MATLAB**](step-3-connect-python-to-matlab-anchor).
+#### Method 1: EEGLAB Extension Manager:
-```{warning}
-**You need to keep track of the path to your Python executable for Step 3.**
-```
+Launch EEGLAB. From the main GUI select:
-(step-2-install-python-if-needed-anchor)=
-#### Step 2: Install Python if needed
+> **File --> Manage EEGLAB extension**
-If you don't have Python on your system, you will need to install it.
-Go to [**Python downloads**](https://www.python.org/downloads/) and pick the correct installer
-for your operating system and version.
+The extension manager GUI will pop up.
-Depending on your OS and the installer options you selected,
-Python may be installed in your user space or in system space for all users.
-- You should keep track of the directory that Python was installed in.
-- You may want to add the location of the Python executable to your PATH.
- (Most installers give you that option as part of the installation.)
+From this GUI look for and select the plug-in *HEDTools* from the main window,
+then click into the *Install/Update* button to install the plug-in.
-#### Installing in a virtual environment
-https://www.mathworks.com/support/search.html/answers/1750425-python-virtual-environments-with-python-interface.html?fq%5B%5D=asset_type_name:answer&page=1
-(step-3-connect-python-to-matlab-anchor)=
-#### Step 3: Connect Python to Matlab
+#### Method 2: Download and unzip
+Download the zip file with the content of the plug-in *HEDTools* either from
+the EEGLAB [**plug-ins summary page**](https://sccn.ucsd.edu/eeglab/plugin_uploader/plugin_list_all.php).
-C:\Users\username\AppData\Local\Programs\Python\python -m venv C:\Users\username\py38
+Unzip file into the folder *../eeglab/plugins* and restart the *eeglab* function in a MATLAB session.
-Setting the Python version uses the MATLAB `pyenv` function with the `'Version'` argument
-as illustrated by the following example.
+(annotating-datasets-anchor)=
+### Annotating datasets
-````{admonition} Example MATLAB function call connect MATLAB to Python.
+We will start by adding HED annotations to the EEGLAB tutorial dataset.
-```matlab
->> pyenv('Version', 'C:\Program Files\Python39\python.exe')
-```
-````
+After installing the *HEDTools* open the EEGLAB main window and load the dataset by selecting the menu item:
-Be sure to substitute the path of the Python that you have found.
-Notice that the executable listed in Step 1 was `pythonw.exe`, but we have used `python.exe` here
-to indicate the command line version.
+> **File --> Load existing dataset** .
-Use the MATLAB `pyenv` function again without arguments to check that your installation is as expected.
+Selecting the tutorial dataset under your eeglab installation _eeglab/sample_data/eeglab_data.set_.
-````{Admonition} Example response for pyenv all with no arguments after setting environment.
+Read a description of the dataset and of its included event codes by selecting:
-```matlab
- PythonEnvironment with properties:
+> **Edit --> About this dataset**:
- Version: "3.9"
- Executable: "C:\Program Files\Python39\python.exe"
- Library: "C:\Program Files\Python39\python39.dll"
- Home: "C:\Program Files\Python39"
- Status: NotLoaded
- ExecutionMode: InProcess
-```
-````
+
-(step-4-install-hedtools-anchor)=
-#### Step 4: Install HEDTools
+The description gives a general idea of the codes found in the event structure.
+Yet, inquisitive researchers interested in the nature of the stimuli
+(e.g., color and exact location of the squares on the screen)
+would have to look up the referenced paper for details.
-The general-purpose package manager for Python is called `pip`.
-By default, `pip` retrieves packages to be installed from the [**PyPI**](https://pypi.org)
-package repository. You will need to use the version of `pip` that corresponds
-to the version of Python that is connected to MATLAB.
-This may not be the default `pip` used from the command line.
+Our goal in using HED tags is to describe the experimental events that are recorded
+in the *EEG.event* data structure in sufficient detail that anyone using the dataset
+in the future will not need to find and read a separate, detailed description of the dataset
+or study to understand the recorded experimental events.
+As demonstrated below, such annotation will allow us to extract epochs using meaningful
+HED tags instead of the alphanumeric codes often associated with shared EEG data.
-````{admonition} Command to install hedtools in MATLAB.
-To install the latest released version of `hedtools` type a pip command such as the
-following in your MATLAB command window.
+#### Launching EEGLAB HEDTools
+To add and view HED tags for the dataset, from EEGLAB menu, select:
-```matlab
-system('"C:\Program Files\Python39\Scripts\pip" install hedtools')
-```
-Use the full path of the pip associated
-with the Python that you are using with MATLAB
-````
+> **Edit --> Add/Edit event HED tags**.
-Giving the full path to `pip` corresponding to the Python installation that MATLAB
-is using ensures that MATLAB knows about `HEDtools`.
-(The version of MATLAB that Python is using may not be the same as the Python in the system PATH.)
+_HEDTools_ will extract information from the *EEG.event* structure,
+automatically detecting the event structure fields and their unique values.
-Also watch the resulting messages in the command window to make sure that HEDtools
-was successfully installed.
-In the case of the above example, the Python being used is in system space,
-which requires administrator privileges.
+The _HEDTools_ ignore the fields the event structure fields *.latency*, *.epoch*, and *.urevent*.
-The first line of the output was:
+A window will appear asking you to verify/select categorical fields:
-```matlab
- Defaulting to user installation because normal site-packages is not writeable
-```
+
-On Windows these packages will be found in a `site-packages` directory such as:
+Here both *position* and *type* are categorical fields.
+*HEDTools* automatically selects fields with less than 20 unique values to be categorical,
+but the user can modify which values are chosen.
-```text
-`C:\Users\username\AppData\Roaming\Python\Python39\site-packages`
-```
+CTagger (for 'Community Tagger') is a graphical user interface (GUI) built to facilitate
+the process of adding HED tags to recorded events in existing datasets.
+Clicking *Continue* brings up the *CTagger* interface:
-On Linux these packages might be found in directory such as:
+
-```text
-/home/username/.local/lib/python3.9/site-packages/
-```
+The CTagger GUI is organized using a split window strategy.
+The left window shows the items to be tagged,
+and the right window shows the current HED tags associated with the selected item.
+The *Show HED schema* button brings up a browser for the HED vocabulary.
+Through the CTagger GUI, users can explore the HED schema,
+quickly look up and add tags (or tag groups) to the desired event codes,
+and use import/export features to reuse tags on from other data recordings in the same study.
-```{warning}
-If your system had a Python 2 installed at some point, your Python 3
-executable might be named `python3` rather than `python`.
+The process of tagging is simply choosing tags from the available vocabulary
+(using the HED schema browser) and associating these tags with each event code.
-Similarly, the `pip` package manager might be named `pip3` instead of `pip`.
+Once familiar with HED and the vocabulary, most users just type the tags directly in the tag window
+shown on the right.
-```
+CTagger is used as part of the HEDTools plug-in in this tutorial,
+but it can also be used as a standalone application.
-The following MATLAB statement can be used to test that everything was installed correctly.
+Instructions on downloading and using the standalone version of CTagger,
+as well as step-by-step guide on how to add HED annotation with CTagger,
+can be found at in [**Tagging with CTagger**](./CTaggerGuiTaggingTool.md).
-````{Admonition} Test that everything is installed.
+#### Tagging the events
+A brief step-by-step guide to selecting tags can be found at
+[**HED annotation quickstart**](./HedAnnotationQuickstart.md).
+The following shows example annotations using the process suggested in the quickstart. we will import the annotation saved in the _events.json file format.
+Download the file [**eeglab-tutorial_events.json**](./_static/data/eeglab-tutorial_events.json) then select:
-```matlab
-pyrun("from hed import _version as vr; print(f'Using HEDTOOLS version: {str(vr.get_versions())}')")
-```
-If everything installed correctly, the output will be something like
+> **File --> Import --> Import BIDS events.json file**
-```matlab
-Using HEDTOOLS version: {'date': '2022-06-20T14:40:24-0500', 'dirty': False, 'error': None, 'full-revisionid': 'c4ecd1834cd31a05ebad3e97dc57e537550da044', 'version': '0.1.0'}
-```
-````
+to import it to CTagger. You can now review all the tags via:
+> **File --> Review all tags**
-(matlab-wrappers-for-HED-tools-anchor)=
-### MATLAB wrappers for HEDTools
+
-The [**hedtools_wrappers**](https://github.com/hed-standard/hed-examples/tree/main/hedcode/matlab_scripts/hedtools_wrappers) directory in the
-[**hed-examples**](https://github.com/hed-standard/hed-examples) GitHub repository
-contains MATLAB wrapper functions for calling various commonly used HED tools.
+#### Validation
+The last step of the annotation process is to validate the HED annotations.
+Click on the *Validate all* button at the bottom pane.
+A window will pop up showing validation results.
+If there are issues with the annotation, there will be a line for each of the issues found.
-#### Direct calls to HEDTools
+Here is an example of validation log file with issues:
-Wrapper functions are provided to some of the more commonly used
-functions in the HEDTools suite.
+
-The following example shows the MATLAB wrapper function
-[**validateHedInBids.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/hedtools_wrappers/validateHedInBids.m),
-which contains the underlying calls to HEDTools Python BIDs validation.
+If the annotation was correct, a message will appear confirming the validity:
+
-````{admonition} A MATLAB wrapper function for validating HED in a BIDS dataset.
-:class: tip
+Click *Finish* on the main CTagger window to end the annotation.
-```matlab
-function issueString = validateHedInBids(dataPath)
- py.importlib.import_module('hed');
- bids = py.hed.tools.BidsDataset(dataPath);
- issues = bids.validate();
- issueString = string(py.hed.get_printable_issue_string(issues));
+The tag review window will show up again for a final review and the option to save
+the annotation into an _events.json file for distribution just as with the *eeglab-tutorial_events.json*.
+Hit *Ok* to continue after that.
-```
+A last window will pop up asking what you would like to overwrite the old dataset with the
+tagged one or save new dataset as a separate file. Click **Ok** when you're done.
-Example MATLAB calling code for this function:
+
-```matlab
-dataPath = 'H:\datasets\eeg_ds003645s_hed';
-issueString = validateHedInBids(dataPath);
-if isempty(issueString)
- fprintf('Dataset %s has no HED validation errors\n', dataPath);
-else
- fprintf('Validation errors for dataset %s:\n%s\n', dataPath, issueString);
-end
+You just finished tagging! *HEDTools* generates the final HED string for each event
+by concatenating all tags associated with the event values of that event (separated by commas).
+The final concatenated version is put the string in a new field **HED** in EEG.event.
-```
-````
-In above example assumes that the BIDS dataset was located at `H:\datasets\eeg_ds003645s_hed`.
-We tested it with the [**eeg_ds003645s_hed**](https://github.com/hed-standard/hed-examples/tree/main/datasets/eeg_ds003645s_hed) available on GitHub.
-You can download and use this test data or set `dataPath` to the root directory of your own dataset.
+### HED-based epoching
-#### Calls to HED remodeling tools
+The EEGLAB *pop_epoch* function extracts data epochs that are time locked to specified event types.
+This function allows you to epoch on one of a specified list of event types as defined by the
+*EEG.event.type* field of the EEG structure.
-Many of the most useful HEDTools functions are packaged in the
-HED remodeling tool suite.
-These tools allow operations such as creating summaries, validating the dataset, and transforming event files to be run on an entire dataset.
+*HEDTools* provides a simple way for extracting data epochs from annotated datasets using
+a much richer set of conditions.
+To use HED epoching, you must have annotated the EEG dataset.
-The following example illustrates a call that creates a summary of the
-experimental conditions for a HED-tagged dataset.
+If the dataset is not tagged, please refer to [Annotating datasets](#annotating-datasets)
+on how to tag a dataset.
-````{admonition} A MATLAB wrapper function for a remodeling operation to create a summary.
-:class: tip
+Start by choosing the menu option: **Tools --> Extract epochs by tags**:
-```matlab
-function runRemodel(remodel_args)
- py.importlib.import_module('hed');
- py.hed.tools.remodeling.cli.run_remodel.main(remodel_args);
-```
+
-Example MATLAB calling code for this function:
-```matlab
+This will bring up a window to specify the options for extracting data epochs:
-dataPath = 'G:\ds003645';
-remodelFile = 'G:\summarize_hed_types_rmdl.json';
-remodel_args = {dataPath, remodelFile, '-b', '-x', 'stimuli', 'derivatives'};
-runRemodel(remodel_args);
+
-```
-````
+The *pop_epochhed* menu is almost identical to the EEGLAB *pop_epoch* menu with the
+exceptions of the first input field (**Time-locking HED tag(s)**) and the second
+input field (**Exclusive HED tag(s)**).
-The command line arguments to the various remodeling functions are
-given in a cell array, rather than a regular MATLAB array.
-For the remodeling operations, first and second operation must be the dataset root
-directory and the remodeling file name, respectively.
-In this example, dataset `ds003645` has been downloaded from [**openNeuro**](https://openneuro.org) to the `G:\` drive.
-The remodeling file used in this example can be found at
-See [**File remodeling quickstart**](FileRemodelingQuickstart.md)
-and [**File remodeling tools**](FileRemodelingTools.md) for
-additional information.
-The wrapper functions are available on GitHub in
-the [**hedtools_wrappers**](https://github.com/hed-standard/hed-examples/tree/develop/src/matlab_scripts/hedtools_wrappers) directory.
+Instead of passing in or selecting from a group of unique event types,
+the user passes in a comma separated list of HED tags.
+For each event all HED tags in this list must be found for a data epoch to be generated.
+Clicking the adjacent button (with the label …) will open a search tool to help you select
+HED tags retrieved from the dataset.
-### MATLAB functions for Python
+
-The following table lists the relevant MATLAB functions that are available.
-You should refer to the help facility for your version of MATLAB to get the details of what is
-supported for your version of MATLAB.
+When you type something in the search bar, the dialog displays a list below containing possible matches.
+Pressing the "up" and "down" arrows on the keyboard while the cursor is in the search bar moves to the
+next or previous tag in the list.
-| MATLAB command | Purpose |
-| -------------- | --------|
-| `pyenv` | Setup your Python environment in MATLAB.