From 6ad41142a2c63c47a567eaf4a8040a26963b84c6 Mon Sep 17 00:00:00 2001 From: Kay Robbins <1189050+VisLab@users.noreply.github.com> Date: Wed, 31 Jan 2024 08:18:20 -0600 Subject: [PATCH] Updated docs and removed matlab code from this repo --- README.md | 36 +--- docs/requirements.txt | 2 +- docs/source/FileRemodelingQuickstart.md | 5 +- docs/source/FileRemodelingTools.md | 166 ++++++++++++------ docs/source/HedMatlabTools.md | 142 ++++++++------- docs/source/HedValidationGuide.md | 6 +- src/README.md | 32 +++- .../bids/find_event_combinations.ipynb | 2 +- 8 files changed, 234 insertions(+), 157 deletions(-) diff --git a/README.md b/README.md index 2c22fe61..7f82089f 100644 --- a/README.md +++ b/README.md @@ -14,38 +14,14 @@ Most of these datasets have complete versions available on [**openNeuro**](https://openneuro.org/). See [**datasets/README.md**](./datasets/README.md) for details. -The [**hedcode**](https://github.com/hed-standard/hed-examples/tree/main/hedcode) -subdirectory contains MATLAB scripts, Python Jupyter Notebooks, -and Python scripts with direct calls to HedTools. -The repository also contains example code in python and matlab. -See [**hedcode/README.md**](src/README.md) for details. -The Python scripts and notebooks require the installation of -`hedtools` whose [**installation**](./README.md#installation-of-hedtools) -is described below. +The [**src**](https://github.com/hed-standard/hed-examples/tree/main/hedcode) +subdirectory contains Python Jupyter notebooks demonstrating calls to HedTools. +For MATLAB support for HED see the [**hed-matlab**](https://github.com/hed-standard/hed-matlab) +GitHub repository. The [**docs**](https://github.com/hed-standard/hed-examples/tree/main/docs) -subdirectory contains the main documentation for this and other HED resources. -The [**HED GitHub organization repository**](https://github.com/hed-standard/) +subdirectory contains the source documentation for this and other HED resources. +The [**HED GitHub organization**](https://github.com/hed-standard/) gathers the HED supporting resources, all of which are open source. -### Installation of hedtools - -The most of the Python-related resources in this repository -require the installation of the HEDTools Python module, which can be -installed using `pip` or directly from its GitHub repository as follows: - -To use `pip` to install `hedtools` from PyPI: - - ``` - pip install hedtools - ``` - -To install directly from the -[GitHub](https://github.com/hed-standard/hed-python) repository: - - ``` - pip install git+https://github.com/hed-standard/hed-python/@master - ``` - -HEDTools require python 3.7 or greater. diff --git a/docs/requirements.txt b/docs/requirements.txt index f6d03171..63e26e28 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1,3 +1,3 @@ Sphinx>=5.2.2 myst-parser>=0.18.1 -sphinx_rtd_theme>=1.0.0 +sphinx-rtd-theme>=1.0.0 diff --git a/docs/source/FileRemodelingQuickstart.md b/docs/source/FileRemodelingQuickstart.md index 11b49abc..b9c90ae0 100644 --- a/docs/source/FileRemodelingQuickstart.md +++ b/docs/source/FileRemodelingQuickstart.md @@ -309,7 +309,7 @@ because *remove_parent_event* is false. rows in the data file no longer represent trials, but individual events within the trial.) Next we specify how the new events are generated in the *new_events* dictionary. -Each new event has a name, which is a key in the *new_events* dictionary. +Each type of new event has a name, which is a key in the *new_events* dictionary. Each key is associated with a dictionary specifying the values of the following parameters. @@ -338,7 +338,8 @@ The AOMIC data report indicates that the stop signal lasted 500 ms. The copy columns indicate which columns from the parent event should be copied to the newly-created event. We would like to transfer the *response_accuracy* and the *response_hand* information to the response event. -*Note:* Copy columns is an optional key in the new events dictionary. If you do not want to carry over any column values from the parent event to the new events you can omit this key. +*Note:* Copy columns is an optional key in the *new_events* dictionary. +If you do not want to carry over any column values from the parent event to the new events you can omit this key. The final remodeling file can be found at: diff --git a/docs/source/FileRemodelingTools.md b/docs/source/FileRemodelingTools.md index ce16c1d1..d9b0fbc5 100644 --- a/docs/source/FileRemodelingTools.md +++ b/docs/source/FileRemodelingTools.md @@ -661,15 +661,20 @@ Except for the validation summary, the underlying remodeling code raises excepti ### Errors in the remodel file Each operation requires specific parameters to execute properly. -The necessary parameters for every operation are defined using [json schema](https://json-schema.org/). -When passing a remodeler file to the `run_remodel` program the remodeler validator is called, -which compiles the json schema from individual operation -and validates the remodeler file against the compiled json schema. +The underlying implementation for each operation defines these parameters using a [**json schema**](https://json-schema.org/) +as the `PARAMS` property of the operation's class definition. +The use of the JSON schema allows the remodeler to specify and validate requirements on most of an +operation's parameters using standardized methods. + +The remodeler [**validator**](https://raw.githubusercontent.com/hed-standard/hed-python/develop/hed/tools/remodeling/validator.py) +compiles a JSON schema for the remodeler from individual operations and validates +the remodel file against the compiled JSON schema. The validator should always before executing any remodel operations. -If there are any errors in the remodel file, -no operations are run, but the errors for all operations in the list are reported. +For example, the command line [**run_remodel**](https://raw.githubusercontent.com/hed-standard/hed-python/develop/hed/tools/remodeling/cli/run_remodel.py) +program calls the validator before executing any operations. +If there are errors, `run_remodel` reports the errors for all operations and exits. This allows users to correct errors in all operations in one pass without any data modification. -The [**HED online tools**](https://hedtools.ucsd.edu/hed) are particularly useful for debugging +The [**HED online tools**](https://hedtools.org/hed) are particularly useful for debugging the syntax and other issues in the remodeling process. (execution-time-remodel-errors-anchor)= @@ -728,7 +733,7 @@ The parameters for each operation are listed in An operation may have both required and optional parameters. Optional parameters may be omitted if unneeded, but all parameters are specified in the "parameters" section of the dictionary. -The full specification of the remodel file is also provided as a [json schema](https://json-schema.org/). +The full specification of the remodel file is also provided as a [**JSON schema**](https://json-schema.org/). The remodeling JSON files should have names ending in `_rmdl.json` to more easily distinguish them from other JSON files. @@ -2466,22 +2471,41 @@ Sidecar: (remodel-implementation-anchor)= ## Remodel implementation -Operations are defined as classes that extent `BaseOp` regardless of whether +Operations are defined as classes that extend `BaseOp` regardless of whether they are transformations or summaries. However, summaries must also implement an additional supporting class that extends `BaseSummary` to hold the summary information. In order to be executed by the remodeling functions, an operation must appear in the `valid_operations` dictionary. -All operations must provide have the attributes `NAME`, specifying the operation name (string), -`PARAMS`, a dictionary specifying the parameters of the operation as a json schema, -a constructor that extends the base class constructor, a `do_ops` method and a `validate_input_data` method. +Each operation class must have a `NAME` class variable, specifying the operation name (string) and a +`PARAMS` class variable containing a dictionary of the operation's parameters represented as a json schema. +The operation's constructor extends the `BaseOp` class constructor by calling: + +````{admonition} A remodel operation class must call the BaseOp constructor first. +:class: tip +```python + super().__init__(parameters) +``` +```` + +A remodel operation class must implement the `BaseOp` abstract methods `do_ops` and `validate_input_data`. ### The PARAMS dictionary -The class-wide `PARAMS` dictionary specifies the required and optional parameters of the operation as a [json schema](https://json-schema.org/). We currently use draft-2020-12. The basic vocabulary allows specifying the type of parameters that are expected, whether a parameter is required or optional, but it is also possible to add dependencies between parameters. More information can be found in the [documentation](https://json-schema.org/learn/getting-started-step-by-step). +The class-wide `PARAMS` dictionary specifies the required and optional parameters of the operation as a [**JSON schema**](https://json-schema.org/). +We currently use draft-2020-12. +The basic vocabulary allows specifying the type of parameters that are expected and +whether a parameter is required or optional. + +It is also possible to add dependencies between parameters. More information can be found in the JSON schema +[**documentation**](https://json-schema.org/learn/getting-started-step-by-step). -On the highest level the type should always be specified as an object, as the parameters are always provided as a dictionary or json object. Under the properties key, the expected parameters should be listed, along with what datatype is expected for every parameter. The specification can be nested, for example, the rename columns operation requires a parameter column_mapping, which should be a json object whose keys are any valid string, and whose values are also strings. This is represented in json schema in the following way: +On the highest level the type should always be specified as an object, as the parameters are always provided as a dictionary or json object. +Under the properties key, the expected parameters should be listed, along with what datatype is expected for every parameter. +The specifications can be nested, for example, the `rename_columns` operation requires a parameter `column_mapping`, +which should be a JSON object whose keys are any valid string, and whose values are also strings. +This is represented in the following way: ```json { @@ -2508,9 +2532,19 @@ On the highest level the type should always be specified as an object, as the pa } ``` -The PARAMS dictionary is read by the validator and compiled into a single json schema which represents the specification of an entire remodeler file. +The `PARAMS` dictionaries for all available operations are read by the `validator` and compiled into a +single JSON schema which represents the specification for remodeler files. +The `properties` dictionary explicitly specifies the parameters that are allowed for this operation. +The `required` list specifies which parameters must be included when calling operation. +Parameters that are not required may be omitted in the operation call. +The `"additionalProperties": false` is the way that JSON schema use to +indicate that no other parameters are allowed in the call to the operation. -There is one limitation to json schema vocabulary. Although it can handle specific dependencies between keys in the data, it cannot validate the data that is provided in the json file against other data in the json file. For example, if the requirement is a list of elements whose length should be specified by another parameter, json schema does provide a vocabulary for setting this dependency. Instead, we handle these type of dependencies in the `validate_input_data` method. +A limitation to JSON schema representations is that although it can handle specific dependencies between keys in the data, +it cannot validate the data that is provided in the JSON file against other data in the JSON file. +For example, if the requirement is a list of elements whose length should be specified by another parameter, +JSON schema does not provide a vocabulary for setting this dependency. +Instead, we handle these type of dependencies in the `validate_input_data` method. (operation-class-constructor-anchor)= ### Operation class constructor @@ -2538,14 +2572,18 @@ values to class properties. Validation takes place before the operation classes (the-do_op-implementation-anchor)= ### The do_op implementation +The remodeling script is meant to be executed by the `Dispatcher`, +which keeps a compiled version of the remodeling script to execute on each tabular file to be remodeled. The main method that must be implemented by each operation is `do_op`, which takes -an instance of the `Dispatcher` class as the first parameter and a Pandas `DataFrame` -representing the event file as the second parameter. -A third required parameter is a name used to identify the event file in error messages and summaries. +an instance of the `Dispatcher` class as the first parameter and a Pandas [`DataFrame`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) +representing the tabular file as the second parameter. +A third required parameter is a name used to identify the tabular file in error messages and summaries. This name is usually the filename or the filepath from the dataset root. An additional optional argument, a sidecar containing HED annotations, only need be included for HED operations. +Note that the `Dispatcher` is responsible for holding the appropriate version of the HED schema if +HED remodeling operations are included. The following example shows a sample implementation for `do_op`. @@ -2571,16 +2609,17 @@ This operation is performed by the `Dispatcher` static method `post_proc_data`. ### The validate_input_data implementation -This method exist to handle additional input data validation that cannot be specified in json schema. -It is a class method which is called by the validator. If there is no additional validation to be done, +This method exist to handle additional input data validation that cannot be specified in JSON schema. +It is a class method which is called by the `validator`. +If there is no additional validation to be done, a minimal implementation of this method should take in a dictionary with the operation parameters and return an empty list. -In case additional validation is required, the method should directly implement validation and return a list of user friendly -error messages (string) if validation fails, or an empty list if there are no errors. +In case additional validation is required, the method should directly implement validation and return a list of user-friendly +error messages (strings) if validation fails, or an empty list if there are no errors. -The following implementation of `validate_input_data` method, for the operation factor hed tags, checks whether -the parameter `query_names` is the same length as the input for parameter `queries`, since the names specified in -the first are meant to represent the queries provided in the latter. The check only takes place if `query_names` exist, -since naming is handled automatically otherwise. +The following implementation of `validate_input_data` method, for the `factor_hed_tags` operation, +checks whether the parameter `query_names` is the same length as the input for parameter `queries`, +since the names specified in the first parameter are meant to represent the queries provided in the latter. +The check only takes place if `query_names` exists, since naming is handled automatically otherwise. ```python @staticmethod @@ -2620,8 +2659,8 @@ summary name is already in the dispatcher's `summary_dict`. If that summary is not yet in the `summary_dict`, the operation creates a `BaseSummary` object for its summary (e.g., `ColumnNameSummary`) and adds this object to the dispatcher's `summary_dict`, -otherwise the operation fetches the `BaseSummary` object from -It then asks its `BaseSummary` object to update the summary based on the dataframe +otherwise the operation fetches the `BaseSummary` object from the dispatcher's `summary_dict`. +It then asks this `BaseSummary` object to update the summary based on the `DataFrame` as explained in the next section. (additional-requirements-for-summarization-anchor)= @@ -2632,7 +2671,7 @@ This class is used to hold and accumulate the information specific to the summar This support class must implement two methods: `update_summary` and `get_summary_details`. The `update_summary` method is called by its associated `BaseOp` operation during the `do_op` -to update the summary information based on the current dataframe. +to update the summary information based on the current `DataFrame`. The `update_summary` information takes a single parameter, which is a dictionary of information specific to this operation. @@ -2644,7 +2683,7 @@ specific to this operation. ```` In the example [do_op for ColumnNamesOp](implementation-of-do-op_summarize-column-names-anchor), -the dictionary is contains keys for `name` and `column_names. +the dictionary contains keys for `name` and `column_names. The `get_summary_details` returns a dictionary with the summary-specific information currently in the summary. @@ -2662,33 +2701,62 @@ implementation ### Validator implementation -The required input for the Remodeler is specified in json-schema format. See also [The PARAM dictionary](#the-params-dictionary). -Since every operation has its own parameter requirements, the complete json-schema is compiled from the specifications in the individual operations before validation. -The compiler takes the specification of the BASE_ARRAY. We then specify that each item in the BASE_ARRAY should follow the OPERATION_DICT specification. -To get all the requirements of the individual operations we use the template parameter specification, which sets up the dependency between the operation value and parameter value. -We then get the PARAM dictionary from every operation and add it as the parameter specification. -The input Remodel file is then validated directly against the compiled json-schema using [jsonschema](https://python-jsonschema.readthedocs.io/en/stable/). +The required input for the remodeler is specified in JSON format and must follow +the rules laid out by the JSON schema. +The parameters in the remodeler file must conform to the properties specified +in the corresponding JSON schema associated with each operation. +The errors are retrieved from the validator but are not passed on directly but instead +modified for display as user-friendly error messages. + +Validation errors are organized by stages as follows. + +#### Stage 0: top-level structure -The errors are retrieved from the jsonschema validator but are not passed on directly. Instead, we process the errors provided by json schema and modify them to user friendly error messages. +Stage 0 refers to the top-level structure of the remodel JSON file. +As specified by the validator's `BASE_ARRAY`, +a JSON remodel file must be an array of operation dictionaries containing at least 1 item. -Validation errors can occur on different levels. The levels refer to the depth of the error in terms of nesting in the json schema. +#### Stage 1: operation dictionary format -Zero is the base array/list of operations. It validates two properties, type and minItems. Each remodeler file should be an array/list with at least one item. +Stage 1 validation refers the structure of the individual operations as specified by the validator's `OPERATION_DICT`. +Every operation dictionary should have exactly the keys: `operation`, `description`, and `parameters`. -The first level is the Operation dictionary. Every item in the base array should be an Operation Dictionary. It validates three properties, type, required, and additional properties. Every operation dictionary should be a dictionary/object, it should have the keys, operation, description, and parameters. It should have no other keys. +#### Stage 2: operation dictionary values -The second level concerns the values or input to the keys in the operation dictionary. It validates several properties. Some validation of individual remodeler parameters already take place on this level. +Stage 2 validates the values associated with the keys in each operation dictionary. +The `operation` and `description` keys should have a string values, +while the `parameters` key should have a dictionary value. -Type: This validates the type of input to an operation dictionary key. It should be a string for operation, a string for description, and an object/dictionary for parameters. The type of parameter values are validated on the next level. +Stage 2 validation also verifies that the operation value is one of the valid operations as +enumerated in the `valid_operations` dictionary. -Enum: This validates whether the string provided in the operation key is a valid operation. +Several checks are also applied to the `parameters` dictionary. +The keys in the `parameters` dictionary must appear as keys in the `properties` specification in +the operation's schema. +Further the properties listed as `required` in the schema must appear as keys in the `parameters` dictionary. -Required: This validates the keys in the parameters object. It checks if all keys required on the first level of the parameters object are there. +If additional properties are not allowed, as designated by `"additionalProperties": False` in the JSON schema, +the validator verifies that parameters not mentioned in the schema do not appear. -Additional properties: This validates the keys in the parameters object. It checks if none of the provided keys are outside of the specification. +If the schema for the operation has a `dependentRequired` dictionary, the validator +verifies that the indicated keys are present only if the listed parameter values are also present. +For example, the `factor_column_op` only allows the `factor_names` parameter if the `factor_values` +parameter is also present. Otherwise, the operation automatically generates the factor names. -Dependent required: This validates the keys in the parameters object. Certain keys may only be required if another key is present. If this case it checks whether this dependency requirement is met. +#### Later validation stages -All higher levels concern validation of the values given within the parameter object. They are all handled in a general way. The user is provided with the operation index, name and the 'path' of the value that is invalid. Note that while parameters always contains an object, the values in parameters can be of any type. So parameter values can be objects, whose values might also be expected to be objects, or arrays, or arrays of objects. Right now, there are appropriate messages for the following properties, which is everything that is used by the remodeler. When another property, one that is not part of the following list, is added to a operation specification, an appropriate error message needs to be added in the validator. +Later stages in validation concern the values given within the parameter object, which can be nested to an arbitrary level +and are handled in a general way. +The user is provided with the operation index, name and the 'path' of the value that is invalid. +Note that while parameters always contains an object, the values in parameters can be of any type. +Thus, parameter values can be objects whose values might also be expected to be objects, arrays, or arrays of objects. +The validator has appropriate messages for these properties, +but if a property is added with additional requirements, additional error messages may need to be added to the validator. -When validation against json-schema passes, additional data specific validation is performed. For each operation, the validate_input_data method is called to verify input data that is outside of the scope of json schema. Also see the [validate_input_data method](#the-validate_input_data-implementation). +When validation against JSON schema passes, +the validator performs additional data-specific validation by calling `validate_input_data` +for each operation to verify that input data satisfies the +constraints that fall outside the scope of JSON schema. +Also see [**The validate_input_data implementation**](#the-validate_input_data-implementation) and +[**The PARAMS dictionary**](#the-params-dictionary) sections for additional information. +requirements, diff --git a/docs/source/HedMatlabTools.md b/docs/source/HedMatlabTools.md index 1bced03e..77e45256 100644 --- a/docs/source/HedMatlabTools.md +++ b/docs/source/HedMatlabTools.md @@ -1,70 +1,71 @@ # HED MATLAB tools -There are currently three types of support available for HED (Hierarchical Event Descriptors) supports in MATLAB: - -* [**HED services in MATLAB**](hed-services-matlab-anchor) - web services called from MATLAB scripts -* [**EEGLAB plug-in integration**](eeglab-integration-anchor) - EEGLAB plugins and other HED support -* [**Python HEDTools in MATLAB**](python-hedtools-in-matlab-anchor) - explains how to call the HED python tools from within MATLAB. - -HED services allow MATLAB programs to request the same services that are available -through the browser at [https://hedtools.edu/hed](https://hedtools.edu/hed). - -(hed-services-matlab-anchor)= -## HED services in MATLAB - -HED RESTful services allow programs to make requests directly to the -HED online tools available at -[https://hedtools.edu/hed](https://hedtools.edu/hed) or -in a locally-deployed docker module. -See [**HED-web**](https://hed-web.readthedocs.io/en/latest/index.html) -for additional information on the deployment. - -The [**runAllTests.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/runAllTests.m) -is a main script that runs all the example code and reports whether -the code runs successfully. -You should start by running this script to make sure everything is working on your system, +There are currently three types of support available for HED (Hierarchical Event Descriptors) in MATLAB: + +* [**HED web services in MATLAB**](hed-web-services-in-matlab-anchor) - web services called from MATLAB scripts. +* [**EEGLAB plug-in integration**](eeglab-integration-anchor) - EEGLAB plugins and other native MATLAB. +* [**Python HEDTools in MATLAB**](python-hedtools-in-matlab-anchor) - calling HED python tools from within MATLAB. + + +(hed-web-services-in-matlab-anchor)= +## HED web services in MATLAB + +HED web services allow MATLAB programs to request the same services that are available +through the online tools. +These services are available through the [**https://hedtools.org/hed**](https://hedtools.org/hed) server. +Alternatively, these services can be accessed through a locally-deployed docker module. +See the [**hed-web**](https://hed-web.readthedocs.io/en/latest/index.html) +GitHub repository documentation for additional information on the docker deployment. +The MATLAB code to run demos of these services is available on the +[**hed-matlab**](https://github.com/hed-standard/hed-matlab/tree/main/hedmat/web_services) +GitHub repository. + +The following MATLAB code demos are available to show how to access HED web services. + +| Target | MATLAB source | Purpose | +| ------ |-----------------|-----------| +| Overall | [**runAllDemos.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/runAllDemos.m) | Harness for running all demos. | +| Overall | [**demoGetServices.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/demoGetServices.m) | List available services. | +| Events | [**demoEventServices.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/testEventServices.m) | Validation, conversion, sidecar generation. | +| Events | [**demoEventSearchServices.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/demoEventSearchServices.m) | Search, assembly. | +| Schema | *in progress* | For schema library developers. | +| Sidecars | [**demoSidecarServices.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/demoSidecarServices.m) | Validation, conversion, extraction, merging. | +| Spreadsheets | [**demoSpreadsheetServices.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/demoSpreadsheetServices.m) | Validation, conversion. | +| Strings | [**demoStringServices.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/demoStringServices.m) | Validation, conversion. | + + +The [**runAllDemos.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/runAllDemos.m) +script runs all the demo code and reports whether +the demos run successfully. +Before using the HED web services from MATLAB, +you should run this script to make sure everything is working on your system, that you have Internet access, and that the HED services are available. -This script also demonstrates how to call the individual test functions. -Each test function takes a host URL as a parameter and returns a list of errors. -The individual test scripts illustrate how to call each type of available web service. - - -| Target | MATLAB source| Purpose | -| ------ | ------------- | ------- | -| Overall | [**runAllTests.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/runAllTests.m) | Harness for running all tests. | -| Overall | [**testGetServices.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/testGetServices.m) | List available services. | -| Events | [**testEventServices.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/testEventServices.m) | Validation, conversion, sidecar generation. | -| Events | [**testEventSearchServices.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/testEventSearchServices.m) | Search, assembly.| -| Schema | *in progress* | For schema library developers. | -| Sidecars | [**testSidecarServices.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/testSidecarServices.m) | Validation, conversion, extraction, merging. | -| Spreadsheets | [**testSpreadsheetServices.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/testSpreadsheetServices.m) | Validation, conversion.| -| Strings | [**testStringServices.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/testStringServices.m) | Validation, conversion. | - +This script also demonstrates how to call the individual demo functions. +Each demo function takes a host URL as a parameter and returns a list of errors. +The demos all use demo data read by the +[**getDemoData**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/getDemoData.m) +function, which returns a MATLAB `struct` containing all needed test data. +The individual demo scripts illustrate how to call each type of available web service. +(overview-of-service-requests-anchor)= ### Overview of service requests Calling HED services from MATLAB requires the following steps: -1. **Set up a session**: - 1. Establish a session by requesting a CSRF token and a cookie. - 2. Construct a header array using the token and the cookie. -2. **Create a request structure**. -3. **Make a request** using the MATLAB `webwrite`. -4. **Decode the response** returned from `webwrite`. +1. [**Set up a session**](setting-up-a-session-from-matlab-anchor). +2. [**Create a request structure**](creating-a-request-structure-anchor). +3. [**Make a request**](making-a-service-request-anchor). +4. [**Decode the response**](decoding-a-service-response-anchor). Usually, you will do the first step (the session setup) once at the beginning of your script to construct a fixed session header that can be used in subsequent requests in your script. +(setting-up-a-session-from-matlab-anchor)= ### Setting up a session from MATLAB The goal of the session setup is to construct a header that can be used in subsequent web requests. -The first step is to call the [**getHostOptions.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/getHostOptions.m). -This function constructs the services URL from the host URL. -The function also makes a service request to obtain a CSRF token and a cookie. -The function then constructs a header and calls the MATLAB `weboptions` function -to get an options object suitable for use with the MATLAB `webwrite` function -use in all of our examples. +The first step is to call the [**getHostOptions.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/getHostOptions.m). (setting)= `````{admonition} Establish a session. @@ -80,6 +81,12 @@ The `host` should be set to the URL of the webserver that you are using. The call to `getHostOptions`, only needs to be done once at the beginning of your session. The `servicesURL` and the `options` can be used for all of your subsequent requests. +This function constructs the services URL from the host URL. +The function also makes a service request to obtain a CSRF token and a cookie using +[**getSessionInfo**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/getSessionInfo.m). +The function then constructs a header and calls the MATLAB `weboptions` function +to get an options object suitable for use with the MATLAB `webwrite` function +use in all of our examples. The `getHostOptions` does all the setup for using the services. As indicated by the code below, all communication is done in JSON. @@ -87,13 +94,6 @@ However, as demonstrated below, the MATLAB `webwrite` function takes a MATLAB `struct` as its `request` parameter and internally converts to the format specified in the header before making the request. -The `Timeout` parameter indicates how many seconds MATLAB will wait for a response -before returning as a failed operation. -The `timeout` value of 120 seconds is sufficient for most situations. -However, but this can be adjusted upward or downward to suit the user. -The `HeaderFields` sets the parameters of HTTP request. - - (gethostoptions-source-anchor)= `````{admonition} Source for getHostOptions. :class: tip @@ -111,9 +111,13 @@ function [servicesUrl, options] = getHostOptions(host) ``` ````` -In the following examples, we assume that `getHostOptions` has been called to retrieve -`servicesUrl` and `options` for use in the session. +The `Timeout` parameter indicates how many seconds MATLAB will wait for a response +before returning as a failed operation. +The `timeout` value of 120 seconds is sufficient for most situations. +However, this can be adjusted upward or downward to suit the user. +The `HeaderFields` sets the parameters of HTTP request. +(creating-a-request-structure-anchor)= ### Creating a request structure The request structure is a MATLAB `struct` which must have a `service` field and can have an @@ -121,7 +125,10 @@ arbitrary number of fields depending on which service is being requested. The simplest service is `get_services`, which returns a string containing information about the available services. -This service requires no additional parameters. +This service requires no additional parameters. +In this and other examples, we assume that `getHostOptions` has been called to retrieve +`servicesUrl` and `options` for use in the session. + `````{admonition} Request a list of available HED web services. :class: tip @@ -169,28 +176,29 @@ Another possibility is to provide a URL for the schema. The most-commonly used option is to use `schema_version` to indicate one of the supported versions available in the [**hedxml**](https://github.com/hed-standard/hed-specification/tree/master/hedxml) directory of the -[**hed-specification**](https://github.com/hed-standard/hed-specification) repository on GitHub. +[**hed-schemas**](https://github.com/hed-standard/hed-schemas) repository on GitHub. (create-request-sidecar-validate-anchor)= `````{admonition} Create a request for the sidecar_validate web service. :class: tip ```matlab -jsonText = fileread('../../../datasets/eeg_ds003645s_hed/task-FacePerception_events.json'); +jsonText = fileread('../../datasets/eeg_ds003645s_hed_demo/task-FacePerception_events.json'); request = struct('service', 'sidecar_validate', ... - 'schema_version', '8.0.0', ... + 'schema_version', '8.2.0', ... 'json_string', jsonText, ... 'check_for_warnings', 'on'); ``` ````` This example reads the JSON sidecar to be validated as a character array into the variable `jsonText` -and makes a request for validation using HED8.0.0.xml. +and makes a request for validation using `HED8.2.0.xml`. The request indicates that validation warnings as well as errors should be included in the response. If you wish to exclude warnings, use `off` instead of `on` as the `check_for_warnings` field value. -The [**testSidecarServices.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/testSidecarServices.m) +The [**demoSidecarServices.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/demoSidecarServices.m) function shows complete examples of the various HED services for JSON sidecars. +(making-a-service-request-anchor)= ### Making a service request The HED services all use the MATLAB `webwrite` to make HED web service requests. @@ -207,7 +215,7 @@ outputReport(response, 'Example: validate a JSON sidecar'); ``` ````` -The [**outputReport.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/outputReport.m) +The [**outputReport.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/outputReport.m) MATLAB script outputs the response in readable form with a user-provided table. If the web server is down or times out during a request, diff --git a/docs/source/HedValidationGuide.md b/docs/source/HedValidationGuide.md index cb9b9302..fc5f592f 100644 --- a/docs/source/HedValidationGuide.md +++ b/docs/source/HedValidationGuide.md @@ -173,10 +173,10 @@ HED validation in MATLAB is currently done by accessing the HED online tools as #### Direct access to services Users can access these services directly by calling using the HED MATLAB web services functions as explained in -[**HED services in MATLAB**](./HedMatlabTools.md#hed-services-in-matlab). -Download the [**web_services**](https://github.com/hed-standard/hed-examples/tree/main/hedcode/matlab_scripts/web_services) +[**HED web services in MATLAB**](./HedMatlabTools.md#hed-web-services-in-matlab). +Download the [**web_services**](https://github.com/hed-standard/hed-matlab/tree/main/hedmat/web_services) directory from GitHub and include this directory in your MATLAB path. -The [**runAllTests.m**](https://raw.githubusercontent.com/hed-standard/hed-examples/main/hedcode/matlab_scripts/web_services/runAllTests.m) +The [**runAllDemos.m**](https://raw.githubusercontent.com/hed-standard/hed-matlab/main/hedmat/web_services/runAllDemos.m) script calls all the services on test data. #### Access through EEGLAB diff --git a/src/README.md b/src/README.md index 823e6c02..18196a1e 100644 --- a/src/README.md +++ b/src/README.md @@ -9,17 +9,41 @@ gathers the HED supporting resources, all of which are open source. The main documentation for this and other HED resources is available through the -[**HED-examples**](https://hed-examples.readthedocs.io/en/latest/index.html) documentation, +[**HED-resources**](https://www.hed-resources.org) documentation, whose source is contained in the [**docs**](https://github.com/hed-standard/hed-examples/tree/main/docs) subdirectory of this repository. -The [**hedcode**](https://github.com/hed-standard/hed-examples/tree/main/hedcode) -subdirectory contains MATLAB scripts, Python Jupyter Notebooks, -and Python scripts with direct calls to HedTools. +The [**src**](https://github.com/hed-standard/hed-examples/tree/main/src) +subdirectory of the [**hed-examples**] GitHub repository contains Python Jupyter +notebooks with examples of how to call the HEDTools. + +For MATLAB support for HED see the [**hed-matlab**](https://github.com/hed-standard/hed-matlab) +GitHub repository. The [**datasets**](https://github.com/hed-standard/hed-examples/tree/main/datasets) subdirectory contains datasets for testing various aspects of HED. These datasets have stubs for actual imaging data in order to reduce their size. Most of these datasets have complete versions available on [**openNeuro**](https://openneuro.org/). + +### Installation of hedtools + +The most of the Python-related resources in this repository +require the installation of the HEDTools Python module, which can be +installed using `pip` or directly from its GitHub repository as follows: + +To use `pip` to install `hedtools` from PyPI: + + ``` + pip install hedtools + ``` + +To install directly from the +[GitHub](https://github.com/hed-standard/hed-python) repository: + + ``` + pip install git+https://github.com/hed-standard/hed-python/@master + ``` + +HEDTools require python 3.7 or greater. diff --git a/src/jupyter_notebooks/bids/find_event_combinations.ipynb b/src/jupyter_notebooks/bids/find_event_combinations.ipynb index d64103b6..6093d014 100644 --- a/src/jupyter_notebooks/bids/find_event_combinations.ipynb +++ b/src/jupyter_notebooks/bids/find_event_combinations.ipynb @@ -78,7 +78,7 @@ "# Variables to set for the specific dataset\n", "data_root = 'T:/summaryTests/ds002718-download'\n", "output_path = ''\n", - "exclude_dirs = ['stimuli', 'derivatives', 'code', 'sourcecode']\n", + "exclude_dirs = ['stimuli', 'derivatives', 'code', 'sourcedata']\n", "\n", "# Construct the key map\n", "key_columns = [ \"trial_type\", \"value\"]\n",