UnicodeDecodeError when using HEDtools5.0.0

[ManuelaJaeger]

Hi,

I use the HED online tools (hedtools.org/hed/events) to assemble HED annotations based on a tsv and json file for german speech stimuli. This step worked out well, even our event data contains special german characters like ä,ü,ö. Now, I want to use the expanded file (containing the HED annotation) as an input to eeglab to create events in the EEG dataset and to import the HED annotations so that I can more easily epoch the data based on specific properties of the speech stimulus (a specific word or phoneme presented). Based on the HEDtools5.0.0 I managed to import the tsv file and the json file in the Ctagger. Now I have 2 questions:

Q1:
All validation steps were successful and did not show any error, but when I try to epoch the data based on the "extract epochs by tags" option it gives me the following error:

error_type: 'UnicodeDecodeError'
error_msg: '[UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 1423: invalid continuation byte]'

Could it be that the validation web service that is called in the background cannot handle special characters such as 0xe4 = ä? Is it possible to disable the process?

Q2:
Is there a easier way to import the events and HED strings directly into eeglab, without running the Ctagger? Currently, I need to import all event information within a loop in the EEG.event struct to use it with HED and additionally use the Ctagger to load in the tsv and json file. In my idea the HED strings should contain all important information but adding the HED string to the EEG.event struct seems to be not used by the HEDtoolbox, even I think there was a field HED in one of the former versions of the HEDtools. Where can I add the HED strings directly to speed up the process or what kind of function can I use for doing this?

Thank you,
Manu

[neuromechanist]

@dungscout96, this is a very interesting use case. I am glad that Unicode tags are working—and being used—in HED, but it seems that EEGLAB has difficulty handling them. I am not sure if we can do much about it, though. I am looking forward to hearing your thoughts.