-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Creating dataset with R leads to nwbinspector errors "incorrect type for 'subject_id' (got 'bytes', expected 'str')")" #483
Comments
Hi there @nrsc, welcome to NWB Inspector. A few small suggestions for making such big issues easier to read for other fellow developers Consider styling code blocks on GitHub using '```' enclosure, such as /home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py:668: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.1.3 because version 1.8.0 is already loaded.
return func(args[0], **pargs)
/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py:668: UserWarning: Ignoring cached namespace 'core' version 2.2.4 because version 2.5.0 is already loaded.
return func(args[0], **pargs)
Traceback (most recent call last):
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1262, in construct
obj = self.new_container(cls, builder.source, parent, builder.attributes.get(self._spec.id_key()),
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1275, in new_container
obj.init(**kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 667, in func_call
pargs = check_args(args, kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 660, in check_args
raise ExceptionType(msg)
TypeError: Subject.init: incorrect type for 'age' (got 'bytes', expected 'str'), incorrect type for 'sex' (got 'bytes', expected 'str'), incorrect type for 'species' (got 'bytes', expected 'str'), incorrect type for 'subject_id' (got 'bytes', expected 'str')
...
hdmf.build.errors.ConstructError: (root/general/subject GroupBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'Subject', 'object_id': '26c03440-b70a-49c2-ab08-99f72047c724'}, 'groups': {}, 'datasets': {'subject_id': root/general/subject/subject_id DatasetBuilder {'attributes': {}, 'data': b'QN22.26.011.1A.38.01'}, 'species': root/general/subject/species DatasetBuilder {'attributes': {}, 'data': b'Macaca nemestrina'}, 'sex': root/general/subject/sex DatasetBuilder {'attributes': {}, 'data': b'M'}, 'age': root/general/subject/age DatasetBuilder {'attributes': {}, 'data': b'P3Y6M'}}, 'links': {}}, "Could not construct Subject object due to: Subject.init: incorrect type for 'age' (got 'bytes_', expected 'str'), incorrect type for 'sex' (got 'bytes_', expected 'str'), incorrect type for 'species' (got 'bytes_', expected 'str'), incorrect type for 'subject_id' (got 'bytes', expected 'str')") Makes it easier to identify things Also consider hiding long stacks of details inside Traceback (most recent call last):
File "/home/nrsc/.local/bin/nwbinspector", line 8, in
sys.exit(inspect_all_cli())
File "/home/nrsc/.local/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/nrsc/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/nrsc/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/nwbinspector/nwbinspector.py", line 275, in inspect_all_cli
messages = list(
File "/home/nrsc/.local/lib/python3.10/site-packages/nwbinspector/nwbinspector.py", line 414, in inspect_all
nwbfile = robust_s3_read(io.read)
File "/home/nrsc/.local/lib/python3.10/site-packages/nwbinspector/utils.py", line 174, in robust_s3_read
raise exc
File "/home/nrsc/.local/lib/python3.10/site-packages/nwbinspector/utils.py", line 169, in robust_s3_read
return command(*command_args, **command_kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/backends/hdf5/h5tools.py", line 500, in read
return super().read(**kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/backends/io.py", line 60, in read
container = self.__manager.construct(f_builder)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/manager.py", line 286, in construct
result = self.__type_map.construct(builder, self, None)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/manager.py", line 814, in construct
return obj_mapper.construct(builder, build_manager, parent)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1232, in construct
subspecs = self.__get_subspec_values(builder, self.spec, manager)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1161, in __get_subspec_values
self.__get_sub_builders(groups, spec.groups, manager, ret)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1212, in __get_sub_builders
ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1161, in __get_subspec_values
self.__get_sub_builders(groups, spec.groups, manager, ret)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1214, in __get_sub_builders
ret[subspec] = manager.construct(sub_builder)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/manager.py", line 282, in construct
result = self.__type_map.construct(builder, self, parent)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/manager.py", line 814, in construct
return obj_mapper.construct(builder, build_manager, parent)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1266, in construct
raise ConstructError(builder, msg) from ex Anyway, as the error tracebacks indicate this does not seem to be an NWB Inspector specific issue but instead indicates via HDMF and PyNWB that we cannot even open the file in Python A simpler approach to debug the interoperability of your R code would be to simply try opening the file with import pynwb
with pynwb.NWBHDF5IO(path="path to your file", mode="r") as io:
nwbfile = io.read() which should give the same errors At the low level, I'm guessing the R HDF5 API is creating a dataset of dtype 'bytes' (variable-length strings) when NWB seems to expect dtype 'S' (fixed length) See https://docs.h5py.org/en/stable/strings.html#reading-strings for more info |
Hi @nrsc , thanks for creating the issue. Try: rhdf5::h5writeDataset(subject_id, nwb, "general/subject/subject_id", variableLengthString=TRUE, encoding="UTF-8")
rhdf5::h5writeDataset(species, nwb, "general/subject/species", variableLengthString=TRUE, encoding="UTF-8")
rhdf5::h5writeDataset(sex, nwb, "general/subject/sex", variableLengthString=TRUE, encoding="UTF-8")
rhdf5::h5writeDataset(age, nwb, "general/subject/age", variableLengthString=TRUE, encoding="UTF-8") |
Thank you @rly. Looks like that may have done the trick and I no loger get the error regarding the string type. Also a special shout out to @CodyCBakerPhD. Thank you for the tips on formatting. Following up on the issue, I am now running into a problem that looks to be associated with hdmf when trying to validate the nwb files with
Following the advice of @CodyCBakerPhD I tried to load the and read the nwb file using python the python terminal, and run into errors also associated with
Thank you all for your time and insight, Scott |
Both of those errors indicate to me a potential version issue. Any version of PyNWB in the last year or more should have the driver argument in Are you working in a clean, fresh Python (conda preferably) environment? Can you copy/paste the results of |
Hey Cody, Here is the output.
|
Thanks As expected, several of those packages are quite out of date Try using latest of each Either in a new, clean environment, do or in current environment pip install -U pynwb
pip install -U hdmf
pip install -U h5py (may cause more strange errors though, which is why a clean environment is suggested) |
Thank you everyone. Packages have been updated and the inspector has been run successfully. |
Hello,
I've been trying to write metadata information into my nwb files using R and the rhdf5 package. I have been successfully able to write my metadata into the general/subject subjection of the file, but when using the nwbinspector to check over the file, I get an error indicating that what seems to have been written into the nwb file is in the incorrect format. I.e., "got bytes, expected str".
subject_id, age, sex, and species is written to the file with the following code, (of course with the variables defined):
I have been trying to change the encoding type, but have had no luck. Any insight into this problem would be much appreciated.
Error output from running nwbinspector on the nwb file after writing in in R:
hdmf.build.errors.ConstructError: (root/general/subject GroupBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'Subject', 'object_id': '26c03440-b70a-49c2-ab08-99f72047c724'}, 'groups': {}, 'datasets': {'subject_id': root/general/subject/subject_id DatasetBuilder {'attributes': {}, 'data': b'QN22.26.011.1A.38.01'}, 'species': root/general/subject/species DatasetBuilder {'attributes': {}, 'data': b'Macaca nemestrina'}, 'sex': root/general/subject/sex DatasetBuilder {'attributes': {}, 'data': b'M'}, 'age': root/general/subject/age DatasetBuilder {'attributes': {}, 'data': b'P3Y6M'}}, 'links': {}}, "Could not construct Subject object due to: Subject.init: incorrect type for 'age' (got 'bytes_', expected 'str'), incorrect type for 'sex' (got 'bytes_', expected 'str'), incorrect type for 'species' (got 'bytes_', expected 'str'), incorrect type for 'subject_id' (got 'bytes', expected 'str')")
Full output including library information:
/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py:668: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.1.3 because version 1.8.0 is already loaded.
return func(args[0], **pargs)
/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py:668: UserWarning: Ignoring cached namespace 'core' version 2.2.4 because version 2.5.0 is already loaded.
return func(args[0], **pargs)
Traceback (most recent call last):
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1262, in construct
obj = self.new_container(cls, builder.source, parent, builder.attributes.get(self._spec.id_key()),
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1275, in new_container
obj.init(**kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 667, in func_call
pargs = check_args(args, kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 660, in check_args
raise ExceptionType(msg)
TypeError: Subject.init: incorrect type for 'age' (got 'bytes', expected 'str'), incorrect type for 'sex' (got 'bytes', expected 'str'), incorrect type for 'species' (got 'bytes', expected 'str'), incorrect type for 'subject_id' (got 'bytes', expected 'str')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/nrsc/.local/bin/nwbinspector", line 8, in
sys.exit(inspect_all_cli())
File "/home/nrsc/.local/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/nrsc/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/nrsc/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/nwbinspector/nwbinspector.py", line 275, in inspect_all_cli
messages = list(
File "/home/nrsc/.local/lib/python3.10/site-packages/nwbinspector/nwbinspector.py", line 414, in inspect_all
nwbfile = robust_s3_read(io.read)
File "/home/nrsc/.local/lib/python3.10/site-packages/nwbinspector/utils.py", line 174, in robust_s3_read
raise exc
File "/home/nrsc/.local/lib/python3.10/site-packages/nwbinspector/utils.py", line 169, in robust_s3_read
return command(*command_args, **command_kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/backends/hdf5/h5tools.py", line 500, in read
return super().read(**kwargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/backends/io.py", line 60, in read
container = self.__manager.construct(f_builder)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/manager.py", line 286, in construct
result = self.__type_map.construct(builder, self, None)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/manager.py", line 814, in construct
return obj_mapper.construct(builder, build_manager, parent)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1232, in construct
subspecs = self.__get_subspec_values(builder, self.spec, manager)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1161, in __get_subspec_values
self.__get_sub_builders(groups, spec.groups, manager, ret)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1212, in __get_sub_builders
ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1161, in __get_subspec_values
self.__get_sub_builders(groups, spec.groups, manager, ret)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1214, in __get_sub_builders
ret[subspec] = manager.construct(sub_builder)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/manager.py", line 282, in construct
result = self.__type_map.construct(builder, self, parent)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/manager.py", line 814, in construct
return obj_mapper.construct(builder, build_manager, parent)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call
return func(args[0], **pargs)
File "/home/nrsc/.local/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1266, in construct
raise ConstructError(builder, msg) from ex
hdmf.build.errors.ConstructError: (root/general/subject GroupBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'Subject', 'object_id': '26c03440-b70a-49c2-ab08-99f72047c724'}, 'groups': {}, 'datasets': {'subject_id': root/general/subject/subject_id DatasetBuilder {'attributes': {}, 'data': b'QN22.26.011.1A.38.01'}, 'species': root/general/subject/species DatasetBuilder {'attributes': {}, 'data': b'Macaca nemestrina'}, 'sex': root/general/subject/sex DatasetBuilder {'attributes': {}, 'data': b'M'}, 'age': root/general/subject/age DatasetBuilder {'attributes': {}, 'data': b'P3Y6M'}}, 'links': {}}, "Could not construct Subject object due to: Subject.init: incorrect type for 'age' (got 'bytes_', expected 'str'), incorrect type for 'sex' (got 'bytes_', expected 'str'), incorrect type for 'species' (got 'bytes_', expected 'str'), incorrect type for 'subject_id' (got 'bytes', expected 'str')")
The text was updated successfully, but these errors were encountered: