Colcon build fails with UnicodeDecodeError for WstringArrays in Foxy, but successful in dashing

[karthiknit1]

Bug report

Required Info:

• Operating System:
  • Windows 10, linux debian 10 and macOs BigSur
• Installation type:
  • from source( ros2 Foxy )
• DDS implementation:
  • default shipped with ros2 foxy

Steps to reproduce issue

In Foxy, do colcon build for ros2 custom message with Wstrings.msg as shown below:

Wstrings.msg

wstring wstring_value
wstring wstring_value_default1 "Hello world!"
wstring wstring_value_default2 "Hellö wörld!"
wstring WSTRING_CONST="Hello world!"
wstring<=22 bounded_wstring_value
wstring<=22 bounded_wstring_value_default1 "Hello world!"
wstring[3] array_of_wstrings
wstring[<=3] bounded_sequence_of_wstrings
wstring[] unbounded_sequence_of_wstrings

wstring<=20[<=10] wstring_bounded_var_array
wstring<=10[<=10] wstring_bounded_var_array_default ['Hellö1','Hellö2','Hellö3','Hellö4'] 

wstring[<=10] wstring_bounded_array
wstring[<=10] wstring_bounded_array_default ["Hellö wörld!","Hello world!",'Hellö wörld!']

wstring<=20[5] wstring_limit_var_array
wstring<=20[5] wstring_limit_var_array_default ["Hellö wörld!1","Hellö wörld!2",'Hellö wörld!3',"Hellö wörld!4",'Hellö wörld!5'] 

wstring<=20 bounded_variable
wstring<=20 bounded_variable_default 'Hellö wörld!1'

wstring<=20[] wstring_limit_variable_length_array
wstring<=20[] wstring_limit_variable_length_array_default ["Hellö wörld!1","Hellö wörld!2",'Hellö wörld!3']

Build fails with the following error:

Error processing idl file: D:\ROS_2\ros2-foxy-20201211-windows-release.amd64\ros2_msg\build\ros2_msg\rosidl_adapter\ros2_msg\msg\TestWstring.idl
  Traceback (most recent call last):
    File "D:/dds/Foxy_source/ros2_foxy/install/share/rosidl_generator_cpp/cmake/../../../lib/rosidl_generator_cpp/rosidl_generator_cpp", line 40, in <module>
      sys.exit(main())
    File "D:/dds/Foxy_source/ros2_foxy/install/share/rosidl_generator_cpp/cmake/../../../lib/rosidl_generator_cpp/rosidl_generator_cpp", line 35, in main
      args.generator_arguments_file,
    File "D:\dds\Foxy_source\ros2_foxy\install\Lib\site-packages\rosidl_generator_cpp\__init__.py", line 40, in generate_cpp
      post_process_callback=prefix_with_bom_if_necessary)
    File "D:\dds\Foxy_source\ros2_foxy\install\Lib\site-packages\rosidl_cmake\__init__.py", line 96, in generate_files
      raise(e)
    File "D:\dds\Foxy_source\ros2_foxy\install\Lib\site-packages\rosidl_cmake\__init__.py", line 74, in generate_files
      idl_file = parse_idl_file(locator)
    File "D:\dds\Foxy_source\ros2_foxy\install\Lib\site-packages\rosidl_parser\parser.py", line 62, in parse_idl_file
      string = locator.get_absolute_path().read_text(encoding='utf-8')
    File "C:\python37\lib\pathlib.py", line 1207, in read_text
      return f.read()
    File "C:\python37\lib\codecs.py", line 322, in decode
      (result, consumed) = self._buffer_decode(data, self.errors, final)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 985: invalid start byte

Expected behavior

Build should be successful

Actual behavior

Build failed

Additional information

Build is successful in ros2 dashing but broken in foxy.
In the Wstrings.msg file, these are the fields which are actually causing the failure in foxy.

wstring<=10[<=10] wstring_bounded_var_array_default ['Hellö1','Hellö2','Hellö3','Hellö4'] 
wstring[<=10] wstring_bounded_array_default ["Hellö wörld!","Hello world!",'Hellö wörld!']
wstring<=20[5] wstring_limit_var_array_default ["Hellö wörld!1","Hellö wörld!2",'Hellö wörld!3',"Hellö wörld!4",'Hellö wörld!5'] 
wstring<=20[] wstring_limit_variable_length_array_default ["Hellö wörld!1","Hellö wörld!2",'Hellö wörld!3']

stdout_stderr.log
Also, looking at the ros2 foxy codebase, I see that tests are missing for WstringArrays but present for StringArrays:

[sloretz]

I'm able to reproduce this using the most recent Foxy binaries

Foxy rosidl packages installed

ros-foxy-rosidl-adapter/focal,now 1.2.1-1focal.20210423.001432 amd64 [installed]
ros-foxy-rosidl-cmake/focal,now 1.2.1-1focal.20210423.001729 amd64 [installed]
ros-foxy-rosidl-default-generators/focal,now 1.0.1-1focal.20210630.223214 amd64 [installed,automatic]
ros-foxy-rosidl-default-runtime/focal,now 1.0.1-1focal.20210630.223215 amd64 [installed,automatic]
ros-foxy-rosidl-generator-c/focal,now 1.2.1-1focal.20210630.220922 amd64 [installed,automatic]
ros-foxy-rosidl-generator-cpp/focal,now 1.2.1-1focal.20210630.221040 amd64 [installed,automatic]
ros-foxy-rosidl-generator-dds-idl/focal,now 0.7.1-1focal.20210423.001903 amd64 [installed]
ros-foxy-rosidl-generator-py/focal,now 0.9.4-1focal.20210630.222457 amd64 [installed,automatic]
ros-foxy-rosidl-parser/focal,now 1.2.1-1focal.20210423.001559 amd64 [installed]
ros-foxy-rosidl-runtime-c-dbgsym/focal,now 1.2.1-1focal.20210630.220736 amd64 [installed]
ros-foxy-rosidl-runtime-c/focal,now 1.2.1-1focal.20210630.220736 amd64 [installed,automatic]
ros-foxy-rosidl-runtime-cpp/focal,now 1.2.1-1focal.20210423.001410 amd64 [installed]
ros-foxy-rosidl-runtime-py/focal,now 0.9.1-1focal.20210630.233034 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-c-dbgsym/focal,now 1.0.2-1focal.20210630.222356 amd64 [installed]
ros-foxy-rosidl-typesupport-c/focal,now 1.0.2-1focal.20210630.222356 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-connext-c-dbgsym/focal,now 1.0.3-1focal.20210630.222011 amd64 [installed]
ros-foxy-rosidl-typesupport-connext-c/focal,now 1.0.3-1focal.20210630.222011 amd64 [installed]
ros-foxy-rosidl-typesupport-connext-cpp-dbgsym/focal,now 1.0.3-1focal.20210630.221629 amd64 [installed]
ros-foxy-rosidl-typesupport-connext-cpp/focal,now 1.0.3-1focal.20210630.221629 amd64 [installed]
ros-foxy-rosidl-typesupport-cpp-dbgsym/focal,now 1.0.2-1focal.20210630.222647 amd64 [installed]
ros-foxy-rosidl-typesupport-cpp/focal,now 1.0.2-1focal.20210630.222647 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-fastrtps-c-dbgsym/focal,now 1.0.3-1focal.20210630.221752 amd64 [installed]
ros-foxy-rosidl-typesupport-fastrtps-c/focal,now 1.0.3-1focal.20210630.221752 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-fastrtps-cpp-dbgsym/focal,now 1.0.3-1focal.20210630.221557 amd64 [installed]
ros-foxy-rosidl-typesupport-fastrtps-cpp/focal,now 1.0.3-1focal.20210630.221557 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-interface/focal,now 1.2.1-1focal.20210423.001439 amd64 [installed]
ros-foxy-rosidl-typesupport-introspection-c-dbgsym/focal,now 1.2.1-1focal.20210630.220923 amd64 [installed]
ros-foxy-rosidl-typesupport-introspection-c/focal,now 1.2.1-1focal.20210630.220923 amd64 [installed,automatic]
ros-foxy-rosidl-typesupport-introspection-cpp-dbgsym/focal,now 1.2.1-1focal.20210630.221153 amd64 [installed]
ros-foxy-rosidl-typesupport-introspection-cpp/focal,now 1.2.1-1focal.20210630.221153 amd64 [installed,automatic]

Build succeeds

wstring test_default "ö"

wstring[] test_default ["H"]

string test_default "ö"

Fails to build with UnicodeDecodeError

wstring[] test_default ["ö"]

string[] test_default ["ö"]

[sloretz]

I'm also able to reproduce on master branches, so this is probably an issue in ROS Galactic and ROS Rolling too.

[sloretz]

Looks like an encoding issue in the generated idl file, the problem is probably somewhere in rosidl_adapter

// generated from rosidl_adapter/resource/msg.idl.em
// with input from wstr_msg/msg/WstrTest.msg
// generated code does not contain a copyright notice


module wstr_msg {
  module msg {
    struct WstrTest {
      @default (value="('�',)")
      sequence<string> test_default;
    };
  };
};

[karthiknit1]

Probably, the issue is coming due to latin-1 encoding which is merged as part of this commit:aae1e9a

This is not merged in dashing and also the build succeeds there.

[sloretz]

Probably, the issue is coming due to latin-1 encoding which is merged as part of this commit

I'm not sure about that. It seems to be the right encoding to output to: https://www.omg.org/spec/IDL/4.2/PDF Section 7.2

IDL uses the ASCII character set, except for string literals and character literals, which use the ISO Latin-1 (8859-1) 
character set.

[karthiknit1]

Is there any fix/workaround available for this issue?

[jacobperron]

See #620 for a fix.