Question about string encoding (UTF-16/UTF-8)

[tmpfs]

Dart strings are UTF-16 and Rust strings are UTF-8.

I just hit an error on iOS where a particular value caused a panic in a dependency that was calling std::str::from_utf8().unwrap().

The value is dish’s - notice the backtick-style apostrophe.

My question is, should FRB be handling conversion between the string encodings or is there something I need to do on the Rust side to convert these strings to UTF-8 correctly?

/cc @fzyzcjy

[fzyzcjy]

Good observation! Not sure whether we should use sth like: https://api.dart.dev/stable/2.18.6/dart-convert/dart-convert-library.html (or something similar on Rust side).

It is a question whether to do it inside frb or outside frb (i.e. let the users do it), since the conversions can take overhead

[tmpfs]

Thanks, @fzyzcjy for the reply. We can keep discussing this but I do think this particular problem is with the dependency I am using.

I am using a library for generating n-grams to use in a search index and I think it is splitting the string naively to generate the n-grams.

Let me do some more research.

[tmpfs]

@fzyzcjy, I fixed the problem by rolling my own n-gram implementation which respects unicode graphemes:

Lambda-Logan/creature_feature#5

However, I am still very curious how FRB handles the differences in string encoding between Dart and Rust. How are strings handled when they go across the bridge? Is it the case that there may be some bugs lurking if we are not handling conversion between encodings?

[fzyzcjy]

Good job!

However, I am still very curious how FRB handles the differences in string encoding between Dart and Rust. How are strings handled when they go across the bridge? Is it the case that there may be some bugs lurking if we are not handling conversion between encodings?

I guess so. IIRC we have only used utf8 encoding? Not remember the exact details, you can see the source code

[tmpfs]

Looks good to me, you already call out to the dart utf8 encoder:

  @protected
  ffi.Pointer<wire_uint_8_list> api2wire_String(String raw) {
    return api2wire_uint_8_list(utf8.encoder.convert(raw));
  }

Thanks 👍

[fzyzcjy]

You are welcome!