Recommended data provider type for libraries depending on ICU4X

[rizzen-yazston]

I finished creating a library that uses ICU4X as its backend, while learning Rust. Successfully created toy examples using the library using icu_testdata, and I learnt quite a bit of Rust in the process. For my library I used the DataProvider for as the interface to CLDR data (currently just using icu_testdata, though seen the page to generate customised datasets). I did also at the beginning also played with the FsDataProvider, yet thought my library should be more generic in nature, thus used DataProvider instead.

Now busy creating a small application using my library, yet ran into a snag in using my library. main() can't have generics/traits specified.

So now I am wondering what would be the recommended data provider to use for a library using ICU4X as its backend? In order to avoid the issue of generics / trait types needing to be specified in the main() function.

[Manishearth]

If you know the data you want at build time, I suggest using a baked data provider, otherwise use a Blob one with postcard.

You can generate data using these steps

In the 1.3 release there will be a compiled_data feature that lets you include data by default, kinda like testdata but intended for production.

[rizzen-yazston]

Sadly I would not know what data provider the users of my library would be using at build time, thus trying to figure out best approach to support any type of provider that the users would use for my library.

compiled_data feature may just be what my library could use without the need for users to supply data provider for my library, if I understand the intended purpose of this up coming feature. Where is this feature located in the master, so I may start looking at it for design purposes, while waiting for 1.3 release?

[rizzen-yazston]

Firstly thank you for considering the question for your upcoming developers call.

So for simple components such as properties, segmenters, plurals and plurals to name a few, the built-in BakedDataProvider will provide the data if the user does not pass a provider. Also the data is broken down into individual component BakedDataProviders, thus only data will be included for components that is actually used, thus smaller data footprint in final build. Complex components, rather pass a suitable provider.

Using BufferProvider for intermediate libraries (such as mine) is a reasonable solution, in that BufferProvider can support both FsDataProvider and BlobDataProvider (the very versatile in memory provider), yet can use the functions internal built-in BakedDataProvider in the absence of user passing a BufferProvider object. The use of BufferProvider can allow for custom data in either FsDataProvider or BlobDataProvider that uses the private subtags of language tags (BCP 47). Just some thoughts swirling in my mind.

[rizzen-yazston]

I am working on an experimental branch to use BufferProvider and BakedDataProvider via compiled_data feature. I am making both optional, but at least 1 must be selected.

[rizzen-yazston]

What components are planed for 1.3 to support compiled_data?

What components already implement compiled_data on master, that are functional (not just a placeholder for implementation?

[Manishearth]

All components.

[rizzen-yazston]

I have updated my intermediate library (branch: provider) to use the ICU4X master branch, using the internal BakedDataProvider, BlobDataProvider and FsDataProvider. So far it seems to working on new application project using the library. I am still learning rust, thus library is not fully optimised. I do see ICU4X uses various techniques, that I still need to learn.

New branch message now available, with concurrency support and redesigned pattern provider.

I believe it may be possible to also support AnyProvider along with the existing BlobDataProvider, FsDataProvider and built-in compiled_data.