Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option for embedded config and fallback resources #262

Merged

Conversation

Kuuuube
Copy link
Contributor

@Kuuuube Kuuuube commented Jun 13, 2024

Partial fix for #218

Allows static binaries to function properly in rust by falling back on default config and resources. Missing support for user defined embedded files but I think sudachi.rs should have at least basic support for static binaries.

Basic example code creating a tokenizer using embedded defaults (dictionary must still be added separately):

let embedded_dictionary = include_bytes!("./system_full.dic");
let sudachi_dic_data = SudachiDicData::new(Storage::Borrowed(embedded_dictionary));
let config = Config::new_embedded().unwrap();
let japanese_dictionary = JapaneseDictionary::from_cfg_storage_with_embedded_chardef(&config, sudachi_dic_data).unwrap();

let tokenizer = StatelessTokenizer::new(&japanese_dictionary);

@mh-northlander mh-northlander self-requested a review June 25, 2024 08:03
Copy link
Collaborator

@mh-northlander mh-northlander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally the embedded resource files should be used as a system default (replacing current default-resource-directory based configuration).
But as a first step we can have both.

I accept this PR with some miner changes.

sudachi/src/config.rs Outdated Show resolved Hide resolved
sudachi/src/dic/mod.rs Outdated Show resolved Hide resolved
sudachi/src/dic/mod.rs Show resolved Hide resolved
sudachi/src/dic/dictionary.rs Outdated Show resolved Hide resolved
@Kuuuube
Copy link
Contributor Author

Kuuuube commented Jun 26, 2024

Thank you for the review! Everything should be resolved.

@mh-northlander
Copy link
Collaborator

@Kuuuube please apply cargo fmt.

@Kuuuube
Copy link
Contributor Author

Kuuuube commented Jun 27, 2024

Done

@mh-northlander mh-northlander merged commit d78bf49 into WorksApplications:develop Jun 28, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants