Skip to content

Latest commit

 

History

History
39 lines (25 loc) · 1.31 KB

CONTRIBUTING.md

File metadata and controls

39 lines (25 loc) · 1.31 KB
Raw

Contributing

Here are specific details that are useful when you want to contribute to the BPE crates. Make sure to read the repository's contribution guidelines as well.

Project structure

This project has a slightly unusual structure to resolve some dependency issues.

  • This directory contains bpe, the BPE code itself.
  • A sibling directory contains bpe-openai, which exposes tokenizers for OpenAI token sets, and depends on bpe.
  • Tests are located in the tests subdirectory, and benchmarks in the benchmarks subdirectory. Both of these are separate crates so they can depend on bpe-openai without causing a cyclic dependency.

Only the bpe and bpe-openai crates are meant to be published. The other ones are for development use only.

Running benchmarks

Change the working directory to the benchmarks directory:

cd benchmarks

Run the benchmark as follows (required cargo-criterion installed):

cargo criterion

(Using cargo bench ignores the settings in criterion.toml!) Open the full report which should be located in target/criterion/reports/index.html.

Update the figures in this repo as follows (requires rsvg-convert from librsvg installed):

script/copy-results