Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rework handling of special tokens #45

Merged
merged 12 commits into from
Sep 20, 2024
Merged

Rework handling of special tokens #45

merged 12 commits into from
Sep 20, 2024

Commits on Jun 28, 2024

  1. allow configuration of bos/eos/unk/pad, optional_eos for decoding

    @francoishernandez
    francoishernandez committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    2d096cd View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. fix conflicts

    @francoishernandez
    francoishernandez committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    bb9fd27 View commit details
    Browse the repository at this point in the history

  2. reorder special tokens definitions for clarity

    @francoishernandez
    francoishernandez committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    a5e15b3 View commit details
    Browse the repository at this point in the history

  3. deprecate default_specials flag, update convert_HF to grab actual spe… 

    …cial tokens from config
    @francoishernandez
    francoishernandez committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    110a4e2 View commit details
    Browse the repository at this point in the history

  4. update special tokens FAQ entry, remove default_specials in comet recipe

    @francoishernandez
    francoishernandez committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    825fe5c View commit details
    Browse the repository at this point in the history

  5. use specials from vocabs/config instead of DefaultTokens

    @francoishernandez
    francoishernandez committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    ea14921 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. fix conflicts

    @francoishernandez
    francoishernandez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    f5378cc View commit details
    Browse the repository at this point in the history

  2. get rid of unnecessary decoder_start_table

    @francoishernandez
    francoishernandez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    0c0f60b View commit details
    Browse the repository at this point in the history

  3. fix conflicts

    @francoishernandez
    francoishernandez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    0930041 View commit details
    Browse the repository at this point in the history

  4. simplify gpt2 inference.yaml

    @francoishernandez
    francoishernandez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    1a21bb8 View commit details
    Browse the repository at this point in the history

  5. some fix for better pad_token handling, still not perfect

    @francoishernandez
    francoishernandez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    00eed8b View commit details
    Browse the repository at this point in the history

  6. add some basic run.sh to facilitate testing recipes, fix some configs

    @francoishernandez
    francoishernandez committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    955a27f View commit details
    Browse the repository at this point in the history