Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Fused CEL #457

Draft
wants to merge 48 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
4b6dff8
initial commit
jeromeku May 9, 2024
a6d8300
add trainer utils
jeromeku May 9, 2024
022c3d3
temp
jeromeku May 9, 2024
5b438be
rebase
jeromeku May 9, 2024
2af523d
add utils
jeromeku May 9, 2024
1116c7c
edit cel
jeromeku May 10, 2024
b2e5377
add reshaping to fused cel
jeromeku May 10, 2024
e6d449e
add logits warning
jeromeku May 10, 2024
6c3c90d
add shifting
jeromeku May 10, 2024
ac67948
fused out test
jeromeku May 10, 2024
6312670
fix loss dtype
jeromeku May 10, 2024
9ba1f81
add cel test
jeromeku May 10, 2024
382a8ec
refactor callback
jeromeku May 10, 2024
71889ea
fix grad reshape issue
jeromeku May 10, 2024
4088670
add benchmarking script
jeromeku May 10, 2024
e2b74c1
clean up bench script
jeromeku May 11, 2024
3304895
fix sanity check
jeromeku May 11, 2024
a25b9be
fix gradient scaling
jeromeku May 11, 2024
a54425a
change label shifting
jeromeku May 11, 2024
6c537e9
change n_loop_iters impl
jeromeku May 11, 2024
c777c50
add lm head test
jeromeku May 11, 2024
e1e55ce
add gitignore
jeromeku May 11, 2024
7ec969e
remove pycache
jeromeku May 11, 2024
b83c8e8
fix test
jeromeku May 12, 2024
e39ace8
fix test
jeromeku May 12, 2024
5a0a703
add more test configs
jeromeku May 12, 2024
291e3e2
add fixture
jeromeku May 12, 2024
46d0e7a
add benchmark
jeromeku May 12, 2024
adaebbf
add benchmark scripts
jeromeku May 12, 2024
91511c5
add pytest config
jeromeku May 12, 2024
e1a77c9
adjust testing tolerances
jeromeku May 12, 2024
cd22ece
initial unsloth bench
jeromeku May 12, 2024
842ba0d
add benchmark hf only script
jeromeku May 12, 2024
37e6e92
add debugging
jeromeku May 12, 2024
736786e
add notes to test
jeromeku May 12, 2024
a21f92a
fix fused config
jeromeku May 12, 2024
a204950
refactor test utils
jeromeku May 12, 2024
2da14c4
add simplified hf benchmark
jeromeku May 12, 2024
ec7e5dd
clean up hf test bench
jeromeku May 12, 2024
bca8815
add num trainable parameters
jeromeku May 13, 2024
c053af5
add readme
jeromeku May 13, 2024
1a8d836
update readme
jeromeku May 13, 2024
10a0b57
remove notebooks
jeromeku May 13, 2024
fae9e81
reorganize tests
jeromeku May 13, 2024
6fa0846
add cel test utils
jeromeku May 13, 2024
f047019
remove pycache
jeromeku May 13, 2024
fd92b6b
revert llama
jeromeku May 13, 2024
f4cd689
remove formatting
jeromeku May 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
Loading