tokenizer: the new wi17k_base tokenizer using the wikitext

Hk669 · Jun 4, 2024 · 4f713b6 · 4f713b6
1 parent a8a5d66
commit 4f713b6
Show file tree

Hide file tree

Showing 2 changed files with 34,387 additions and 1 deletion.
diff --git a/.gitignore b/.gitignore
@@ -153,4 +153,8 @@ logs
 .DS_Store
 
 output/
-*.pkl
+*.pkl
+
+# Ignore training files
+wk100k_base.py
+datasets/