New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

文本token数量 #17

Open

ii35322 opened this issue Sep 23, 2023 · 2 comments

ii35322 commented Sep 23, 2023

您好，看文档写的文本数据量约1TB，想询问下面这样总共包含多少token？

Arcmoon-Hu commented Sep 26, 2023

同样想问，现在本地计算完大概只有十几B的tokens，离论文里的1.2万亿差太多了

JerryDaHeLian commented Sep 28, 2023

1个中文2到3个token，一个中文占两个字节，至少2T以上吧。1094.04GB 至少 1T 以上的token吧。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment