[BUG]text 中含有"<"时 tokenizer 报错， #21
Labels
question
Further information is requested
Comments
|
输入里的 |
|
这样处理是否合理？很多代码相关的出现 < 频率很高，转义就改变含义了。还有一些html类的 有” <body> </a></li>“ 等，要写很多规则来判断是否转义 |
这些html无需判断直接转义即可，模型实际看到的不是 |
要是代码中出现的左移操作<<要替换成<<<<吗？ |
是的 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
运行下面的代码会报错，经过测试是因为含有"<"
"""
from cpm_live.models import CPMBeeTorch, CPMBeeConfig
from cpm_live.tokenizers import CPMBeeTokenizer
config = CPMBeeConfig.from_json_file("config/cpm-bee-10b.json")
tokenizer = CPMBeeTokenizer()
print(tokenizer._special_tokens)
text = "if 成绩 < 60"
tokens = tokenizer.tokenize(text)
"""
File "text_generation.py", line 28, in
tokens = tokenizer.tokenize(text)
File "/root/CPM-Bee/src/cpm_live/tokenizers/bee.py", line 143, in tokenize
raise ValueError("Unexpected end of text
{}".format(text))ValueError: Unexpected end of text
if 成绩 < 60The text was updated successfully, but these errors were encountered: