Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to convert .txt file to .h5 file #74

Open
XiaoXiaoYi123 opened this issue Mar 8, 2022 · 4 comments
Open

How to convert .txt file to .h5 file #74

XiaoXiaoYi123 opened this issue Mar 8, 2022 · 4 comments

Comments

@XiaoXiaoYi123
Copy link

XiaoXiaoYi123 commented Mar 8, 2022 •
edited
Loading

ese /indices and /phrases?
@li-car-fei
Copy link

请问可以提供将txt文本序列转为对应的.h5文件的实现代码吗？

@guxd
Copy link
Owner

guxd commented May 13, 2022

@li-car-fei
https://github.com/guxd/DialogBERT/blob/master/prepare_data.py
参考这个，用pytable 把dialog (a list of sequences)转成earray.

@li-car-fei
Copy link

@li-car-fei https://github.com/guxd/DialogBERT/blob/master/prepare_data.py 参考这个，用pytable 把dialog (a list of sequences)转成earray.

在deepcs中，你是使用了哪里的tokenizer呢？

@guxd
Copy link
Owner

guxd commented May 14, 2022

deepcs没有用第三方tokenizer,是自己实现的简易的tokenizer. 直接把字符转换成字典里的序号

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@guxd @XiaoXiaoYi123 @li-car-fei