Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] corpus를 pre-processing하는 script를 개발한다. #3

Open
seopbo opened this issue Sep 12, 2020 · 1 comment
Open

[FEATURE] corpus를 pre-processing하는 script를 개발한다. #3

seopbo opened this issue Sep 12, 2020 · 1 comment
Assignees
@seopbo
Labels
enhancement New feature or request

Comments

@seopbo
Copy link
Contributor

seopbo commented Sep 12, 2020

🚀 Feature

BART의 pre-training을 위해 corpus를 pre-processing하는 script를 개발한다.

  • 사전에 corruption을 해서 integer 형태로 떨궈놓기
  • tokenizers package를 이용, byte pair encoding하기

Motivation

Pitch

Additional context
@seopbo seopbo added the enhancement New feature or request label Sep 12, 2020
@seopbo seopbo self-assigned this Sep 12, 2020
seopbo added a commit that referenced this issue Sep 12, 2020
@seopbo
test: commit test
7ce5962 
 Refs: #3
@seopbo
Copy link
Contributor Author

seopbo commented Sep 27, 2020

20/09/28

huggingface tokenizers 이용해서 tokenizer 작성

seopbo added a commit that referenced this issue Sep 28, 2020
@seopbo
feat: byte-pair bpe 학습코드 추가 및 학습
0ba6eb9 
 Refs: #3
seopbo added a commit that referenced this issue Sep 29, 2020
@seopbo
feat: dev.Dockerfile 추가
7266c6e 
 Refs: #3
seopbo added a commit that referenced this issue Sep 29, 2020
@seopbo
Merge branch 'feature/#3' of github.com:HephaestusProject/pytorch-KoB…
444a505 
…ART into feature/#3
seopbo added a commit that referenced this issue Sep 29, 2020
@seopbo
feat: requirements.txt 추가
368f918 
 Refs: #3
@seopbo seopbo mentioned this issue Sep 29, 2020
Merged
2 tasks
seopbo added a commit that referenced this issue Sep 30, 2020
@seopbo
Merge pull request #6 from HephaestusProject/feature/#3
554644b 
Feature/#3
chagmgang added a commit that referenced this issue Sep 30, 2020
@chagmgang
Merge pull request #7 from HephaestusProject/feature/#3
cf5670d 
Feature/#3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@seopbo seopbo

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@seopbo