Data preprocessing code. #1
Conversation
There was a problem hiding this comment.
We should add a .gitignore entry for this
Edit: Oh there already is one let's stop tracking it then
There was a problem hiding this comment.
lol. I noticed after the first push. It is added on .gitignore.
There was a problem hiding this comment.
This is a rename of question.json right?
There was a problem hiding this comment.
Yeah, it is an artifact. No longer needed. I should have removed both question.json and naep_meta.json. Just need to run preprocess_data.py
There was a problem hiding this comment.
I'll review the change. I push some more code yesterday night. We need to keep question.json since I added more attributes for each question which can help me merge some variables. I will write a document to clarify it.
There was a problem hiding this comment.
@kikumaru818 Oh, i see the context_var. I'll apply that to the code.
There was a problem hiding this comment.
Not only "context_var", I have more. Maybe modify it after we meet.
I think keeping the question.json is ok. Definitely we if had a code version that can produce question.json is also great.
There was a problem hiding this comment.
I don't think having question.json by typing through our hands is a good idea. I made a function add_additional_info that takes care of additional infos you want to add to the json file. After discussing with you, I'll make the version that creates the json format you want. FYI, i just renamed it into meta_info.json and placed under the data directory.
There was a problem hiding this comment.
Yes, I think we should have a code version to generate it. I just mean it's ok to keep question.json file. Don't need to generate it every time.
There was a problem hiding this comment.
Oh, yeah. I'll add the code that loads the json file if it exists.
|
LGTM! |
@kikumaru818 @wmcnicho @wmcnicho @alexscarlatos
Hi everyone, made a preprocessing code on top of mengxue's implementation.
Just let me know if something is wrong.
Thanks.
mkdir data/ and put the all_items_train.txt in it.python3 preprocess_data.py