Cluster the document based on their topics and distribute the code task evenly by its clustered topic

[YanLiang1102]

No description provided.

[YanLiang1102]

step 1: I am using gensim lda model to train the data, then for each document, say we set the topic numbers to be 20, then after the gensim lda model training, for each document , we get a 20 dimensional vector, and each element in that vector is the probability that this document belongs to the topics.

Then I am using this vector for each document as a feature vector to feed in the sklearn kmeans mode to train the data, and the clustered index will be store in a dictionary indexed by the document id.

step2:
need to store this info in our mongo db, and come up algorithms to evenly distributed the docs in that way

something to think about : to make user choose what the topic they want to code with from the interface.
@ahalterman @cegme

[YanLiang1102]

training the mode now , it is how heavy it is using the resources. :)
@cegme @ahalterman

[YanLiang1102]

set the default topic to be 10000 for each documents.... from mongo command line
db.your_collection.update({},{$set : {"new_field":1}},false,true)
**********steps to migrate a db collection from one server to another
mongodump --db eventData --collection documents_arabic --port ... -out ...
scp -r ./eventData yan@server:/home/yan/dump/lexisnexis/eventData
sudo mongorestore -d dbname -c collectioname --port ... -u username -p password

[YanLiang1102]

The english documents training using like 3 hours to be trained, need to think about play with the gensim parallel part , just for fun, to see how fast it can be enhanced.