How to predict after loading in trained model in local?

[asharm0662]

[Novice]
Hi,

I was successfully able to train model on my data , on Google Colab initially, and save a model to my Google drive and download that model to my local machine using Haystack tutorial here. I saved using this line:

reader.save(directory='path/to/directory')

Now that I have a saved/trained model, I am wanting to take the trained model and apply it locally on the same docs that were used for training in google colab, to the same docs on my local machine.

I can read in the pretrained model in my local like so:

from haystack.reader.farm import FARMReader
new_reader = FARMReader(model_name_or_path="Downloads/my_model/.",use_gpu=False)

Now that I can read the model, I tried rerunning the prediction code like so:

prediction = pipe.run(query= "is this working? " ,params={"retriever": {"top_k": 10}, "reader": {"top_k": 5}})

However, this wont work and doesnt, because I dont have code on my local machine to run the pipe.run logic. My understanding was I wouldnt have to run the same training code again for predicting after training is done.

My question is, once I have trained a model on a set of documents in google colab, is there any easy way to pass the model to someone else and have them predicting without running all the code that I used for training, which includes the pipe.run code? Or what is the easiest way to start predicting using a pretrained model?

If I need to process documents again, in my training code, can I save the text documents that are converted to paragraphs and use those for running the model on at a later date? It seems counter intuitive to have to reprocess documents as done in training.

Thank you in advance.

---

Local Machine Specs:

1. Macbook pro M1 chip 2020

[julian-risch]

Hi @asharm0662 preprocessing of the documents is not needed again if you store the preprocessed documents in one of Haystack's document stores and re-use that document store. However, the document stores don't have something like an export method so we don't really support out of the box that you share a document store with somebody else using some other machine.

You're right that you don't need to run the training code again to make predictions after having trained a model. However, pipe.run needs to be executed to make predictions, for example, on an ExtractiveQAPipeline. I am bit confused about how you "trained" a model. Tutorial 1 does not do any training but you linked it in your issue. Maybe there is a misunderstanding? Could you please tell me what lines of code you used for training?

[asharm0662]

@julian-risch thanks for the reply.

To your question, I think I miss wrote. I used the existing model to train over my own documents. I saved the model that was trained to local on google drive. Does that make sense?

Is there a way to incremental train and save models as new documents come in versus having to retrain say everyday on the entire document store?

Thanks in advance!

[bogdankostic]

Hi @asharm0662! Please let me try to help you. I think there is a misunderstanding, given that the tutorial you have linked does not contain any training but only the application of already trained QA models (= making predictions). You do not need to train a model in order to make predictions.

You should be able to load one of the existing Reader models (for example "deepset/roberta-base-squad2") on your local machine just like in the tutorial and ask questions on your documents by following the steps in Tutorial 1 or Tutorial 3. In order to make predictions on your own documents instead of the Game-of-Thrones sample documents, just write your own documents to the document store.

I hope this makes things a bit clearer, if not, I am happy to answer your questions.