Difference in Question And Answer prediction score while using HuggingFaceTransformer pipeline and FARMReader pipeline

[karndeepsingh]

Hello,
I have trained a Question Answering Model using FARMReader and when I do the inferencing using Pipeline provided in Haystack, I am getting good score. Following are the results.

After, converting the trained model into hugging face transformers model file and when loading the model with hugging face transfomer question answering pipeline, I am not getting the same score for the same input queries. Following are the results:

It is quiet visible that the difference in score as well answer is huge.
Is that some weights are being loss while converting it to hugging face transformer model?
Please help me, I want to deploy the model on server as Haystack doesn't support any batch mode to supply list of queries to the model once hence I have to do by using hugging face pipeline but it seems my current trained is loosing the information and unable to predict the same way it has been doing with FARMReader.

Please help me to resolve this issue. Thanks.

[bogdankostic]

Hi @karndeepsingh, it seems that you are using different questions in haystack and in transformers, which probably is the reason for the high discrepancy that you are observing.
In haystack, you use the question: what is the grantor's first party name id code or common id code or trust name?.
In transformers, you use the question: what is the {ans}'s first party name id code or common id code or trust name?.

[karndeepsingh]

Hi @karndeepsingh, it seems that you are using different questions in haystack and in transformers, which probably is the reason for the high discrepancy that you are observing. In haystack, you use the question: what is the grantor's first party name id code or common id code or trust name?. In transformers, you use the question: what is the {ans}'s first party name id code or common id code or trust name?.

Hi @bogdankostic,
Following image shows the result with correct question:

[bogdankostic]

Transformers handles the answer span prediction and scoring a bit different than we do in haystack, for more information please have a look at the issues #248 where we explain the differences in more detail.

[karndeepsingh]

@bogdankostic if that is challenge then how I should proceed for deployment of my question answering system as FARMReader doesn’t take up list of queries in one shot to answer from the context and for this reason I switched to using huggingface transformer model because it supports passing list of queries in one shot but it seems it also has problem with respective to accuracy. I am stuck on deciding which way I should proceed. It is becoming challenging. Please help. I want to pass list of question in one shot to the context, how that can be achieved with same accuracy?

[bogdankostic]

Are you just using a Reader or does your Pipeline contain multiple nodes? If you're just using a Reader, the predict_batch method might be a solution for you.

[karndeepsingh]

Are you just using a Reader or does your Pipeline contain multiple nodes? If you're just using a Reader, the predict_batch method might be a solution for you.

@bogdankostic I am using Pipeline. Can you please share the data format required to be passed in predict_batch function as I want to query on multiple question at once? I would like to try it out.

thanks.

[bogdankostic]

query_doc_list = [
	{
		"question": <QUESTION_1>,
	 	"docs": <LIST_OF_DOCUMENTS>
	},
	{
		"question": <QUESTION_2>,
	 	"docs": <LIST_OF_DOCUMENTS>
	},
	...
]

[karndeepsingh]

query_doc_list = [
	{
		"question": <QUESTION_1>,
	 	"docs": <LIST_OF_DOCUMENTS>
	},
	{
		"question": <QUESTION_2>,
	 	"docs": <LIST_OF_DOCUMENTS>
	},
	...
]

Here, suppose I have to ask two questions from same context so, I have to pass question_1 and question_2 with same context in the dictionary mentioned. Correct me if my understanding is wrong.

thanks

[bogdankostic]

Yes, that's true.

[karndeepsingh]

Yes, that's true.

If I convert my trained model to onxx model will I be able to do inferencing in batch ?

also, i would like your suggestion on deploying question answering model in production. What are the things I need to worry on deploying QnA system using Haystack like is there any any memory issue etc. can you please guide me better for deployment.

thanks

[tstadel]

@karndeepsingh we're currently working on simplifying deployment. Checkout #1952