[BUG: Mistral does not work properly when comparing documents

[javixeneize]

Python -VV

Python 3.9.6 (default, Oct  4 2024, 08:01:31)

Pip Freeze

ollama==0.3.3

Reproduction Steps

I want mistral to find the differences between two policy files in JSON. It is, i want to find which policies have been added, which ones have been removed, and which ones have had any value change. I am using mistral through Ollama

Those are the two JSON files i have:

doc1raw = ["{'policy name': 'first rule', 'description': ' description of policy for first rule ', 'recommended value': '12'}",
           "{'policy name': 'second rule', 'description': ' description of policy for second rule', 'recommended value': ' yes '}" ]
doc2raw = ["{'policy name': 'first rule', 'description': ' description of policy 1 ', 'recommended value': '16'}",
           "{'policy name': 'third rule', 'description': ' description of policy for third rule', 'recommended value': ' test '}" ]

This is my system prompt:

sysprompt = [{'role': 'system', 'content': f"""'You are a version control assistant. Your role is to identify what has changed between two documents in a concise way.
                                            You are provided two JSON files, one for the old policy and another for the new one. 
                                            Each JSON contains a list of policies. Consider each policy in each document as a separate item.
                                            Each policy contains
                                            - Policy name
                                            - Description
                                            - Recommended value.
                                            You need to explain:
                                           1 - Policies removed: You need to identify items with a given policy name that exist in the first document but do not exist in the second one. Once you have done this, you need to do those two task as well:
                                            
                                           2 - Policies changed: It is, policies with the same title, but with a different description or recommended value
                                           3 - Policies added: Which policies have been added in the new document and are not present in the old one
                                           
                                           'Here is the old document: {doc1raw}'
                                           'And here is the new one {doc2raw}'"""}]

llm_prompt = sysprompt+[{'role': 'user', 'content': 'Find the differences between those two files'}]
stream = chat(
    model='mistral',
    messages=llm_prompt,
    stream=True
)

While it always returns correctly policies added and policies changed, it struggles to identify which policies have been removed. Sometimes it returns the correct value (second rule has been removed), but most of the times, it says no policy has been removed

Expected Behavior

My expectation is the LLM to be able to identify which policies have been removed from the first document, and aswer that accurately

Additional Context

No response

Suggested Solutions

No response