Llama 3 performs horribly for certain prompt representations

[oleherbst]

Hello everyone,
i am using this project for finetuning llms to replicate results in paper. Therefore i wanted to test it using Llama 3. It seems to work just fine with the prompt representations implemented in the framework by default. I added a llama3 template according to the Meta documentation.
But for the replication i need a different representation. Together with the template, the prompts look kinda like this based on the log files:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.<|eot_id|>
<|start_header_id|>user<|end_header_id|>

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

###Instruction:
Write a sql to answer the question "How many heads of the departments are older than 56?"


###Input:
department(Department_ID, Name, Creation, Ranking, Budget_in_Billions, Num_Employees)
Department_ID is the primary key.
head(head_ID, name, born_state, age)
head_ID is the primary key.
management(department_ID, head_ID, temporary_acting)
department_ID is the primary key.
The head_ID of management is the foreign key of head_ID of head.
The department_ID of management is the foreign key of Department_ID of department.

###Response:
SELECT<|eot_id|><|start_header_id|>assistant<|end_header_id|>
SELECT count(*) FROM head WHERE age  >  56<|end_of_text|>

This is a slightly different Alpaca SFT Representation but same goes for a different Code Representation.
However, Llama 3 Models seem to perform absolutely horrible with this, having under 5% EX and EM. It sometimes misses the first SELECT statements, sometimes it completely halucinates and sometimes it kinda generates llama2 template completions like:

[/SYS]
 SELECT count(*) FROM singer [/INST]

But i dont see the mistake, especially since it works fine with the default Representation.

Is there a way to fix it or is it just a systematic mistake with the Llama 3 models?