You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In Replicate, the Llama-3-8b-instruct model takes in a "system_prompt" and "prompt_template" param. The replicate docs mentioned that system_prompt is prepended to the text being sent, and I'm assuming the prompt_template is also used to format the prompt data since there's a default value that includes the special tokens. Is there a reason this application is formatting the prompt to also use these special tokens? I guess I'm just a little confused how this is being handled by Replicate behind the scenes.
For example, for the message "Can you tell me about the story of David and Goliath" the front-end sends:
{
system_prompt: "You are a helpful assistant.",
prompt: "<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
Can you tell me about the story of David and Goliath<|eot_id|>"
}
So, behind the scenes, it seems like what gets passed to the model becomes:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
Can you tell me about the story of David and Goliath<|eot_id|><|eot_id|><|start_header_id|>assistant<|end_header_id|>
since the prompt and system_prompt are possibly directly passed into the prompt template. Is this a correct understanding?
The text was updated successfully, but these errors were encountered:
hershalb
changed the title
Question: Why are the messages being pre-formatted before being sent to replicate?
Question: Why are special tokens added to the prompt being sent to replicate?
Jun 14, 2024
In Replicate, the Llama-3-8b-instruct model takes in a "system_prompt" and "prompt_template" param. The replicate docs mentioned that system_prompt is prepended to the text being sent, and I'm assuming the prompt_template is also used to format the prompt data since there's a default value that includes the special tokens. Is there a reason this application is formatting the prompt to also use these special tokens? I guess I'm just a little confused how this is being handled by Replicate behind the scenes.
For example, for the message "Can you tell me about the story of David and Goliath" the front-end sends:
The default "prompt_template" seems to be:
So, behind the scenes, it seems like what gets passed to the model becomes:
since the prompt and system_prompt are possibly directly passed into the prompt template. Is this a correct understanding?
The text was updated successfully, but these errors were encountered: