Calculate perplexity for large language models

[VeritasJoker]

Use this script: https://github.com/hassonlab/247-pickling/blob/dev/scripts/tfsemb_perplexity.py

and this command:
perp-embeddings:
mkdir -p logs
for conv_id in $(CONV_IDS); do
python scripts/tfsemb_perplexity.py
--project-id $(PRJCT_ID)
--pkl-identifier $(PKL_IDENTIFIER)
--subject $(SID)
--conversation-id $$conv_id
--embedding-type $(EMB_TYPE);
done;

It should take less than a minute to run for each of those if we have the model downloaded

With stride 512, 1024, 2048, 4096:

• gpt2-xl
• EleutherAI/gpt-neox-20b
• facebook/opt-6.7b
• facebook/opt-13b
• facebook/opt-30b
• facebook/opt-66b
• meta-llama/Llama-2-7b-hf
• meta-llama/Llama-2-13b-hf
• meta-llama/Llama-2-70b-hf

Actually can you just do the four strides for all of the models. I'm making a table for all the values here:
https://docs.google.com/spreadsheets/d/1E3k9gCvqsWERyPmvXvo-0yfYIyFt5XJQhYNS2ykKooo/edit?usp=sharing

[hvgazula]

where did you find this script?

[VeritasJoker]

HuggingFace lol

Here: https://huggingface.co/docs/transformers/en/perplexity

[hvgazula]

Does it matter if the models are quantized or not?