Where is this logged eval_causal_lm_metrics: ["perplexity"]?

[nazkhan-8451]

In my config to finetune llama-3.1, I put eval_causal_lm_metrics: ["perplexity"]. In trainer_state.json here is output at eval step:

    "epoch": 0.5002141327623126,
      "eval_loss": 0.8674565553665161,
      "eval_runtime": 562.8213,
      "eval_samples_per_second": 1.747,
      "eval_steps_per_second": 0.437,
      "step": 584

Where is perplexity logged? I know I can calculate it from cross-entropy loss but wanted to know this.

[ckgresla]

I believe that setting this value in your config leads to a perplexity being logged over the training set -- here are a few reference wandb job screenshots from a run with this metric enabled:

Additionally, the perplexity here seems oddly high, assuming that the perplexity is calculated as the exponential of the average cross entropy (loss value @ timestep) then we would have:

>>> torch.exp(torch.tensor(1.7759))
tensor(5.9056)  # expected perplexity for loss @ timestep

but at ~roughly the same timestep, the training perplexity is logged as ~1207 – the implementation of the metric in perplexity.py seems to be correct, would folks have an intuition as to why this value is higher than the expected perplexity?

I am new to this project and was wondering why the eval_causal_lm_metrics get logged only for the training set, the corresponding eval section of the wandb run looks like this:

Is that the expected behavior, or perhaps is something awry in my setup of the repo?

Ideally, we could track these casual lm metrics for specified subsets of the data with something like the following:

train_causal_lm_metrics: ["perplexity", "accuracy"]
eval_causal_lm_metrics: ["perplexity", "accuracy"]

@winglian – would you be open to this becoming a part of the axolotl?

[NanoCode012]

Hey @ckgresla, thanks for looking into this. I've been digging into this as well. It turns out that it is indeed from eval. It's just that wandb sees it as train split.

If you check this section below, it's within the CausalLMBenchEvalCallback during on_evaluate callback which compute using data from eval_dataloader.

axolotl/src/axolotl/utils/callbacks/__init__.py

Lines 468 to 471 in e1915f5

	def predict_with_generate():
	eval_src, eval_pred, eval_ref = [], [], []
	for batch in tqdm(eval_dataloader):

We used this to log which had that side effect.

axolotl/src/axolotl/utils/callbacks/__init__.py

Line 553 in e1915f5

trainer.log(evaluate_preds(*eval_preds))

This PR should address it #1952

[NanoCode012]

Hey @nazkhan-8451 , the perplexity should show up in there with the changes in #1952 . Don't forget to turn on do_causal_lm_eval: true to ensure the eval runs.