Separate Evaluation Datasets

[svenschultze]

I am training models on a mixture of datasets from different tasks, which I combine using an external script that creates conversations based on these building blocks. To evaluate the models during training, I need to define a separate evaluation dataset that contains building blocks that are not in the training dataset.

My Question:
Is it possible to load a separate eval dataset instead of taking a fraction of the training dataset with axolotl?

[Taytay]

It looks like it is now thanks to this new commit:
5787e1a