sequence_len

[thistleknot]

how does the framework handle prompts (e.g. completion / raw format) larger than sequence_len?

[thistleknot]

nm, looks like it does break up (sequence_len: 1024)
Downloading data files: 100%|...
Generating train split: 128293 examples [00:03, 40725.21 examples/s]
Map (num_proc=12): ...128293/128293 [03:49<00:00, 557.82 examples/s]
...
Saving the dataset (8/8 shards):...325114/325114 [00:01<00:00, 233573.96 examples/s]

sequence_len 32678
Map (num_proc=12): 100%|... 128293/128293 [03:46<00:00, 566.37 examples/s]
[2023-10-29 10:47:03,744] [INFO] [axolotl.load_tokenized_prepared_datasets:354] [PID:469034] [RANK:0] merging datasets
[2023-10-29 10:47:03,749] [INFO] [axolotl.load_tokenized_prepared_datasets:361] [PID:469034] [RANK:0] Saving merged prepared dataset to disk... last_run_prepared/e15bf2105a07282d7bd0640ad6328059
Saving the dataset (8/8 shards): 100%|... 128337/128337 [00:01<00:00, 84667.09 examples/s]