Error while doing multimodal PPO on llava-1.5-7b

[yxwang-10]

I got the following error when I was doing multimodal PPO on llava-1.5-7b.

[rank1]: Traceback (most recent call last):
[rank1]:   File "/mnt/cachenew/wangyixu/workspace/LLaMA-Factory/src/llamafactory/launcher.py", line 23, in <module>                                                    [rank1]:   launch()
[rank1]:   File "/mnt/cachenew/wangyixu/workspace/LLaMA-Factory/src/llamafactory/launcher.py", line 19, in launch
[rank1]:     run_exp()
[rank1]:   File "/mnt/cachenew/wangyixu/workspace/LLaMA-Factory/src/llamafactory/train/tuner.py", line 92, in run_exp
[rank1]:     _training_function(config={"args": args, "callbacks": callbacks})
[rank1]:   File "/mnt/cachenew/wangyixu/workspace/LLaMA-Factory/src/llamafactory/train/tuner.py", line 70, in _training_function
[rank1]:     run_ppo(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank1]:   File "/mnt/cachenew/wangyixu/workspace/LLaMA-Factory/src/llamafactory/train/ppo/workflow.py", line 72, in run_ppo                                               
[rank1]:     ppo_trainer.ppo_train(resume_from_checkpoint=training_args.resume_from_checkpoint)                                                                            
[rank1]:   File "/mnt/cachenew/wangyixu/workspace/LLaMA-Factory/src/llamafactory/train/ppo/trainer.py", line 242, in ppo_train                                             
[rank1]:     batch = next(dataiter)                                                                                                                                        
[rank1]:             ^^^^^^^^^^^^^^                                                                                                                                        
[rank1]:   File "/mnt/petrelfs/wangyixu/anaconda3/envs/LLaMA-Factory/lib/python3.11/site-packages/accelerate/data_loader.py", line 550, in __iter__                        
[rank1]:     current_batch = next(dataloader_iter)                                                                                                                         
[rank1]:                     ^^^^^^^^^^^^^^^^^^^^^                                                                                                                         
[rank1]:   File "/mnt/petrelfs/wangyixu/anaconda3/envs/LLaMA-Factory/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 701, in __next__                   
[rank1]:     data = self._next_data()                                                                                                                                      
[rank1]:            ^^^^^^^^^^^^^^^^^                                                                                                                                      
[rank1]:   File "/mnt/petrelfs/wangyixu/anaconda3/envs/LLaMA-Factory/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 757, in _next_data                 
[rank1]:     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration                                                                                          
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                     
[rank1]:   File "/mnt/petrelfs/wangyixu/anaconda3/envs/LLaMA-Factory/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 55, in fetch                     
[rank1]:     return self.collate_fn(data)                                                                                                                                  
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                  
[rank1]:   File "/mnt/cachenew/wangyixu/workspace/LLaMA-Factory/src/llamafactory/data/collator.py", line 120, in __call__                                                  
[rank1]:     features[0]["labels"] = [IGNORE_INDEX] * len(fake_input_ids) + features[0]["labels"]                                                                          
[rank1]:                                                                    ~~~~~~~~~~~^^^^^^^^^^                                                                          
[rank1]: KeyError: 'labels' 

It's very strange. I checked that the dataset obtained in
dataset_module = get_dataset(template, model_args, data_args, training_args, stage="ppo", **tokenizer_module)
contains labels normally, but the features printed at the beginning of MultiModalDataCollatorForSeq2Seq have no labels.

[hiyouga]

The multimodal PPO might has some bugs now, you can use DPO or KTO instead

[yxwang-10]

Thanks for your reply. I checked the code further and found that this is due to the parameter remove_unused_columns in PPOConfig, which is set to True by default, thus removing those columns.