Skip to content

Commit

Permalink
Fix shape error detected by benchmark
Browse files Browse the repository at this point in the history
Signed-off-by: Chendi.Xue <[email protected]>
  • Loading branch information
xuechendi committed Dec 14, 2024
1 parent 717c3a5 commit d812acf
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion vllm/sequence.py
Original file line number Diff line number Diff line change
Expand Up @@ -1182,7 +1182,8 @@ def update(self,
second_last_token_hidden_states: Optional[torch.Tensor] = None):
"""Update hidden states from target model invocation. Only used for
decode steps"""
assert len(seq_group_metadata_list) == len(hidden_states)
if len(seq_group_metadata_list) < len(hidden_states):
hidden_states = hidden_states[:len(seq_group_metadata_list)]
self._seq_ids.extend(get_all_seq_ids(seq_group_metadata_list))
self.hidden_states = torch.cat([self.hidden_states, hidden_states])

Expand Down

0 comments on commit d812acf

Please sign in to comment.