Skip to content

Commit

Permalink
Make use of prefilled cache in beam_search when `predict_batch_with_a…
Browse files Browse the repository at this point in the history 
…ux.prompt_with_targets` is set to True

Without this change, the targets that are actually prompts will still be decoded AR style in order to build the cache, the logits are thrown away.

With this change, and setting `EncoderDecoder.predict_batch_with_aux.prompt_with_target` to `True` we instead skip over the given prompts all together, speeding up the decode.

PiperOrigin-RevId: 559256044
  • Loading branch information
@afrozenator @t5-copybara
afrozenator authored and t5-copybara committed Oct 4, 2023
1 parent 5107638 commit de7577e
Show file tree
Hide file tree
Showing 2 changed files with 403 additions and 39 deletions.
Loading

0 comments on commit de7577e

Please sign in to comment.