Skip to content

Commit

Permalink
Make use of prefilled cache in beam_search when `predict_batch_with_a…
Browse files Browse the repository at this point in the history 
…ux.prompt_with_targets` is set to True

Without this change, the targets that are actually prompts will still be decoded AR style in order to build the cache, the logits are thrown away.

With this change, and setting `EncoderDecoder.predict_batch_with_aux.prompt_with_target` to `True` we instead skip over the given prompts all together, speeding up the decode.

PiperOrigin-RevId: 571184873
  • Loading branch information
@chenli88 @t5-copybara
chenli88 authored and t5-copybara committed Oct 6, 2023
1 parent 34f1e74 commit 3de4e0e
Show file tree
Hide file tree
Showing 2 changed files with 39 additions and 403 deletions.
Loading

0 comments on commit 3de4e0e

Please sign in to comment.