-
Notifications
You must be signed in to change notification settings - Fork 149
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix vision model graph capture not creating static buffers for embedd…
…ing (#942) This change essentially reverses the assignment of the embeddings memory. Instead of creating the embeddings tensor in the embedding model and pointing the embeddings of the text model to it, we now create the embeddings tensor inside the text model and point the embeddings of the embedding model to it. The reason to do that is that the text model can possibly be in "graph capture mode", which means that it allocates static buffers that it uses between iterations, and even between generators. If we allocate the memory in the embedding model and point the text model to it, the memory will become invalid when the generator is destroyed and the captured graph will exhibit undefined behavior (mostly spitting out garbage output). But by pointing the embeddings output of the embedding model towards the static buffer created by the text model, we can be certain that the memory will stay alive for the duration of the model. This PR doesn't change the behavior of the non-graph capture mode since it really doesn't matter in that scenario whether the tensor is created by the embedding model or the text model, but it fixes graph capture usage for vision models.
- Loading branch information
1 parent
c8e931d
commit 77a88c3
Showing
3 changed files
with
31 additions
and
30 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters