VGP models don't work with keras optimizers in TF 2.11 and 2.12

[uri-granta]

Describe the bug
Calling minimize on a VGP model using a keras optimizer no longer works when using TF 2.12 or 2.12.

Additional context
This is due keras-team/tf-keras#161 and is most easily fixed on the tenorflow side. We should therefore decide whether we're happy to wait or if we want a workaround for the affected versions.