Add MAX_INPUT_TOKENS to tgi

New tgi 2.0.0 version will set MAX_INPUT_TOKENS to 4095 by default. This will cause tgi error. First set to 1024 as default, may make it configurable by user later. Signed-off-by: Dolpher Du <[email protected]>
opea-project · Jul 10, 2024 · 2fcbb0d · 2fcbb0d
1 parent fbaa6ab
commit 2fcbb0d
Showing 1 changed file with 4 additions and 0 deletions.
diff --git a/helm-charts/common/tgi/templates/deployment.yaml b/helm-charts/common/tgi/templates/deployment.yaml
@@ -38,6 +38,10 @@ spec:
               value: {{ .Values.global.HUGGINGFACEHUB_API_TOKEN | quote}}
             - name: HF_TOKEN
               value: {{ .Values.global.HUGGINGFACEHUB_API_TOKEN | quote}}
+            - name: MAX_INPUT_TOKENS
+              value: "1024"
+            - name: MAX_TOTAL_TOKENS
+              value: "4096"
             - name: http_proxy
               value: {{ .Values.global.http_proxy }}
             - name: https_proxy