How could I correctly pass model's arguments from langchain to Trulens ?

[kththbo]

Hello,

I'm trying to use TruLens to evaluate my RAG application by having the Provider running locally.
I am developping on a Sagemaker instance on AWS.

From a quick research online; I use langchain and the following lines to load the model

from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain_community.llms import LlamaCpp
llm_provider = LlamaCpp (model_path="/home/ec2-user/SageMaker/efs_mount/Llama-2-13B-GGUF/llama-2-13b.Q4_K_M.gguf",
n_gpu_layers=-1,
n_batch=64, 
n_ctx=1024,
f16_kv=True,
callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),
verbose=True,)

I tested the loaded model with the following line and it works fine.

llm_provider.invoke("Q: Name the planets in the solar system? A: ")

Then, I assign the loaded model to be the Trulen's Provider

from trulens_eval.feedback.provider.langchain import Langchain
provider = Langchain(chain =llm_provider)

However, when I tested the model with a direct method call:

provider.relevance("dog", "pet")

I got an error saying the following:

RuntimeError: Endpoint langchain request failed 4 time(s):
Requested tokens (512) exceed context window of 512
Requested tokens (512) exceed context window of 512
Requested tokens (512) exceed context window of 512
Requested tokens (512) exceed context window of 512

My hypothesis is that the context window (n_ctx) specified earlier in the LlamaCpp() Langchain method wasn't passed to Trulens' Provider.

The question is then, how could I correctly pass the n_ctx from when I first load the model with langchain to when I specify it to be Trulens' Provider?