An error occurred: Some selectors do not exist in the app or record.

[lorenzobalzani]

I get this error:

Source of argument context to Context Relevance (CoT) does not exist in app or  
expected record:                                                                

                                                                                
 __record__.app.retrieve[:].rets.rets[:]                                        
 # or equivalently                                                              
 Select.RecordCalls.retrieve[:].rets.rets[:]

With the following code:

context = Select.RecordCalls.retrieve[:].rets.rets[:]

# Define a groundedness feedback function
f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons, name="Groundedness (CoT)")
    .on(context)
    .on(Select.RecordOutput)
)

# Define feedback function for answer relevance
f_answer_relevance = (
    Feedback(provider.relevance_with_cot_reasons, name="Answer Relevance (CoT)")
    .on_input_output()
)

# Define feedback function for context relevance
f_context_relevance = (
    Feedback(provider.context_relevance_with_cot_reasons, name="Context Relevance (CoT)")
    .on_input()
    .on(context)
    .aggregate(np.mean)
)

I also tried:

Select.RecordCalls.retrievers[:]._get_relevant_documents.rets[:].page_content

with no luck.

[dosubot[bot]]

To resolve the error "Source of argument context to Context Relevance (CoT) does not exist in app or expected record," you need to ensure that the structure of your application matches the expected structure that these selectors are trying to access.

First, inspect the JSON-like structure of your application to verify that the paths retrieve[:].rets.rets[:] and retrievers[:]._get_relevant_documents.rets[:].page_content exist. Here is how you can inspect the structure:

response = my_llm_app(query)

from trulens_eval import TruChain
tru_recorder = TruChain(
    my_llm_app,
    app_id='Chain1_ChatApplication'
)

response, tru_record = tru_recorder.with_record(my_llm_app, query)
json_like = tru_record.layout_calls_as_app()

# Print the JSON-like structure to inspect it
import json
print(json.dumps(json_like, indent=2))

After inspecting the structure, ensure that the paths you are using in your selectors exist. If they do not, you will need to adjust your selectors to match the actual structure of your application.

For example, if the structure shows that the correct path is retrieve.rets instead of retrieve[:].rets.rets[:], you should update your selector accordingly:

context = Select.RecordCalls.retrieve.rets

If the structure is more complex and involves multiple levels, ensure that each level exists and is correctly referenced in your selector.

Additionally, if you encounter an InvalidSelector exception, it indicates that the selector names something that is missing in the record or app. You can handle this by checking the existence of the selector in the source data and providing appropriate feedback or adjustments:

from trulens_eval.feedback.feedback import InvalidSelector

try:
    input_combinations = list(
        self._extract_selection(
            source_data=source_data,
            combinations=self.combinations,
            **kwargs
        )
    )
except InvalidSelector as e:
    print(f"Selector {e.selector} does not exist in source data.")
    # Handle the missing selector case

This approach ensures that you handle missing selectors gracefully and provide meaningful feedback for debugging and resolving the issue ^[1].

To continue talking to Dosu, mention @dosu.