Simpler evaluations — part 2.4 #533

csansoon · 2024-11-04T08:38:35Z

Although legacy evaluation configuration is still available from the metadata (the configuration jsonb column), all the code has been adapted to use the new configuration attribute instead (the respective EvaluationConfiguration table for each result type).

Additionally, added the prompt that will generate from the configuration of the new "simple evaluations"

csansoon · 2024-11-04T08:49:07Z

packages/core/src/services/evaluations/prompt/index.ts

+    `
+---
+provider: ${providerResult.value!.name}
+model: ${evaluation.metadata.model}
+temperature: 0.2
+schema:
+  type: object
+  properties:
+    result:
+      type: ${resultSchema[evaluation.resultType]}
+    reason:
+      type: string
+  required:
+    - value
+    - reason
+---
+
+You’re an expert LLM evaluator. Your objective is to evaluate the response from an LLM model based on this goal:
+
+${evaluation.metadata.objective}
+
+${evaluation.metadata.additionalInstructions ? 'Additionally, you should follow these instructions:\n\n' + evaluation.metadata.additionalInstructions : ''}
+
+Now, evaluate the assistant response for the following conversation, based on your instructions:
+
+{{ messages.all }}
+
+Also, here is some aditional metadata about the LLM response. It may or may not be relevant to your objective.
+ - Cost: {{ cost }} cents.
+ - Duration: {{ duration }} milliseconds.
+
+You must respond with a JSON object with the following properties:
+ - value: ${valueInformation(evaluation)}
+ - reason: A string explaining your evaluation decision.


The evaluation that will be generated from the new "simple evaluations" configuration!

...private)/evaluations/(evaluation)/[evaluationUuid]/dashboard/_components/EvaluationStats.tsx

...id]/documents/[documentUuid]/evaluations/[evaluationId]/_components/MetricsSummary/index.tsx

packages/core/src/services/evaluations/compiler/index.ts

packages/core/src/services/evaluations/prompt/index.ts

...itor/Editor/RefineModal/steps/2_SelectEvaluationResults/SelectableEvaluationResultsTable.tsx

packages/core/src/services/evaluations/prompt/index.ts

geclos · 2024-11-04T10:10:38Z

packages/core/src/services/evaluations/prompt/index.ts

+
+You must respond with a JSON object with the following properties:
+ - value: ${valueInformation(evaluation)}
+ - reason: A string explaining your evaluation decision.


is that even necessary? we are already adding a json schema and structure output in the ai request

This also adds the descriptions for these values, which are defined for the user. Also, OpenAI recommends to explain the schema in the context in addition to defining it in the configuration.

description can also be added in the json schema then

packages/core/src/services/evaluations/prompt/index.ts

csansoon commented Nov 4, 2024

View reviewed changes

csansoon force-pushed the simpler_evaluations_part_2.4 branch 4 times, most recently from 8249b97 to 84d0e4b Compare November 4, 2024 10:02