Bug on Anthropic Claude when oneOf union type comes.

[samchon]

Summary

When union type comes, Anthropic Claude fills to JSON string value in the function calling.

Such string value filling problem occurs even when the target type is not string, but object.

// EXPECTED BEHAVIOR
{
  input: {
    type: "circle",
    radius: 5,
    center: { x: 0, y: 0 },
  },
};

// ACTUAL BEHAVIOR
{
  input" '{"type": "circle", "radius": 5, "center": {"x": 0, "y": 0}}',
};

Here is the TypeScript type (and JSON schema file) tried to run the LLM function calling.

type Shape = IRectangle | ICircle;
interface IRectangle {
  type: "rectangle";
  p1: IPoint;
  p2: IPoint;
}
interface ICircle {
  type: "circle";
  radius: number;
  center: IPoint;
}
interface IPoint {
  x: number;
  y: number;
}

https://github.com/samchon/openapi/blob/master/examples/function-calling/schemas/claude.union.schema.json

How to reproduce

git clone https://github.com/samchon/openapi

At first, clone this repository.

CLAUDE_API_KEY=<your_claude_api_key>

And then, create a .env file like above.

npm install
npm run build
npm run test -- --include claude_function_calling_union

TypeGuardError: Error on typia.assert(): invalid type on $input, expect to be (ICircle | IRectangle)
    at Object._assertGuard (D:\github\samchon\openapi\node_modules\typia\src\internal\_assertGuard.ts:10:16)
    at D:\github\samchon\openapi\bin\test\features\llm\claude\test_claude_function_calling_union.js:235:142
    at D:\github\samchon\openapi\bin\test\features\llm\claude\test_claude_function_calling_union.js:245:39
    at D:\github\samchon\openapi\bin\test\features\llm\claude\test_claude_function_calling_union.js:248:20
    at Generator.next (<anonymous>)
    at D:\github\samchon\openapi\bin\test\features\llm\claude\test_claude_function_calling_union.js:31:71
    at new Promise (<anonymous>)
    at __awaiter (D:\github\samchon\openapi\bin\test\features\llm\claude\test_claude_function_calling_union.js:27:12)
    at Object.handleCompletion (D:\github\samchon\openapi\bin\test\features\llm\claude\test_claude_function_calling_union.js:157:38)
    at D:\github\samchon\openapi\bin\test\utils\ClaudeFunctionCaller.js:1005:25 {
  method: 'typia.assert',
  path: '$input',
  expected: '(ICircle | IRectangle)',
  value: '{"type": "circle", "radius": 5, "center": {"x": 0, "y": 0}}'
}

https://github.com/samchon/openapi/blob/master/test/features/llm/claude/test_claude_function_calling_union.ts

At last, install, build and run test program function of above.

Then you may meet the above error message:

anyOf instead of oneOf

npm run test -- --include claude_function_calling_union --model chatgpt

https://github.com/samchon/openapi/blob/master/examples/function-calling/schemas/chatgpt.union.schema.json

If you run the test program function again with the --mode chatgpt argument, the test program function will utilize the OpenAI ChatGPT's JSON schema speification instead. And there is not any problem in that case. It is because OpenAI supports only anyOf type, and it has banned the oneOf type. And the anyOf type does not occur the string value filling bug.

By the way, following the Claude documents, it seems like Claude is supporting the oneOf type. Also, I prefer the oneOf type and think that the oneOf type is much proper for LLM function calling, because it can utilize the discriminator property which directly indicates how to distinguish the union type.

Therefore, I report this phenomenon considering as a bug.

JSON Schema Speficiation

Looking at guide documents of Claude and taking experiments of function calling, I've defined JSON schema speficiation for Anthropic Claude like below. If you find something wrong, then please comment me to fix it.

• IChatGptSchema: OpenAI ChatGPT
• IClaudeSchema: Anthropic Claude (same with ILlmSchemaV3_1)
• ILlmSchemaV3_1: middle layer based on OpenAPI v3.1 specification