☔️ When edit, etc. hits output token limits, output is truncated in edit and chat

[dominiccooney]

Version

Main 032645f

Describe the bug

This is an umbrella bug to collect a family of related issues with the same underlying cause.

Effect on edits:

• bug: Inline Edit Mode Cannot Add Lines in Selection #3408
• bug: Deletes end of long code sections  #3388

Effect on chats:

• bug: Output limit and stoppage during the rewrite of a 200 line class. #3472
• bug: Cody cuts off mid chat response jetbrains#1176

Expected behavior

• Edit should not truncate existing code when LLM output limit is reached. (Note, the typical use case can delete code, deleting trailing code looks like truncation, but that is the RIGHT thing to do in those cases.)

Exactly what we should do to communicate we hit a limit is unclear. @toolmantim, do you have ideas here?

Additional context

The underlying cause is: Edit, document, etc. work by having the LLM generate code. In some cases, re-generate input code with modifications. When that output is too large we can hit the token output limit of the LLM. Current behavior is to treat this as success, in the product UX and Telemetry. But that is wrong.

[dominiccooney]

Repro steps:

1. Create an empty Julia file
2. Option-K, ask Claude/Opus 3 to write a Lua parser and interpreter. (Claude/Opus 3 is not up to this task, although its attempt is interesting...)

We should add metrics for stop reason in edit output to assess how often this happens. It appears we record the interaction as a success.

Actual behavior: The output is truncated after 127 lines ending abruptly with function interpret(ast, scope=LuaScope(nothing. This is almost certainly hitting a stop token but we are not giving any feedback about what's happening.

Expected behavior: ?

[toolmantim]

What are our technical options here? Can we continue the stream in some way automatically, or with a button? Is the token limit too small currently?

[dominiccooney]

@toolmantim

What are our technical options here?

• We get a different result from the LLM when we've hit a limit. So we know when we're entering the danger zone.
• We have a bunch of "generation" cases (document, generate unit test, edits with an empty selection) and we could treat them differently.
• Edit-in-place is a harder case. We can guess at the suffix of the input that would be deleted, but only roughly at the margin an there may be some duplication or mess there. (Because we display the "thinking" blue/orange rect highlights we could give up in this case, question how we communicate that. Generation we stream in, it would be frustrating to see awesome code and have it snatched away from you at the end.)

Can we continue the stream in some way automatically, or with a button?

For generation, that might be plausible.

Needs investigation to work out what the prompt would be. First step is to gather some real-life long generation prompts. @chillatom, in the work you and @rishabhmehrotra were doing looking at chat use, do you have some "long form code generation" user input we could look at?

Is the token limit too small currently?

Evidently for some people but unquantified. Short term we should count hitting the token limit and bump the limits if we can.

[github-actions]

This issue is marked as stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed automatically in 5 days.