using ai-shell against the grok API - "Your script" section is empty often but not always

[felixniemeyer]

I often get answers like this:

┌  AI Shell
│
◇  Your script:



•
│
◇  Explanation:

The script is a simple Python program that greets a user and performs basic arithmetic.

Steps:
1. Prompt user for their name
2. Print personalized greeting
3. Ask user for two numbers
4. Calculate and display sum of numbers

Sometimes the answer does contain a script.

From reading other issues this might be about model quality or model behavior. It would be awesome if ai-shell would work with grok API or any other openai-api-supporting LLM endpoint :)

[felixniemeyer]

Here is a log of the stream from grok, that too resulted in a empty "Your script" section

◇  What would you like me to do?
│  say hi
│
◇  Your script:

data: {"id":"573a9b37-df1f-4bb1-88e3-cc80056db8cf","object":"chat.completion.chunk","created":1734430703,"model":"grok-2-1212","choices":[{"index":0,"delta":{"content":"```","role":"assistant"}}],"usage":{"prompt_tokens":88,"completion_tokens":1,"total_tokens":89,"prompt_tokens_details":{"text_tokens":88,"audio_tokens":0,"image_tokens":0,"cached_tokens":0}},"system_fingerprint":"fp_c0dce543ab"}
data: {"id":"573a9b37-df1f-4bb1-88e3-cc80056db8cf","object":"chat.completion.chunk","created":1734430703,"model":"grok-2-1212","choices":[{"index":0,"delta":{"content":"echo","role":"assistant"}}],"usage":{"prompt_tokens":88,"completion_tokens":2,"total_tokens":90,"prompt_tokens_details":{"text_tokens":88,"audio_tokens":0,"image_tokens":0,"cached_tokens":0}},"system_fingerprint":"fp_c0dce543ab"}
data: {"id":"573a9b37-df1f-4bb1-88e3-cc80056db8cf","object":"chat.completion.chunk","created":1734430703,"model":"grok-2-1212","choices":[{"index":0,"delta":{"content":" \"","role":"assistant"}}],"usage":{"prompt_tokens":88,"completion_tokens":3,"total_tokens":91,"prompt_tokens_details":{"text_tokens":88,"audio_tokens":0,"image_tokens":0,"cached_tokens":0}},"system_fingerprint":"fp_c0dce543ab"}
data: {"id":"573a9b37-df1f-4bb1-88e3-cc80056db8cf","object":"chat.completion.chunk","created":1734430703,"model":"grok-2-1212","choices":[{"index":0,"delta":{"content":"hi","role":"assistant"}}],"usage":{"prompt_tokens":88,"completion_tokens":4,"total_tokens":92,"prompt_tokens_details":{"text_tokens":88,"audio_tokens":0,"image_tokens":0,"cached_tokens":0}},"system_fingerprint":"fp_c0dce543ab"}
data: {"id":"573a9b37-df1f-4bb1-88e3-cc80056db8cf","object":"chat.completion.chunk","created":1734430703,"model":"grok-2-1212","choices":[{"index":0,"delta":{"content":"\"","role":"assistant"}}],"usage":{"prompt_tokens":88,"completion_tokens":5,"total_tokens":93,"prompt_tokens_details":{"text_tokens":88,"audio_tokens":0,"image_tokens":0,"cached_tokens":0}},"system_fingerprint":"fp_c0dce543ab"}
data: {"id":"573a9b37-df1f-4bb1-88e3-cc80056db8cf","object":"chat.completion.chunk","created":1734430703,"model":"grok-2-1212","choices":[{"index":0,"delta":{"content":" |","role":"assistant"}}],"usage":{"prompt_tokens":88,"completion_tokens":6,"total_tokens":94,"prompt_tokens_details":{"text_tokens":88,"audio_tokens":0,"image_tokens":0,"cached_tokens":0}},"system_fingerprint":"fp_c0dce543ab"}
data: {"id":"573a9b37-df1f-4bb1-88e3-cc80056db8cf","object":"chat.completion.chunk","created":1734430703,"model":"grok-2-1212","choices":[{"index":0,"delta":{"content":" festival","role":"assistant"}}],"usage":{"prompt_tokens":88,"completion_tokens":7,"total_tokens":95,"prompt_tokens_details":{"text_tokens":88,"audio_tokens":0,"image_tokens":0,"cached_tokens":0}},"system_fingerprint":"fp_c0dce543ab"}
data: {"id":"573a9b37-df1f-4bb1-88e3-cc80056db8cf","object":"chat.completion.chunk","created":1734430703,"model":"grok-2-1212","choices":[{"index":0,"delta":{"content":" --","role":"assistant"}}],"usage":{"prompt_tokens":88,"completion_tokens":8,"total_tokens":96,"prompt_tokens_details":{"text_tokens":88,"audio_tokens":0,"image_tokens":0,"cached_tokens":0}},"system_fingerprint":"fp_c0dce543ab"}
data: {"id":"573a9b37-df1f-4bb1-88e3-cc80056db8cf","object":"chat.completion.chunk","created":1734430703,"model":"grok-2-1212","choices":[{"index":0,"delta":{"content":"tts","role":"assistant"}}],"usage":{"prompt_tokens":88,"completion_tokens":9,"total_tokens":97,"prompt_tokens_details":{"text_tokens":88,"audio_tokens":0,"image_tokens":0,"cached_tokens":0}},"system_fingerprint":"fp_c0dce543ab"}
data: {"id":"573a9b37-df1f-4bb1-88e3-cc80056db8cf","object":"chat.completion.chunk","created":1734430703,"model":"grok-2-1212","choices":[{"index":0,"delta":{"content":"```","role":"assistant"}}],"usage":{"prompt_tokens":88,"completion_tokens":10,"total_tokens":98,"prompt_tokens_details":{"text_tokens":88,"audio_tokens":0,"image_tokens":0,"cached_tokens":0}},"system_fingerprint":"fp_c0dce543ab"}
data: {"id":"573a9b37-df1f-4bb1-88e3-cc80056db8cf","object":"chat.completion.chunk","created":1734430703,"model":"grok-2-1212","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":"stop"}],"usage":{"prompt_tokens":88,"completion_tokens":11,"total_tokens":99,"prompt_tokens_details":{"text_tokens":88,"audio_tokens":0,"image_tokens":0,"cached_tokens":0}},"system_fingerprint":"fp_c0dce543ab"}
data: [DONE]

[felixniemeyer]

i tracked it down to an issue with the back ticks.
grok answers something like

```ls -ls```

and maybe the start detection doesn't work with that or so:

            // Use buffer only for start detection
            if (!dataStart) {
              // Append content to the buffer
              buffer += content;
              if (buffer.match(excludedPrefix ?? '')) {
                dataStart = true;
                // Clear the buffer once it has served its purpose
                buffer = '';
                if (excludedPrefix) break;
              }
            }

I just instructed the llm to use no backticks at all. This is a workaround that works for me as far as I can see.

const generationDetails = dedent`
    Only reply with the single line command. No backticks....