gptel-copilot feature: looking for testers and feedback!

[karthink]

I've added a gptel-complete command for a "copilot" or coding assistant-style workflow. This feature is on the "copilot" branch of gptel.

If you are interested in this feature, I would be glad to get your feedback on it. There are two aspects to the feature:

1. The quality of the completions.
2. The quality of the UI and the user experience.

This testing focuses on the latter, so please let me know how you find the UI.

The former depends heavily on what context we send to the LLM. Right now the context is limited to everything in the current before before (point), so the quality will not be good. We will fix this soon, once I start working on the context UI. We can use LSP/eglot, imenu and project.el to find a reasonable amount of context to include with requests.

---

To use it, run gptel-complete anywhere in your code. It will help to have a code comment explaining what you're trying to do somewhere above the cursor.

Move your cursor into the response, this activates additional actions.

While the response is streaming in,

• you can abort it with C-c C-g.

After the response is in, you can

• Edit it as you'd like.
• Mark it with C-c SPC.
• Regenerate it with C-c M-RET if you are not happy with it.
• Delete it with C-c DEL
• Ediff it against previous responses with C-c =
• Finalize it with C-c RET.

All other gptel features are available as usual from the transient menu. (You may notice some new options as well.)

[isaacphi]

This really good so far! My usability notes:

• I couldn't get the ediff working with the hotkey or by manually calling gptel-complete-ediff. I generated one completion, then did C-c M-RET to get another then tried getting an ediff. No logs appeared in *Messages*
• I would love to be able to deal with the response more quickly as it's generating and after it's generated. Ideally be able to cancel during generation with just C-g or ESC, and after it's generated regenerate directly with M-RET, and all the other actions without prefixing with C-c.
• I know this is specific to everyone's configuration but It would be really nice to disable flycheck while it's generating and then turn it back on when it's done or if it's cancelled.

[isaacphi]

Thank you for the replies!
The cursor is in the region and toggle-debug-on-error isn't triggering anything. gptel-complete-ediff is getting triggered and gptel-complete--bounds is getting the right values. ediff is still working elsewhere like if I use gptel with refactor. Nothing interesting appears in gptel-log or Messages. Is there something else I can check?

I'm probably missing something, but how do I switch between past responses in the transient menu?

[karthink]

I'm probably missing something, but how do I switch between past responses in the transient menu?

With the cursor inside a response region:

You need to have regenerated the response at least once, OR you need to have replaced the prompt with the response (using the i Replace/Delete prompt option in the transient menu.)

EDIT: Please ignore the fact that some of my keybindings are different from the defaults in the transient menu.

[karthink]

Is there something else I can check?

I can't think of anything else except edebugging the function gptel--ediff in gptel.el. This is the function that gptel-complete-ediff uses to run Ediff. It's possible that this function is not running at all due to an install issue -- otherwise I would expect at least an error message.

[isaacphi]

When I log inside of gptel--diff, (gptel--at-response-history-p) is nil. I also never see Previous Variant, Next variant, or Ediff previous options. I'm not exactly sure what's going on and my elisp debugging skills are pretty low so maybe let's see if anyone else reports a problem.

[karthink]

When I log inside of gptel--diff, (gptel--at-response-history-p) is nil.

The only thing I can think of is that the package isn't installed correctly -- it needs to be rebuilt (straight-rebuild-package if you use straight, I'm not sure about the Doom emacs command).

[isaacphi]

Not sure if you've considered this, but for my workflow it would be really helpful to include the context from a chat buffer as well. My main use case is that I'll be seeking advice for how to implement something generally and then switch over to a code buffer to implement it. The chat context would probably really help.

[karthink]

Setting context is an unsolved UI problem so far. I don't yet know how to make it intuitive to add/specify context. It's also expensive: I need to indicate to the user that all the context will be sent with each request.

[isaacphi]

True. In a sense I guess context is already exposed to the user via "Number of past messages to send" and gptel-set-topic (Maybe "directive" could even be considered part of the context). I'm just throwing ideas around, but could something like this work?

Context is just about as important as the immediate prompt so as a user I'd like to have it be customizable with defaults that are sensible and not too expensive like only including the current buffer.

[karthink]

The "number of past messages to send" option is clunky and will probably be removed or renamed. Besides that, the context menu placement is fine, I like the estimated token cost entry. I'm just not sure what the entries should be or how they should work.

This is the default context behavior:

• if region is active: contents of region.
• if region is inactive: everything up to (point).

This default makes sense for most chat-based interactions, so I don't want to change it. At the same time, here are some other contexts that make sense:

• Full contents of current-buffer
• One or more other buffers
• One or more other files
• Files/buffers according to some predicate: project, name-regexp, etc
• "Auto": magic behavior where gptel identifies a useful subset of things to send. (This is going to be the default for the gptel-complete functionality)

It's easy to put these entries in a "Context" menu like in your screenshot. But here are some questions I don't have answers for yet:

1. how should the default and additional contexts compose? Should it be additive, on top of the default context? Or should it replace the default context? I can think of scenarios where either behavior makes sense.
2. What does having a region selected signify when adding additional context?
3. Should they be set for the next query only (like the plumbing arguments "p", "m", "k" etc), or persistent for the session like the model parameters? Both versions make sense in different contexts. Also, how should this behavior be communicated to the user?
4. From several issues in this repo, it's clear that many users believe the LLM has some internal state and only their latest query is being sent with each request, as opposed to the entire conversation. So the assumption will be that a large payload of additional context is sent only once. There needs to be some kind of guard against this. I'd like to avoid asking for confirmation each time, even if there's a customization option to turn it off.
5. Should this context be part of the system message/directive or included as part of the first query?
6. What should the UI be for specifying other buffers/files? completing-read-multiple is somewhat messy, but using dired or buffer-list/ibuffer might be overkill.

[isaacphi]

These are tough questions. I think it's important for the user to have a very clear and consistent mental model of how it really works, and also have a clear definition for each term involved. Here's my 2 cents

1. how should the default and additional contexts compose? Should it be additive, on top of the default context? Or should it replace the default context? I can think of scenarios where either behavior makes sense.

Additive because I can't think of examples where I'd want to exclude content and not be fine with turning it off in the menu. Additive is also more flexible and intuitive IMO. Making something that tries to have both AND and OR options together can be quite confusing.

2. What does having a region selected signify when adding additional context?

I guess here I'd really like to clarify exactly what context means. To me, context is Directive + Additional files, buffers, etc. I would separate that from the prompt or the first chat message that is sent. So if something is selected, that's the prompt, if not the prompt is the buffer up to the point. The context is always the same.

3. Should they be set for the next query only (like the plumbing arguments "p", "m", "k" etc), or persistent for the session like the model parameters? Both versions make sense in different contexts. Also, how should this behavior be communicated to the user?

Very much just personal preference, but I'd like them to persist because I can picture myself using the same args over and over. I guess the plumbing arguments can be saved so it doesn't matter too much.

4. From several issues in this repo, it's clear that many users believe the LLM has some internal state and only their latest query is being sent with each request...

Same, I needed to update my mental model of how these LLMs work once I started diving in deeper. Maybe having a token estimate for messages sent helps. I also assume most people read the top of the Readme and/or watch your intro video before using it, so I think education there could also help. I agree on no confirmation message.

5. Should this context be part of the system message/directive or included as part of the first query?

I'm guessing system message if only because it makes things clearer conceptually. Maybe one or the other will actually result in better performance though.

6. What should the UI be for specifying other buffers/files? completing-read-multiple is somewhat messy, but using dired or buffer-list/ibuffer might be overkill.

No strong opinion on this, but I don't mind using something more powerful if I need to navigate my source tree a bit and tick multiple files or buffers.

[tusharhero]

I am trying to use Gemma but it keeps generating commentary. How are you avoiding that?

[karthink]

The Gemini API doesn't provide for a system message, so the directive is sent as part of the query. There's not much you can do, unfortunately. Other models seem to do a better job of producing only code.

[karthink]

After working on this some more I think reusing the chat API to generate completions is unworkable. I'm going to reimplement gptel-complete using the completions APIs (for providers that offer it), and see if that fares better.

[jjlee]

Re your last comment about "completion APIs": does that boil down to "Fill In the Middle" (FIM) / "Infill"? From scattered comments on the interwebs I gather that is what copilot is doing? That would seem to make sense: for example, to fill in a method definition on a class, it may often be necessary to have as context method definitions or declarations below to know what methods to call.

For example:

• https://ollama.com/library/codellama

Fill-in-the-middle (FIM) or infill

• https://github.com/rjmacarthy/twinny?tab=readme-ov-file#fill-in-the-middle-code-completion
  https://github.com/rjmacarthy/twinny/blob/199e20b0c92152df2ff4b59721150059a17087dc/src/extension/fim-templates.ts#L27

return `<PRE>${fileContext} \n${heading}${prefix} <SUF> ${suffix} <MID>`

[jjlee]

Feedback on UI, which perhaps just reflects my ignorance of emacs overlays/fringe:

Having run gptel-complete, I end up with two icons in the fringe. I then run gptel-complete-reject, which leaves me with a left-pointing arrow in the left fringe like this: 🞀. I expected there to be nothing in the fringe at this point. At this point describe-char on that line lists no overlays, and (delete-overlay gptel-complete--overlay) (via eval-expression while on that line) yields nil.

Seems my buffer is going to get full of those fringe icons? Is that the intention, or has something gone wrong there?

[karthink]

Is that the intention, or has something gone wrong there?

This is a bug, thanks for letting me know.

I've stopped working on this feature at the moment, since the chat API is not very conducive to completion-style requests. I plan to come back to this after taking a look at other available LLM APIs.