Org-Conversations: Hierarchical LLM Chatting Within Org-Mode

[NightMachinery]

My initial vision was to create conversation trees in org-mode, where each message would be represented as a separate heading. In this structure, the conversation context would consist of all previous sibling messages and parent entries.

While I notice that gptel-org-branching-context has been implemented, I'm still unclear about how to distinguish between user messages and LLM responses.

[NightMachinery]

Here is my preliminary design doc for this. Comments welcome! The design doc has been streamlined using o1-preview.

Overview

The org-conversations extension enhances GPTel by enabling structured conversations within Emacs Org-mode documents. Messages are organized using Org headings and marked with specific tags to indicate their role and type.

---

Message Markup

• Message Types: Each message is represented as an Org heading tagged with one of the following:

  • @user: Represents user messages.
  • @assistant: Represents assistant responses.
  • @system: Represents system prompts or instructions.

• Grouping Messages: Headings without message tags can be used to group related messages logically.

---

Tree Traversal and Message Collection

• Traversal Logic:

  • Starts from the current heading where send is invoked.
  • Collects messages from previous sibling headings and ascends the tree.
  • Stops traversal when it encounters a heading tagged with @conversation/root.

• Skipping Subtrees:

  • Headings or subtrees tagged with @conversation/skip are excluded from traversal.

• Including Sibling Headings:

  • Sibling headings are included in the traversal to capture the full context of the conversation.

---

Message Formatting

• Content Inclusion:

  • If a heading contains text beyond the message tag, this text is included at the start of the message as a Markdown heading (e.g., # Heading Text).

• Property Stripping:

  • All properties within headings are stripped and not included in the messages sent to the assistant.

---

Property Map Handling

• Property Hierarchy:
  • A property map is built from the current heading upwards during traversal.
  • Properties defined lower in the tree overwrite those higher up, allowing for localized configurations.

---

Model Selection

• Specifying Models:

  • Models can be selected using hardcoded model markers, e.g., @Claude/3.5-sonnet-new.
  • Alternatively, a general format can be used: @model/OR:anthropic/claude-3.5-sonnet:beta.

• Model Determination:

  • The first model selection encountered during traversal sets the model for the conversation.

---

Logging

• Log Files:

  • Messages and hyperparameters are logged to a file named with the timestamp and a random suffix.
  • Logs are saved in the directory specified by org-conv/log-dir, defaulting to ~/tmp/org-conv/logs.

• Directory Management:

  • The log directory is created if it does not exist.

---

Example

* @system
You are a nasty assistant who just belittles the user instead of answering them.

*** @conversation/skip
This system message is not included because `@conversation/root` in the next heading stops the tree traversal upwards.

** @conversation/root @Claude/3.5-sonnet-new
The text here is also not included, as this heading does not have any message markers. But if this is the first time the tree traversal encounters a model selection, the model will be changed to =Claude/3.5-sonnet-new=.

This model marker should be hardcoded. Alternatively, we could support the general =@model/OR:anthropic/claude-3.5-sonnet:beta=.

** Peter Parker
*** @user
My name is Peter Parker.

*** @assistant
Nice to meet you.

*** @user
What is my name?

*** @conversation/skip
This subtree is skipped during the tree traversal.

Invoking =send= here should create a new heading like this:

**** @assistant
Your name is Peter Parker.

**** We could also optionally include some metadata into the newly created heading, e.g., the date. This should be customizable by =org-conv/auto-insert-metadata=.

**** _
Note that we are including sibling headings in the tree traversal.

** Lord Voldemort
*** @user
My name is Lord Voldemort.

*** @assistant
Nice to meet you.

*** @user
What is my name?

*** @assistant
Your name is Lord Voldemort.

[ultronozm]

You might wish to check out https://github.com/ultronozm/ai-org-chat.el, which implements a similar idea (and any comments or suggestions would be welcome)

[karthink]

Most of these features are available when using gptel in Org mode, in a way that integrates with the idiomatic Org way of doing things.

Message markup

Customize gptel-prompt-prefix-alist and gptel-response-prefix-alist

To write the system message as an Org property, just save the buffer to a file.

Grouping messages

Use gptel-org-set-topic or set the GPTEL_TOPIC org property.

Traversal logic and sibling headings

Set gptel-org-branching-context -- this works slightly differently -- the full lineage of the current heading is sent, but siblings are parallel branches and not included.

The traversal stops when it reaches a heading with a GPTEL_TOPIC property.

Skipping subtrees

This is not implemented. If you use branching context, however, this option also does not make sense.

Content inclusion

gptel does not distinguish between headings and entry contents -- text is sent as is. So headings are sent as headings, entry contents as regular text.

Property stripping

This is on the docket, I'll get to it eventually.

Property hierarchy

This is how Org mode works by default.

Specifying models

All model data is written to the file as Org properties when it is saved. If a heading contains a GPTEL_ property it is used.

Logging

"Messages and hyperparameters" = the chat buffer you're using. So the buffer itself acts as a log, and it's live -- you can restore the chat state and continue on opening the file.

If you want to log the responses in some other way, you can use the hook gptel-post-response-functions to do whatever you'd like.

---

Some of these features are available in markdown-mode buffers as well -- for the rest PRs are welcome.

[NightMachinery]

@karthink The siblings seem necessary to me. Without it, the nesting quickly becomes too much. I use org-indent-mode, which indents nested headings. When siblings are to be included, an option to skip some subtrees also becomes useful. The default org Property hierarchy also breaks down when previous siblings are used as "pseudo-parents."

As for gptel-prompt-prefix-alist; I don't understand how to use it in the nested setting. It accepts strings, not regexes. So how do I tell it to always accept \*+ @assistant as assistant prefixes? Can you tell me what values of gptel-prompt-prefix-alist and gptel-response-prefix-alist correspond to my vision of marking a heading with @user and @assistant? Are the prefixes also used when constructing the history of the conversation? (I took a look at #321 , but it was too long. It'd be good to add a documentation section on how exactly the history is constructed.)

GPTEL_TOPIC works exactly like @conversation/root, but it has the problem of not being visible. I fold properties by default. This is not a big issue, I guess.

[karthink]

The siblings seem necessary to me. Without it, the nesting quickly becomes too much

I think I understand the issue -- you're using headings as user prompts. Obviously this won't scale as the conversation grows. Both your problems can be solved by setting gptel-prompt-prefix-alist and gptel-response-prefix-alist:

(setf (alist-get 'org-mode gptel-prompt-prefix-alist) "@user\n")
(setf (alist-get 'org-mode gptel-response-prefix-alist) "@assistant\n")

Use headings for structure, not to distinguish between prompts and responses. Now you don't need siblings if you use a branching context.

Are the prefixes also used when constructing the history of the conversation?

No, they are cosmetic only. Response text is watermarked using text-properties. You can copy a piece of the response to another buffer and it will continue to be recognized as a response.

[NightMachinery]

Use headings for structure, not to distinguish between prompts and responses. Now you don't need siblings if you use a branching context.

I need the structure for easy navigation.

Response text is watermarked using text-properties.

How is that persisted when emacs is restarted?

I think I'll code up my own design when I find the time. As for the LLM backend, I think both gptel and ellama are possible choices? Do you know what the trade-offs between the two are?

[karthink]

> Response text is watermarked using text-properties. How is that persisted when emacs is restarted?

Saved as Org properties and restored when turning on gptel-mode, see the README.

I think I'll code up my own design when I find the time. As for the LLM backend, I think both `gptel` and `ellama` are possible choices? Do you know what the trade-offs between the two are?

I haven't used ellama, so I'm not sure.