Some additional org-ai commands

[yhslai]

Hello, I'm trying to implement some new commands, for example:

• org-ai-clear-chat: Clear all [ME] and [AI] but keep [SYS]
• org-ai-clear-chat-below: Clear all [ME] and [AI] below the current cursor position
• org-ai-summarize-block-inplace: Summarize the current #+begin_ai block and insert the summary after #+end_ai
• org-ai-new-chat: Create a new #+begin_ai block with the same [SYS] and options as the current one.

And perhaps a bit more options for syntax highlighting:

#+begin_ai `lang: Syntax highlighting the block as lang language but only for text between ` or between ```
#+begin_ai `auto: Syntax highlighting the text between ``` and choose the language according what's after ```

But before writing the code I'd like to know if you consider these good ideas, and if you mind me, an Elips newbie, contributing to this codebase ;p

[rksm]

Hi there, definitely, contributions are very welcome! And no worries about elisp familarity :)

For the syntax highlighting I'll probably stick with the solution mentioned in #80 (comment) which uses the github-style language specifier after the backticks. This is what the models will most likely produce and if you want to change it later you can by just editing the output.

[yhslai]

For the syntax highlighting I'll probably stick with the solution mentioned in #80 (comment) which uses the github-style language specifier after the backticks

Yeah that's what I meant by "choose the language according what's after ```", but since that you're already dealing with this issue I won't try to do it over.

My suggestion of new options is more about to NOT syntax highlight the text that's not within ``` or ` tho. Since it currently looks like this:

It's a bit distracting to see in this case as is highlightened just because these words happen to be C# keywords.

[rksm]

Hmm interesting, this shouldn't happen actually and I can't reproduce that locally.

The following does markdown + keyword highlighting, do you get the same result with that?

Code

(defcustom org-ai-hl-highlighted-words '("\[SYS\]:" "\[ME\]:" "\[AI\]:")
  "Words to highlight"
  :group 'org-ai-hl-mode)

(defvar org-ai-hl-search-list-re (regexp-opt org-ai-hl-highlighted-words)
  "regexp constructed from 'org-ai-hl-highlighted-words")

(defface font-lock-org-ai-hl-face '((((background light)) (:background "#8200f9"))
                                    (((background dark)) (:background "blue")))
  "Face used for highlighting AI keywords."
  :group 'org-ai-hl-mode)

(defun org-ai-hl-fontify-keyword (limit)
  "Highlight the next org-ai keyword in the buffer, up to LIMIT."
  (save-match-data
    (while (re-search-forward org-ai-hl-search-list-re limit t)
      (add-text-properties
       (match-beginning 0) (match-end 0)
       '(face font-lock-org-ai-hl-face)))))

(defun org-ai-hl-fontify-ai-block (limit)
  "Function used to advice `org-fontify-meta-lines-and-blocks-1' to
highlight keywords in AI blocks."
  (let ((case-fold-search t))
    (save-match-data
      (when (re-search-forward "^#\\+begin_ai[^\t\n]*" limit t)
        (let ((beg (match-beginning 0))
	      (end-of-beginline (match-end 0))
	      ;; Including \n at end of #+begin line will include \n
	      ;; after the end of block content.
	      (block-start (match-end 0))
	      (block-end nil)
	      (bol-after-beginline (line-beginning-position 2))
              (whole-blockline org-fontify-whole-block-delimiter-line))

          (when (re-search-forward "#\\+end_ai" nil t)
	    (setq block-end (match-beginning 0)
                  beg-of-endline (match-beginning 0)
		  end-of-endline (match-end 0))

            ;; Fontify the block content
            (add-text-properties
	     beg end-of-endline '(font-lock-fontified t font-lock-multiline t))

            (add-text-properties
	     block-start beg-of-endline '(face org-block))

            ;; Fontify markdown code blocks
            (save-excursion
              (goto-char block-start)
              (while (re-search-forward "^```\\(.*\\)" block-end t)
                (let ((beg-1 (match-beginning 0))
                      (md-start (+ 1 (match-end 0)))
                      (lang (match-string 1)))
                  ;; if we are inside an org block, don't fontify so that the
                  ;; syntax highlighting is less confusing
                  (when (or (string= lang "org")
                            (string= lang "org-mode"))
                    (setq lang nil))
                  (when (re-search-forward "^```$" block-end t)
                    (let ((md-end (match-beginning 0))
                          (end-2 (match-end 0)))
                      ;; prefer markdown  mode for highlighting if  available as
                      ;; it does a better job, but fall back to org mode
                      (if (fboundp 'markdown-fontify-code-block-natively)
                          (markdown-fontify-code-block-natively lang md-start md-end)
                        (org-src-font-lock-fontify-block lang md-start md-end)))))))

            ;; Fontify `org-ai-hl-highlighted-words' in the block
            (save-excursion
              (goto-char block-start)
              (org-ai-hl-fontify-keyword block-end))

	    ;; Fontify the #+begin and #+end lines of the blocks
	    (add-text-properties
	     beg (if whole-blockline bol-after-beginline end-of-beginline)
	     '(face org-block-begin-line))
            (unless (eq (char-after beg-of-endline) ?*)
	      (add-text-properties
	       beg-of-endline
	       (if whole-blockline
	           (let ((beg-of-next-line (1+ end-of-endline)))
	             (min (point-max) beg-of-next-line))
	         (min (point-max) end-of-endline))
	       '(face org-block-end-line)))))))))

[yhslai]

You meant to not activated org-ai-mode, but to call org-ai-hl-fontify-ai-block directly?

[rksm]

No inside org mode with org-ai by hooking it up with (advice-add 'org-fontify-meta-lines-and-blocks-1 :before #'org-ai-hl-fontify-ai-block). It should make the org block outside of the markdown blocks be of one text/background color.

[yhslai]

(advice-add 'org-fontify-meta-lines-and-blocks-1 :before #'org-ai-hl-fontify-ai-block)

Sorry for the late reply (I was caught in the Unity drama).

I tried the code you posted above and this is the result:

It seems that text outside ``` isn't colored any more. But it's not treated as markdown either? As #, ## and ` don't get any special treatment.

For comparsion, this is what it looks like without the code above:

I think I'm using the stable version from MELPA. I'll try to pull the newest main branch of this repo and see if the problem presents.

[yhslai]

Still looks like this. I checked org-ai's path with find-library, and it shows I'm using the local version. (with the newest commit 0fbbfca)

[tavisrudd]

Re the syntax highlighting, wouldn't the following, which already works if this is set(setq markdown-fontify-code-blocks-natively t), be sufficient?

#+begin_ai markdown
[ME]: blah blah
[AI]: blah blah 
# (ignore the escapes at these backticks. Just working around github markdown)
\```elisp
(setq xxxx nil)
\```

#+end_ai

[rksm]

Yeah, you are right, this is the better solution and actually we put it in the readme after you previously hinted at that. Sorry, I forgot. It didn't work for me correctly as you also need the markdown-mode package to be installed of course which I didn't had in my test setup. I've updated the readme. Maybe it's reasonable to add a dependency to that directly...

[tavisrudd]

I think that would be totally reasonable. The org-ai yasnippets could include markdown in the header as well.

[yhslai]

The instructions in README seems to work!

But I'm not sure if it's intentional that it colors [ME] section too?

[tavisrudd]

@yhslai That's accidental as [ME] / [AI] / [SYS] all get treated as markdown references.