org-mode source code blocks

[wuqui]

How can use eglot for org-mode source code blocks? I'm using Doom Emacs and have just activated eglot it via the module flags. It works fine in a pure Python file, but it doesn't work in org-mode buffers (with jupyter-python blocks), neither if I edit the block in a Python buffer provided by org-src-edit. It's starting a project and connecting to the server, but I don't get any completions etc.

Am I missing something? I couldn't find anything online and I'm quite surprised because I'd assume many would want to use LSP with org source blocks. lsp-mode kind of works both in the org-mode buffers and in the dedicated ones, but with some glitches.

[joaotavora]

As far as I know, these blocks aren't real files, or rather, that aren't in files by themselves. Or if they are, they are only temporarily so. This is fundamentally contrary to how LSP works, it works with the concept of source code documents, and each document as a unique path, is opened/closed, and is persisted on some file system. It may be possible to trick an LSP server into believing there is a project that consists of multiple small files of which only one is opened at any given time, and then extract this project from an org file. But this would vary with the server, and needs lots of temporary files and management. No idea how lsp-mode does this or where its glitches come from. João

…

On Mon, Aug 17, 2020, 21:50 wuqui ***@***.***> wrote: How can use eglot for org-mode source code blocks? I'm using Doom Emacs and have just activated eglot it via the module flags. It works fine in a pure Python file, but it doesn't work in org-mode buffers (with jupyter-python blocks), neither if I edit the block in a Python buffer provided by org-src-edit. It's starting a project and connecting to the server, but I don't get any completions etc. Am I missing something? I couldn't find anything online and I'm quite surprised because I'd assume many would want to use LSP with org source blocks. lsp-mode kind of works both in the org-mode buffers and in the dedicated ones, but with some glitches. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#523>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC6PQ6GV3MZI7NTSCPJS4LSBGJZ5ANCNFSM4QCGHK3Q> .

[wuqui]

Sounds complicated indeed. Maybe some day when there is more people than just me interested in using LSP with org-mode and literate programming. Thanks for clarifying, João.

[mbarton98]

Sounds complicated indeed. Maybe some day when there is more people than just me interested in using LSP with org-mode and literate programming. Thanks for clarifying, João.

I'm also interested in this type of support or learning how to modify my workflow to take advantage of LSP.

I use Org Mode for a monthly report that has embedded python blocks to produce tables based on exported data in the directory. Works nice until something breaks and then it is a pain to step through the code to see if the exported CSV has changed the number of comment rows before the actual header row.

To make it work with LSP, I have used org tangle to save the src blocks into a file. The trick is getting the file changes back into the src blocks again.

[muffinmad]

To make it work with LSP, I have used org tangle to save the src blocks into a file. The trick is getting the file changes back into the src blocks again.

@mbarton98 Take a look at org-babel-detangle function.

test.org:

#+PROPERTY: header-args :comments link

* test-detangle
#+begin_src python :tangle "test-org.py"
from datetime import datetime
#+end_src

Tangle it, than edit test-org.py. Invoking org-babel-detangle will bring changes from test-org.py back to the test.org file.

[mbarton98]

@muffinmad Thanks for the suggestion! I was trying that out last night and it works well. It also mirrors what I do with Jupyter notebooks with the jupytext extension that mirrors the ipynb file with a py file. If I change the py file the ipynb file will be updated when launched in Jupyter lab. In that case I explore using the notebook, but want to use emacs to format the code. Anyway, my point is that the jupytext extension works for ipynb files like tangle/detangle does for Org babel source blocks.

[gmoutso]

Based on the suggestions, I made a "tangle and go to block in tangled file" function here.

[rudolf-adamkovic]

+1

I have just migrated from lsp-mode to eglot without realizing that eglot does not have lsp-org-like functionality. Dang!

@muffinmad Thanks for suggesting org-babel-detangle. Unfortunately, I get "file-equal-p: Wrong type argument: stringp, nil" when I try it.

[muffinmad]

@salutis org-babel-detangle works fine for me. Consider reproduce your error in emacs -Q and file the bug for org-mode.

[timlod]

I've been thinking about this, and was wondering if a competent solution here may be based upon indirect buffers.
Specifically, this would involve creating an indirect buffer containing all the code for each language/session combination present in the org file.

For example, if you have an org file with python src-blocks (single session, same interpreter), all of their content would be mirrored in an indirect buffer to be used for LSP. This could be a flat python project, but if the org-file resides within a project, it could also be linked to the source there. The tricky bit to me seems like it would involve efficiently tracking all src-block content in that indirect buffer, keeping it up-to-date.

I'm not an expert in elisp, so a) this may be impossible due to some technical constraint I'm not aware of or b) I may be vastly underestimating the complexity. Any expert opinions on this perhaps?

This solution would really help my workflow, so I wouldn't mind spending time implementing something functional.

[joaotavora]

Just skimmed your post @timlod and from my part you can try something. In other situations I thought about indirect buffers to solve other problems (the multi-mode problem, if I recall correctly) and I do remember @monnier reasonably competently trashing my idea :-) right away. But perhaps that criticism (the specifics of which I completely forgot) doesn't apply to this problem at all? And so it may be worth a try.

[timlod]

OK! This doesn't sound too discouraging, so I'll give it a shot ;)

Do you by any chance remember in which discussions @monnier discouraged the use of indirect buffers? Perhaps that criticism does applie here as well and I can learn from it.

[monnier]

Do you by any chance remember in which discussions @monnier discouraged the use of indirect buffers? Perhaps that criticism does applie here as well and I can learn from it.

I consider indirect buffers as an attractive nuisance. You might want to check the `lentic` package instead.

[timlod]

I researched this a bit today, and I think I agree 🙃
There is also no straight-forward way to narrow to multiple regions, which I wasn't aware of. So my idea is significantly harder than I thought it may be. I will have a look at lentic, thanks!

[timlod]

OK, this is silly, but bear with me:

What I wanted is jumping to definition, and finding references, for python source blocks in org files.
Just for fun, I did set an eglot-server-program with org-mode and pylsp.
And it just works!

One does have to disable flymake, lest there are lots of visual complaints (as a lot of the document obviously isn't valid python). But the basic functionality I sought, meaning xref, eldoc and the like, is there.

Do you see any major issues with this usage besides the obvious craziness of it?
(If there are none, I would still go and tweak the server not to report on a lot of things that will throw warnings across all the non-python - essentially turning off codestyle)

[joaotavora]

@timlod this is a bit silly indeed but there might be a slither of a possible elegant solution there. The way that LSP clients and servers communicate with each other involves transmitting between themselves the "view" of a certain file. That view does not have to correspond to what is stored in the hard drive (in fact it frequently doesn't: as you edit the file without saving, the LSP server "knows" about it). So it should be possible, though perhaps a bit laborious, to trick the LSP server into thinking you are editing a Python file when in fact you are editing an org-mode block with embedded Python blocks.

I don't have much time to detail or even outline a solution but look here https://microsoft.github.io/language-server-protocol/specification#textSynchronization-side and particularly to the didChange notification.

Just another thought: that untangling/tricking could be done by a completely independent LSP-aware and org-mode-aware proxy server.

[timlod]

I had a quick look. I think I'm out of my depth here, but I assume (at least part of) your point is that the didChange notification should only be sent when actual code is edited, ie. the point is inside a src block? This would be after also just transmitting src blocks to the server, and not the entire org document.

Unfortunately I don't think I have the time to undertake this, so I'll actually see how far I can take my dumb solution - for what it's worth, the main things work, and I'm just ignoring most of the events I know LSP is sending in the background (which say that something is very wrong with my python file :))

[dcunited001]

@wuqui @timlod @muffinmad

this is besides the point -- for specific org-src blocks, some functionality can be modified by matching against (buffer-name). the org-src-fontification buffers just handle formatting.

(defun dc/unless-org-src-fontification-activate (mode)
  "enable mode unless in an org-mode block"
  (unless (string-match (regexp-quote "*org-src-fontification:") (buffer-name))
    (apply mode '(+1))))

(add-hook! (emacs-lisp-mode clojure-mode clojurescript-mode common-lisp-mode scheme-mode)
           #'(lambda () (dc/unless-org-src-fontification-activate 'prism-mode)))

the best ways to do this are with either the detangling approach or by using the C-c ' buffers

org-src-mode-hook provides a way to inject behavior not found in the language major mode. i haven't used eglot or pylsp, but if the session has an ID that is globally available to emacs, then you can load this into your C-c ' buffer using this hook. you will maybe need a buffer-local variable.

if you look into org-src.el and ob-core.el, there is more information there. an overly complicated approach would be to modify the completion functionality based on the point context. org-babel tests whether the point is inside a #+BEGIN_SRC context.

several times, i've had mmm-mode and polymode recommended, but i havent tried them.

[nemethf]

I've just read that LSP 3.17.0 gained support for "Notebook Documents". At first glace, it seems the notebook feature might be useful for this issue. So, the lsp insufficiency label might not be true any more. I mean it surely won't be a general solution, but if a python server supports notebooks, then there's a chance Eglot/org-mode can be extended to handle python code blocks in an org-file.

[joaotavora]

Indeed that is good news, now the job to do is to figure out how well the "notebook documentes" LSP abstraction maps onto Emacs's org-mode. Do we have volunteers for that investigation?

Org-mode is part of Emacs, so it's OK to make Eglot depend on it. Nevertheless, even so, I'd like to keep these dependencies to a well-defined minimum.

[nicolas-graves]

There's also this recent post on reddit basically proposing a code that should work, only implemented for rust. https://www.reddit.com/r/emacs/comments/w4f4u3/using_rustic_eglot_and_orgbabel_for_literate/

EDIT: It relies on modifying org-babel functions to work with eglot more than by the solution of "Notebook Documents". I may try to implement that using a python workflow, to see if it still works.

[cxa]

eglot can be launched if (buffer-file-name) return non-nil value, so I add an advice to org-edit-src-code to make it work:

(require 'eglot)

(defun sloth/org-babel-edit-prep (info)
  (setq buffer-file-name (or (alist-get :file (caddr info))
                             "org-src-babel-tmp"))
  (eglot-ensure))

(advice-add 'org-edit-src-code
            :before (defun sloth/org-edit-src-code/before (&rest args)
                      (when-let* ((element (org-element-at-point))
                                  (type (org-element-type element))
                                  (lang (org-element-property :language element))
                                  (mode (org-src-get-lang-mode lang))
                                  ((eglot--lookup-mode mode))
                                  (edit-pre (intern
                                             (format "org-babel-edit-prep:%s" lang))))
                        (if (fboundp edit-pre)
                            (advice-add edit-pre :after #'sloth/org-babel-edit-prep)
                          (fset edit-pre #'sloth/org-babel-edit-prep)))))

[monnier]

(defun sloth/org-babel-edit-prep (info) (setq buffer-file-name (or (alist-get :file (caddr info)) "org-src-babel.tmp")) (eglot-ensure))

Hmmm... `setq` here is problematic (`let` would be expected instead, tho I don't know if that would work as well). And `buffer-file-name` should contain an absolute file name. But more importantly this suggests Eglot should be adjusted so it can be used in buffers that aren't "file buffers".

[joaotavora]

But more importantly this suggests Eglot should be adjusted so it can be
used in buffers that aren't "file buffers".

What would you suggest Eglot supply to the LSP server as the document URI in that case? See https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_synchronization

[joaotavora]

*would

[monnier]

> But more importantly this suggests Eglot should be adjusted so it can be > used in buffers that aren't "file buffers". What would you suggest Eglot supply to the LSP server as the document URI in that case?

Would it make sense to use the `default-directory`? Not sure it need to work in any arbitrary buffers, so maybe it could use an `eglot-document-uri` variable for that?

[joaotavora]

Would it make sense to use the default-directory?

Not really. Not only doesn't that designate a document, we're not talking about a dummy or virtual value: same (most?) servers really do need to have access to the document, virtually always a file, via means other than the LSP description of its contents.

Not sure it need to work in any arbitrary buffers, so maybe it could use an eglot-document-uri variable for that?

That sounds a lot like the buffer-file-name variable

[monnier]

> Would it make sense to use the `default-directory`? Not really. Not only doesn't that designate a document, we're not talking about a dummy or virtual value: same (most?) servers really do need to have access to the document, virtually always a file, via means other than the LSP description of its contents.

Ah, so we really need a real file? In that case, we may as well use a real "file buffer" (with a non-nil `buffer-file-name`).

> Not sure it need to work in any arbitrary buffers, so maybe it could use > an `eglot-document-uri` variable for that? That sounds a lot like the `buffer-file-name` variable

`buffer-file-name` has other consequences, e.g. with respect to auto-save and various other things, so if there's no actual file by that name, it's often better to avoid setting `buffer-file-name`.

[joaotavora]

Ah, so we really need a real file?

Presumably. sometimes/often.

buffer-file-name has other consequences, e.g. with respect to
auto-save and various other things, so if there's no actual file by that
name, it's often better to avoid setting buffer-file-name.

I tend to think that variables controlling those other consequences should be tweaked if they produce unwanted behaviour.
I don't think that auto-saving a short-lived file is so bad though. Might even be useful, I suppose, if Emacs crashes