Symex 2.0 Release Prep

[countvajhula]

Summary of Changes

All the changes needed for a 2.0 release, including polishing the tree-sitter functionality, improvements and bug fixes.

Remaining Work

See the ✨ Kanban Board ✨

WIP

• Reduce reliance on evil (incl. eliminating evil-surround) (Replace evil -- first pass #118 )
• Idempotence of null operations (Idempotence of null operations #98 )

Already Done

(reverse-chronological, most recently completed at the top)

• Streamline basic computation model (Streamline basic computation model #125 )
• Add effect form to be able to specify side effects to traversals (Side effect form #124 )
• Distinguish UX abstraction level (Distinguish UX abstraction level #123 )
• Ability to do transformations in the DSL (Ability to do transformations in the DSL #104 )
• More gardening 2 (More gardening 2 #107 )
• More gardening (More gardening #96 )
• Trim dependencies (Trim dependencies #94 )
• Low-level UX improvements (Low level UX improvements #93 )
• streamline definition of symex commands behind a common set of macros (Macros for defining symex commands #69)
• fully deprecating hydra (Deprecate the Hydra modal provider #70)
• Migrate old kanban board and convert the draft tickets to real issues
• fix all links to point to the drym-org account instead of countvajhula

Public Domain Dedication

• In contributing, I relinquish any copyright claims on my contribution and freely release it into the public domain in the simple hope that it will provide value.

(Why: The freely released, copyright-free work in this repository represents an investment in a better way of doing things called attribution-based economics. Attribution-based economics is based on the simple idea that we gain more by giving more, not by holding on to things that, truly, we could only create because we, in our turn, received from others. As it turns out, an economic system based on attribution -- where those who give more are more empowered -- is significantly more efficient than capitalism while also being stable and fair (unlike capitalism, on both counts), giving it transformative power to elevate the human condition and address the problems that face us today along with a host of others that have been intractable since the beginning. You can help make this a reality by releasing your work in the same way -- freely into the public domain in the simple hope of providing value. Learn more about attribution-based economics at drym.org, tell your friends, do your part.)

[countvajhula]

Hey all, hope everyone's doing great. I'm getting the ball rolling on this effort for anyone who's around to work on it 😊 I'll aim to document outstanding items more thoroughly in the coming days, but I think the Kanban board from last time should still roughly be current if you want to take a crack at anything in particular. I think we made an effort last time to keep the tasks pretty bite-sized. I'll review them more closely and update them as needed soon. Get pumped for Symex 2.0! 😉 👍

@markgdawson @anonimitoraf @pepperblue @doyougnu @dcostaras @devcarbon-com

[countvajhula]

Thanks for reviewing the changes already in the branch @j-shilling ! Just to clarify, this branch is going to serve as the integration branch for pre-2.0 changes (including the WIP PRs #69 and #70 which will be merged here as opposed to master). This is partially as a way to organize for collaboration, and partially so that we can move faster as we only need to worry about testing everything once, at the end, rather than in depth for every PR.

[countvajhula]

I've migrated and updated the Kanban board (added in the PR description) and converted all tickets into actual issues on the repo, and also added any relevant user bug reports to the board. I think the tasks are all ready to be worked on. For the numerous tree-sitter features, there are no descriptions in the issues as they should function analogously to the Lisp features. Happy to answer any questions or help in any way! I'm not a huge chat user nowadays but I'll aim to hang out in #emacs IRC as another communication option in case that's easier for anyone. If you are interested in contributing, feel free to shoot me a message there!

[countvajhula]

For anyone who wants to play with the latest and greatest in this branch (there are lots of improvements already, and of course more exciting work still planned 😸 ), you could use config resembling this:

(use-package symex
  :straight
  (symex
   :type git
   :host github
   :repo "drym-org/symex.el")
   :branch "2.0-integration"
   :config (symex-initialize))

If you're interested in working on anything in particular, comment here (or find me on IRC if I happen to be there) and I'll assign the ticket to you. Help would especially be appreciated on:

• any tree-sitter item -- e.g. shift-forward/backward etc. Each of the features should be bite-sized and doable in a single sitting (and if not, that would be good feedback) cc @polaris64
• Highlight doesn't always stay up to date with position #67 -- highlight sometimes doesn't keep up with actual cursor position
• Moving things in tree-sitter enabled buffers is -- weird.  #68 -- probably this is just that shift-forward/backward hasn't been implemented
• Using the dot (repeat) operator disables scroll margins #86 -- using the (evil) dot operator causes the scroll margins from the top/bottom of the screen to become deactivated cc @j-shilling
• Resolve Tree-sitter/Lisp parentheses handling in Insert mode #84 -- could warrant a new minor mode or two, cc @tommy-mor

[devcarbon-com]

Is the plan to replace tsc with the built-in treesit library?

[countvajhula]

@devcarbon-com That is the eventual goal, yes. It isn't currently planned to be changed as part of the 2.0 release, but I know @polaris64 has some WIP on this front so if he champions it or if it turns out to be a simple change we might include it.

[devcarbon-com]

Okay, sounds good. A precursory look seems like it will be pretty straight forward, except for maybe tsc-changed-ranges, I haven't seen a equivalent for that yet.

[countvajhula]

Hey folks! In case you've been using this branch, there were some bugs that had been introduced a little while back (e.g. in emit and capture) that were recently fixed in #123 , and there were a number of performance improvements in #124 and #125 , so you should update to the latest to get these 🙂

One of the performance improvements is in the "leap branch" feature, which now executes in O(n) where formerly it was O(nlogn). This should make it pretty seamless to use and in my experience it doesn't cause noticeable delays in most cases as it used to formerly. Some other performance improvements were in (1) making traversal execution tail-recursive, which allows it to be more memory efficient (since the in-progress results are passed into recursive calls instead of retained in the calling scope to be combined with the result after recursion). And (2) the "recursive" features like recursive tidy/indent (M-tab), recursive eval (M-e) were formerly causing some delays in execution as well, but now they are pretty zippy! In fact, when I tried recursive tidy on the master branch it actually caused Emacs to crash 😅 . Not sure if that's just me or if it's actually doing that at the moment - that would be bad, lol. In any case, these issues are fixed in this branch as it is more memory-efficient all around. The DSL is also more expressive now, with the ability to express conditional recursion via a new loop form, and arbitrary side effects via effect, and that should allow us to implement features (including existing features) more conveniently going forward.

There are a couple of introduced bugs, however -- (1) counts/quantifiers don't work for deletion at the moment, and (2) "soar" across trees (M-} M-{) operates like "leap" (} {) within a tree, instead of always crossing trees. I'm working on it and will hopefully have fixes soon, though I need to take a short break to finish some other stuff.

Enjoy, and please let me know if you run into any issues while using this branch.

[devcarbon-com]

Hurray! 🎉

Also, one of the the "bugs" sounds like a feature I want - a leap-or-soar command :D

[countvajhula]

Good to know! One of the reasons leap doesn't already soar is because accidentally crossing trees formerly had a high cost. Since the cost isn't as high now, we could potentially make leap-or-soar the default behavior, with soar being an explicit "leave this tree."

[dcostaras]

Hi @countvajhula :)

Apologies for being so quiet, it hasn't been because I'm not excited by Symex and Rigpa, work has just been nuts for the last 2 years.

For better or worse that has now come to an abrupt end and I'd love to be at least a bit more involved in the projects. I'm going to start by just catching up with the developments of the last year and a half.

Cheers,
d

[countvajhula]

Hi @dcostaras . Congrats! This is the first step on the road to liberation from the reductive capitalist imperatives that imprison us all! 😆

Less flippantly, I'm glad to hear that. Welcome back. I'm currently taking a short break from Symex to focus on Jewel, but I'll be returning to work on 2.0 soon.

Re: what you could do, I think directly helping on 2.0 release work (e.g. tree-sitter, package decomposition, DSL, refactoring, etc.) would be tricky to coordinate in the immediate timeframe, but here are some broad things on my mind that are more decoupled and could be good to work on:

• Highlight doesn't always stay up to date with position #67
• Operating on regions (Visual mode) #53
• Make spatial orientation (up/down vs left/right) customizable #49
• any ideas on idempotence? Editing the undo stack sounds risky to me, so I'm wondering if there is a clever and simple solution, or a way to use overlays to simulate a change without actually making it until the user continues typing -- or something.
• Re: Rigpa, it is in need of a modal interface provider that is global (like Hydra) and not buffer-specific (like Evil), but which provides clean mode entry and exit hooks (pre-entry, post-entry, pre-exit, post-exit) that are reliable. Hydra is not really designed to be a full fledged modal interface and wrestling with these hooks to try to force it to behave like a modal interface has been a losing battle. I'd like to transition Buffer Mode, Window Mode, and other such "global" modes to using a lightweight modal interface provider whose entire purpose is to track entry and exit from a global "mode" such that its keybindings generally take precedence over "most" other things. I don't know how big of an effort this would be but I suspect it will be not that big. It's important for it to be lightweight. See this issue for some adjacent information: Cursor disappears after exiting View mode through a "foreign key" countvajhula/rigpa#10 . Depending on the solution here, it may make sense to use this modal interface provider even for buffer-local states like Symex, but I'm not sure. I don't really have a clear idea of the considerations here, at the present time. I have a feeling @devcarbon-com might have a few (or many 😆 ) thoughts this.
• Here's another Rigpa item -- I don't fully understand keymap precedence and the utilities provided by Emacs for this, and I feel there must surely be a better solution for this: "Enter" should select in certain modes countvajhula/rigpa#13 . This is a minor but everpresent annoyance in e.g. Magit buffers, Dictionary buffers, Dired, etc. which require custom fixes each time (and sometimes haven't been fixed, like for Magit the one-off approach used for Dired etc doesn't work).

I realize that's a lot of stuff to mull over but in case you can help with any of it, that would be helpful and appreciated. If you decide to work on any component, please do reach out to anyone else who may be on the relevant threads in case they are interested in collaborating or if they have any thoughts or WIP on it. Let me know if I can help clarify anything in particular!

[dcostaras]

Hi @countvajhula haha, don't get me started on capitalism, the ultimate local maximum!

Thanks for the detailed response, in reading the mentioned threads and other issues your thorough communication is beneficial :)

Due to a rising sense that my Doom Emacs config and Evil/Symex hacks (which would be almost entirely solved by using Rigpa) are starting to grow a life of their own (maybe time to declare emacs config bankruptcy again!) and some issues with RSI, I've been thinking about how to do things differently. I've never managed to get Rigpa setup properly on my config; so, I think, that is where I'm going to start. But also I've been looking at other modal paradigms such as Kakoune and Helix and other Emacs model systems like Meow and Boon (@devcarbon-com, it looks like you're also currently looking at Meow specifically). This all makes me quite interested in the work towards a cleaner modal system for use in Rigpa and Symex. I know that's not a good first goal as I don't have the required experience with Evil, Symex and Rigpa and it's not an immediate priority for Symex/Rigpa anyway but that's what's tickling my pure aspirational and intellectual interest.

Once I've got Rigpa and Symex WIP 2.0 working I'll have a look at the above smaller, more immediate tasks.

[dcostaras]

BTW, is general project communication happening on GH issues/PRs and the Symex 2.0 board?

[countvajhula]

Yes, general project communication is happening on GitHub Issues/PRs and there are no other channels in use at the moment.

If you have any difficulties setting up Rigpa or if the instructions are inaccurate / non-ideal / don't work out of the box for any reason, please report that as it's a bug 🙂

[RomeoV]

Excited to see this come to life soon! For now, what I really only want is some vim-like navigation of the syntax tree for non-lisp languages with emacs-29 treesit, e.g. for org mode.
For example, moving in around the headings in org document using hjkl. Is this already viable if I have an org-treesit grammar installed?

[countvajhula]

Hi @RomeoV ! Being able to navigate org documents with hjkl would be great, and I'm curious what would happen if you tried that right now with org-treesit together with Symex. Assuming tree-sitter-mode is true, it should trigger structural navigations in the alpha support Symex already has for tree-sitter in the main branch, and it's possible it does work already (I've tried this with navigating JSON documents in the past and that has worked fine).

Now re: the 2.0 effort, it's high time for a status update so here it is:

I believe the 2.0 work is reasonably close, but there's only so much time in the day and lately I've been focusing on Attribution Based Economics (which Symex is one of the pilot projects for! I'm very excited about the possibilities that ABE offers for open source economic viability as there have been some very interesting breakthroughs lately that I believe resolve some questions I myself have felt were unanswered by ABE thus far), and recently gave a talk at the Foresight Institute on that subject. I will be giving another talk related to ABE at RacketCon in October, so I don't think I will have a lot of time for a deep push on 2.0, but it's silly for so many improvements to be sitting in a branch not being used.

So here's what I'm thinking: I can aim to do a triage of the 2.0 work in the near future and retain anything that is a clear improvement over master, while being more conservative with changes that aren't quite there yet. The goal would be to merge as much of this into main as possible so that further work can continue on the main branch.

What isn't done yet:

1. The initial goal was to have the Symex DSL be a level of abstraction above the primitive tree operations (i.e. tree-sitter and native ELisp), so that most/all structural features would be implemented at a high level of abstraction in Symex, and this would allow the primitives implemented by tree-sitter and Lisp to be relatively small instead of having large sets of primitive operations. We aren't especially close on this front since drawing the new abstraction boundaries will take some time, as Tree-sitter in some ways accomplishes what was initially unique about Symex -- that is, being able to assume an explicit tree representation in implementations of features. I still think it would be good to aim for this since Symex is a higher level language than tree-sitter, and thus easier even for casual users to add nontrivial features in. But as I said, we aren't especially close yet on this front and it will take more work than I can do in the immediate future, and having to use a large set of primitives to provide various features may be OK for now.

2. Tree-sitter UX polish. There are still a number of edge cases in the UX with respect to tree-sitter languages, and it would be good to tackle these one at a time as they come up.

3. Using Emacs's native Tree Sitter either directly or via tree-edit. This wasn't explicitly a goal with this release but it would be a nice one to tackle soon as many folks have brought it up.

4. Package decomposition. Symex is a big package. We want to decompose it into several small packages (e.g. DSL, interface, runtime, etc. in separate packages). We just might be able to do this as part of this release.

5. One big improvement I wanted to make was actually in Rigpa, to use the dynaring package in ways that would make the UX with Symex and Normal state very natural. But it does not seem likely I'll be able to get to it this time around.

6. Reducing reliance on Evil. This one's on @devcarbon-com 🙂

On next steps, I will aim to triage this branch and get it into a reasonably mergeworthy form. At that stage, we could use help with testing on both Lisp and especially Tree Sitter. I will post about that when we're there.

Thanks again for your interest and support.

[RomeoV]

Thanks for the detailed answer. Just one thought from my side: I do think with the inclusion of treesit in emacs, many people are looking to switch away from having both tree-sitter-mode and treesit installed. For me personally, any package focussing on tree-sitter-mode is a non-starter, as it's most likely not the future.

I think a triage for 2.0 would be useful indeed, there can always be a 3.0 release ;) Good luck also with your other work!

[countvajhula]

Heads up: I will be rebasing this branch sometime today to bring it up to date with master.

[countvajhula]

This branch has been rebased! If you happened to have work in progress that was based on this branch, you could use these instructions to bring it up to date and reflect the updated base branch correctly.