Set up myst.tools via mystjs

[choldgraf]

Context

Currently we have all of our documentation written in either Sphinx or Jupyter Book, which both share the same stack. We've had plans for a while now to build out myst.tools as a space to highlight MyST Markdown specifically, in a way that is agnostic to Jupyter Book. However, we haven't yet made the time to do this, see:

• Write out a strategy / vision for MyST markdown #538

Curvenote (👋 @rowanc1 , @stevejpurves, @fwkoch) have been working on another MyST tool for the Curvenote platform, it's also open source and serves a book-like interface via a CLI and server. This behaves kind-of like Jupyter Book with a different take on the implementation and server.

I spoke with @rowanc1 and he is interested in finding ways to upstream more of the Curvenote CLI into the EBP organization. Doing this could be a nice way to build on the "MyST in JS" stack we've started. To learn more, I suggested that we find ways to build experience with the Curvenote CLI and see how it feels. Rowan also said he'd be happy to make a push on the myst.tools site as part of this.

Proposal

Let's try building myst.tools as a Curvenote website that we can host as a collection of MyST Markdown files, and that we serve via something like Netlify. @rowanc1 and the Curvenote team can build on the MyST docs in the process, and we can test out the Curvenote tool for this use-case. If we decide this isn't the right tool, it shouldn't be hard for us to switch to Jupyter Book or some other system, because it is all written in MyST Markdown.

I think as a part of this we should be on the lookout for a few questions:

• What is the UX like for using this tool vs. Jupyter Book? Any learning that can happen in either direction?
• What's it like to configure, extend, etc the Curvenote CLI? Any improvements to make there?
• What functionality could potentially be upstreamed into EBP vs. remaining in the Curvenote org?

I wanted to see if anybody has any opinions, thoughts, objections, etc - @rowanc1 if you'd like to provide your own thoughts or ideas, please do so!

[rowanc1]

Thanks @choldgraf for starting this! I am pretty excited to take a crack at this and will share a bit more about what we are trying at the next team meeting. I have been working pretty heavily on this over the last two weeks and some of the screeshot-progress is documented here:
https://twitter.com/rowancockett/status/1562133043296382976

I think some of the major pieces that are different are the ability to cross-reference content as well as publish multiple sites to be displayed together in a single experience.

[echarles]

...and will share a bit more about what we are trying at the next team meeting.

I was at the meeting, thx for working on this. I understand that you plan to add more features into the GitHub executablebooks organisation to allow processing MyST format with javascript and creating documentation site. I have not thought yesterday to ask more details on the repositories where this would land, on the planned features...

I am interested to know more about these (maybe it is already discussed/documented somewhere, but have not found that). No hurry, I understand this is WIP but good to add as much visibility and plan as we can so anyone can comment.

[rowanc1]

Hi @echarles thanks for coming yesterday, it was good to see you at the meeting!

A lot of the work is being done in https://github.com/curvenote/curvenote at the moment. It is MIT licensed over there, and I think that we will be able to move over the relevant packages in about two weeks to the mystjs repository. There are about three technical blockers left that we are working through.

I agree completely that the visibility isn't great at the moment, and we are working hard to get this into a place where many more people can contribute!

If you do want to try out the tools currently, you can npm install curvenote and then access the myst CLI tool that it exports, or browse the packages/source to get a feel for what is there. Happy to take any questions, and issues can be opened up on mystjs, which will be the ultimate home for this work.

I hope that helps, and glad you are interested in this! :)

[echarles]

Thx @rowanc1 I had already tried curvenote some time ago. The main reason I guess for my previous comment is to make sure that any time invested in using the current curvenote implementation would not be lost, e.g. if lot of things need to be changed to solve the 3 blockers you are referring to. Thx again for the work and openness on this. Please ping me for early test, or even for help/ideas to resolve those issues.

So this will land in https://github.com/executablebooks/mystjs? (just asking as I have seen quite some movement in the executablebooks repos, which is a good sign btw).

[rowanc1]

I have put together as many details as I can think of at the moment in this issue here:

• Major upcoming functionality for mystjs #838
  Thanks for prompting this, and hopefully it is a bit more clear, but I am likely overlooking many things. Can you comment on that issue if things remain unclear or if you want more details about anything?!

I would love to have you poking around and testing/contributing! Is there a place you are interested in specifically (parsers, transformers, renderers, react components, latex, word, ... esm/cjs/jest debugging?!)?

I think some pieces of the CLI that also aren't stable are the way to render/build/export other documents, which I think can be cleaner/more concise for myst than what we have built so far.

[choldgraf]

So my understanding is that currently, myst.tools is built with the CurveNote CLI as it has not yet been upstreamed into MyST JS. The plans for doing that are here:

• Major upcoming functionality for mystjs #838

Is that right?

If so, does it make sense to close this issue when the following two things happen:

• The necessary functionality is upstreamed into MyST JS
• myst.tools now uses MyST JS as part of its continuous deployment system

?

[fwkoch]

That's a great plan for the remaining requirements of this issue.

One subtle clarification: we will be upstreaming functionality into myst-cli; mystjs will continue to be used there as it is used currently in the curvenote cli. Hopefully upcoming diagrams of all these different parts will clarify where each piece fits in...

[choldgraf]

@rowanc1 is this one now done?

[rowanc1]

We haven't yet got the docs deployment into CI, I think that could be taken up in a new issue though! I have opened that up here:

• Deploy documentation through CI jupyter-book/mystmd#204

Next week I will also move over the myst-tools-theme from Curvenote, and give it a better name "myst-tools-website" as it isn't able to be reused by other sites currently, I think that action will closes this issue.