From d0f7c70f1f1c1cfe8f026faa8ba81c071e9c8120 Mon Sep 17 00:00:00 2001 From: Benjamin Friedman Wilson Date: Fri, 20 Oct 2023 08:48:14 +0200 Subject: [PATCH] Update Langium + Monaco Tutorials (#192) * update of langium + monaco tutorial * update generation in the web tutorial --- .../tutorials/generation_in_the_web.md | 534 +++++----------- hugo/content/tutorials/langium_and_monaco.md | 593 ++++++++++++------ 2 files changed, 546 insertions(+), 581 deletions(-) diff --git a/hugo/content/tutorials/generation_in_the_web.md b/hugo/content/tutorials/generation_in_the_web.md index 00d9e275..46f923de 100644 --- a/hugo/content/tutorials/generation_in_the_web.md +++ b/hugo/content/tutorials/generation_in_the_web.md @@ -5,391 +5,172 @@ weight: 7 {{< toc format=html >}} -*Updated on Aug. 2nd, 2023 for usage with monaco-editor-wrapper 2.1.1 & above.* +*Updated on Oct. 4th, 2023 for usage with monaco-editor-wrapper 3.1.0 & above.* -In this tutorial we'll be talking about how to perform generation in the web by executing a custom LSP command. There are multiple ways to hook into Langium to utilize the generator, such as by directly exporting the generator API. However by using the LSP as is, we can save ourselves the effort of doing additional work. By using an LSP command, we can quickly and easily integrate new functionality into our existing Langium + Monaco integration. +In this tutorial we'll be talking about how to perform generation in the web by listening for document builder notifications. There are multiple ways to hook into Langium to utilize the generator, such as by directly exporting the generator API. However, by listening to notifications from the document builder, we can do this with less code. This lets us quickly integrate new functionality into our existing Langium + Monaco integration, and focus more on what we would want to do with the generated output. -We'll assume that you've already looked over most of the other tutorials at this point. It is particularly important that you have a language with working generation, and have a working instance of Langium + Monaco for your language (or another editor of your choice). In the case that you don't have a language to work with, you can follow along with [MiniLogo](https://github.com/langium/langium-minilogo), which is the example language used throughout these tutorials. +*(This tutorial previously utilized custom LSP commands to achieve the same goal of generation. This is still a valid approach, but we've found setting up listening for notifications this way is much more straightforward. We've implemented this in our own example languages as well, and would recommend it going forward.)* -Since we're working with MiniLogo, we already know that our generated output is in the form of drawing instructions that transform some drawing context. The generated output that we've implemented so far consists of a JSON array of commands, making it very easy to interpret. Now that we're working in a web-based context, this approach lends itself naturally towards manipulating an HTML5 canvas. +We'll assume that you've already looked over most of the other tutorials at this point. It is particularly important that you have a language with working generation, and have a working instance of Langium + Monaco for your language (or another editor of your choice). In the case that you don't have a language to work with, you can follow along with [MiniLogo](https://github.com/langium/langium-minilogo), which is the example language used throughout many of these tutorials. -The parts that we still need to setup are: - -- exposing the generator via a custom LSP command -- invoking this custom command and getting the result -- adding a way to translate the generated result into drawing on an HTML5 canvas. - -## Overview of LSP Commands - -Based on the work done in previous tutorials, we already have set up a working generator with MinLogo. If you haven't already set this up you can go back to the [tutorial on generation](/tutorials/generation) and give it a look over. Continuing off of the code written in that tutorial, we want to factor out our existing generator (removing any non-web compatible dependencies, like 'fs'), and invoke it via a custom LSP command handler. - -If you're not familiar with the LSP (or custom commands), that's perfectly fine. The LSP is just a protocol that defines how our client & server communicate with each other, and this works even when they're both in the same application. In our case, the server will be Langium, and the client will be Monaco. This protocol also defines a way to [describe commands](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#command), and to [execute those custom commands from the client](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#workspace_executeCommand). So, we'll be having Monaco execute a custom command, Langium handling that command, and then getting the results returned to Monaco. - -Lastly, Langium itself provides an easy way to register custom handlers for these commands. Handlers are registered for commands by name, and are invoked when that command is received. There are a number of reasons why this is a powerful approach: +Since we're working with MiniLogo here, we already know that our generated output is in the form of drawing instructions that transform some drawing context. The generated output that we've implemented so far consists of a JSON array of commands, making it very easy to interpret. Now that we're working in a web-based context, this approach lends itself naturally towards manipulating an HTML5 canvas. -- Clients can send commands without any knowledge of Langium's internals -- Langium can handle commands without any knowledge of the client sending them -- Commands can be executed server-side with *full access* to Langium's capabilities - -This effectively allows Langium to integrate with just about any application that is capable of working with the LSP (and sending custom commands). It also does this without requiring any tight dependencies on Langium itself, keeping your existing application logic separate from your Langium logic. +The parts that we still need to setup are: -## Adding a Generator Endpoint for the Web +- handle document validations, and generate notifications with our generator output +- listen for these notifications in the client, and extract the generated output +- interpret the generated output as drawing commands, and update the canvas -We'll start by adding a new file **src/web/index.ts** that will act as the generator endpoint for the web. This directory was created in the previous tutorial about running Langium + Monaco in the web, and should already contain an express `app.ts` configuration. +## Handling Document Validations -Our new file will contain a single exported function as our entry point, which will be used by our command handler. For MiniLogo we'll call this function `parseAndGenerate`. Much like the name suggests, this function takes a concrete MiniLogo program, parses it, and then generates output from the corresponding AST. This will share some logic that was used with the CLI before, so the code should be familiar if you've read the tutorial on [customizing the CLI](/tutorials/customizing_cli). +This is the first step we'll need, since without being able to generate notifications in the first place we would have nothing to listen to. -For our `parseAndGenerate` function to work, we will have to make a slight change to the way that we extract an AST node from our document. Previously, we referenced a file on disk to read from. In this context we have no such file, instead our program is a string stored in memory. So, we'll need to create an in-memory document. Once we have this document, the rest of our process is the same. We can write this supporting function for creating in-memory documents like so: +Thankfully a lot of the groundwork has already been done in previous tutorials, as well as within Langium itself. We just need to setup the an onBuildPhase listener for the document builder in our LS. Using the LS entry point **main-browser.ts** that we setup in the last tutorial on Langium + Monaco, we can add the following code to the end of our `startLanguageServer` function. ```ts -import { AstNode, LangiumServices } from "langium"; -import { URI } from "vscode-uri"; - -/** - * Extracts an AST node from a virtual document, represented as a string - * @param content Content to create virtual document from - * @param services For constructing & building a virtual document - * @returns A promise for the parsed result of the document - */ - async function extractAstNodeFromString(content: string, services: LangiumServices): Promise { - // create a document from a string instead of a file - const doc = services.shared.workspace.LangiumDocumentFactory.fromString(content, URI.parse('memory://minilogo.document')); - // proceed with build & validation - await services.shared.workspace.DocumentBuilder.build([doc], { validationChecks: 'all' }); - // get the parse result (root of our AST) - return doc.parseResult?.value as T; -} +// modified import from the previous tutorial: Langium + Monaco +import { + BrowserMessageReader, + BrowserMessageWriter, + Diagnostic, + NotificationType, + createConnection +} from 'vscode-languageserver/browser.js'; + +// additional imports +import { Model } from './generated/ast.js'; +import { Command, getCommands } from './minilogo-actions.js'; +import { generateStatements } from '../generator/generator.js'; + +// startLanguageServer... + +// Send a notification with the serialized AST after every document change +type DocumentChange = { uri: string, content: string, diagnostics: Diagnostic[] }; +const documentChangeNotification = new NotificationType('browser/DocumentChange'); +// use the built-in AST serializer +const jsonSerializer = MiniLogo.serializer.JsonSerializer; +// listen on fully validated documents +shared.workspace.DocumentBuilder.onBuildPhase(DocumentState.Validated, documents => { + // perform this for every validated document in this build phase batch + for (const document of documents) { + const model = document.parseResult.value as Model; + let json: Command[] = []; + + // only generate commands if there are no errors + if(document.diagnostics === undefined + || document.diagnostics.filter((i) => i.severity === 1).length === 0 + ) { + json = generateStatements(model.stmts); + } + + // inject the commands into the model + // this is safe so long as you careful to not clobber existing properties + // and is incredibly helpful to enrich the feedback you get from the LS per document + (model as unknown as {$commands: Command[]}).$commands = json; + + // send the notification for this validated document, + // with the serialized AST + generated commands as the content + connection.sendNotification(documentChangeNotification, { + uri: document.uri.toString(), + content: jsonSerializer.serialize(model, { sourceText: true, textRegions: true }), + diagnostics: document.diagnostics ?? [] + }); + } +}); ``` -Once we have this function in place, we can create our `parseAndGenerate` function in the same file. +And that's it for setting up the onBuildPhase listener itself. We still need to address the usage of `generateMiniLogoCmds`, which is tied to the LS implementation. -```ts -import { EmptyFileSystem } from "langium"; -import { createHelloWorldServices } from '../language/hello-world-module'; -import { Model } from "../language/generated/ast"; -import { generateCommands } from '../generator/generator'; - -/** - * Parses a MiniLogo program & generates output as a list of Objects - * @param miniLogoProgram MiniLogo program to parse - * @returns Generated output from this MiniLogo program - */ -export async function parseAndGenerate (miniLogoProgram: string): Promise { - const services = createHelloWorldServices(EmptyFileSystem).HelloWorld; - const model = await extractAstNodeFromString(miniLogoProgram, services); - // generate mini logo drawing commands from the model - const cmds = generateCommands(model); - return Promise.resolve(cmds); -} -``` +Based on the work done in previous tutorials, we already have set up a working generator with MinLogo. If you haven't already set this up you can go back to the [tutorial on generation](/tutorials/generation) and give it a look over. Ideally, we'll already have setup our `generateStatements` function for MiniLogo, meaning so long as the imported module doesn't have any modules that are browser incompatible, we should be able to use it as is. Based on the previous setup however, we should have a **generator.js** file that is free of such conflicts, as much of them should be separated into the cli directly. -Ah, but we don't yet have a `generator` folder to import from! So let's make that real quick as part of our next step. +This saves us quite a bit of time, since we don't need to handle setting up & dispatching a document for validation, we simply tap into the existing workflow and collect the result when it's ready. This is a great example of how Langium's architecture allows us to easily extend existing functionality, and add new features without having to rewrite existing code. -## Factoring out the Generator +As a concluding note for this section, don't forget to rebuild your language server bundle! It might not be a bad idea to clean as well, just to be sure everything is working as expected at this step. -While factoring out into a separate `generator` folder, it's important to make sure that the code that your generator depends on is not tightly coupled with any file system related functionality -- or anything else that is not compatible with running in the browser. As an example, the yeoman generator example produces a generator that is connected with the CLI, which uses the file system. Thankfully, the implementation is quite simple, and it's not too difficult to decouple the generator from the CLI. +## Listening for Notifications in the Client -First, create a new folder, **src/generator/** . Then, move **src/cli/generator.ts** into **src/generator/generator.ts**. Be sure to update imports in your generator, as well as anything in the CLI that references this. +The next step we need to make is to actually listen for these notifications from the client's end. This takes us back to the [Langium + Monaco](/tutorials/langium_and_monaco) setup in the previous tutorial. -Alright, now we need to decouple the file system related functionality from the generator. To do this, we're going to take our `generateCommands` function, and compress it down to this: +After starting the wrapper successfully, we want to retrieve the MonacoLanguageClient instance (a wrapper around the language client itself) and listen for `browser/DocumentChange` notifications. ```ts -/** - * Generates simple drawing commands from a MiniLogo Model - * @param model Model to generate commmands from - * @returns Generated commands that captures the program's drawing intent - */ -export function generateCommands(model: Model): Object[] { - return generateStatements(model.stmts); -} -``` - -Notice how we dropped all the other parameters, as well as any other logic *besides* the actual generation itself. This is what we want, a simple generator interface that does exactly what it says, and nothing else. However, this completely breaks the existing CLI function `generateAction` (located in **src/cli/index.ts**) that we wrote before, so we need to correct it as well. This involves moving up some of the file system logic into this function instead. +// wrapper has started... -```ts -import { extractDestinationAndName } from './cli-util'; -import path from 'path'; -import fs from 'fs'; - -export const generateAction = async (fileName: string, opts: GenerateOptions): Promise => { - const services = createHelloWorldServices(NodeFileSystem).HelloWorld; - const model = await extractAstNode(fileName, services); - - // invoke generator to get commands - const cmds = generateCommands(model); - - // handle file related functionality here now - const data = extractDestinationAndName(fileName, opts.destination); - const generatedFilePath = `${path.join(data.destination, data.name)}.json`; - if (!fs.existsSync(data.destination)) { - fs.mkdirSync(data.destination, { recursive: true }); - } - fs.writeFileSync(generatedFilePath, JSON.stringify(cmds, undefined, 2)); - - console.log(chalk.green(`MiniLogo commands generated successfully: ${generatedFilePath}`)); -}; -``` - -Now the generator is cleanly separated from our CLI, and thus from our file system dependencies. At this point we're ready to write up a custom command handler, and invoke our generator API through it. - -## Adding a Custom LSP Command Handler to Langium - -To add a custom command handler, start by modifying the existing module file for our language. For MiniLogo, this is located in **src/language/minilogo-module.ts**. In this file we can add our custom command handler as a special class: - -```typescript -import { AbstractExecuteCommandHandler, ExecuteCommandAcceptor } from 'langium'; - -... - -class MiniLogoCommandHandler extends AbstractExecuteCommandHandler { - registerCommands(acceptor: ExecuteCommandAcceptor): void { - // accept a single command called 'parseAndGenerate' - acceptor('parseAndGenerate', args => { - // invoke generator on this data, and return the response - return parseAndGenerate(args[0]); - }); - } +// get the language client +const client = wrapper.getLanguageClient(); +if (!client) { + throw new Error('Unable to obtain language client!'); } -``` - -We only need the one function `registerCommands`, which allows us to accept an arbitrary number of custom commands by name. For this example, we're going to accept a command called `parseAndGenerate`, which matches the name of our generator endpoint. Once we've accepted a command matching this name, we receive an array of arguments, and invoke the generator on the first entry. This isn't well typed in this case (just an array of `any`), but we have advance knowledge that we'll be receiving a single string as an argument -- which corresponds to the concrete text of a MiniLogo program. -To register this custom command handler, we also need to update the `createMiniLogoServices` function in the same file. Specifically we need to register this new command handler as our `ExecuteCommandHandler` for the shared LSP services this language provides. +// listen for document change notifications +client.onNotification('browser/DocumentChange', onDocumentChange); -```typescript -shared.lsp.ExecuteCommandHandler = new MiniLogoCommandHandler(); -``` - -For some context, this should follow the creation of the standard services. - -```typescript -export function createMiniLogoServices(context: DefaultSharedModuleContext): { - shared: LangiumSharedServices, - MiniLogo: MiniLogoServices -} { - const shared = inject( - createDefaultSharedModule(context), - MiniLogoGeneratedSharedModule - ); - const MiniLogo = inject( - createDefaultModule({ shared }), - MiniLogoGeneratedModule, - MiniLogoModule - ); - // add our custom command handler to our 'shared' services - shared.lsp.ExecuteCommandHandler = new MiniLogoCommandHandler(); - shared.ServiceRegistry.register(MiniLogo); - return { shared, MiniLogo }; +function onDocumentChange(resp: any) { + let commands = JSON.parse(resp.content).$commands; + // ... do something with these commands } ``` -And now our implementation features a custom command handler that takes a MiniLogo program, and returns a generated result from that program's AST. To get these changes into the language server itself, you'll want to rebuild & bundle everything once more. If you recall the command from the last tutorial, we can do this via `build:web`. - -```bash -npm run build:web -``` +Now this works, but when do we receive notifications, and how often? Well a good thing you asked, because if you started this up and began editing your program, you would be receiving a notification for every single change! Including whitespace changes. Now that's probably not what we're looking for, but the content is correct, we just want to slow it down a bit. We can do this by setting a timeout and a semaphore to prevent multiple notifications from being processed at once. -## Importing the Generator - -Now, if you've been following along with our prior tutorials, you should have a **src/static/** folder already setup with an HTML and JS file, plus an updated language server bundle. We can now go into the HTML file, and make a couple changes to our HTML file to get things ready to work with our new changes. - -- add a canvas -- add a button to trigger updating the canvas - -You should also replace the previous HTML & CSS files with the following contents. The HTML updates add Monaco, a Canvas, and a build button. The CSS styles these new additions so that they're properly aligned. - -{{< tabs "new-html-css" >}} -{{< tab "HTML" >}} - -```html - - - - - - - MiniLogo in Langium - - -

MiniLogo in Langium

- - -
- -
-
-
-
-
- -
- -
-
- - -
- -
- -
-
-
-

Powered by

- Langium -
- - - - -``` +```ts +let running = false; +let timeout: number | null = null; -{{< /tab >}} -{{< tab "CSS" >}} -We need to update our **styles.css** file as well to allow a side-by-side view of Monaco and our canvas. You can replace your previous CSS content with these new contents to achieve that effect. - -```css -html,body { - background: rgb(33,33,33); - font-family: 'Lucida Sans', 'Lucida Sans Regular', 'Lucida Grande', 'Lucida Sans Unicode', Geneva, Verdana, sans-serif; - color: white; - /* for monaco */ - margin: 0; - padding: 0; - width: 100%; - height: 100%; -} -h1 { - text-align: center; -} -#minilogo-canvas { - display: block; - margin: 8px auto; - text-align: center; -} -#page-wrapper { - display: flex; - max-width: 2000px; - margin: 4px auto; - padding: 4px; - min-height: 80vh; - justify-content: center; -} -#page-wrapper .half { - display: flex; - width: 40vw; -} -.build { - display: block; - margin: 8px auto; - width: 300px; - height: 30px; - background: none; - border: 2px #fff solid; - color: #fff; - transition: 0.3s; - font-size: 1.2rem; - border-radius: 4px; -} -.build:hover { - border-color: #6cf; - color: #6cf; - cursor: pointer; -} -.build:active { - color: #fff; - border-color: #fff; -} -footer { - text-align: center; - color: #444; - font-size: 1.2rem; - margin-bottom: 16px; -} -@media(max-width: 1000px) { - #page-wrapper { - display: block; - } - #page-wrapper .half { - display: block; - width: auto; +function onDocumentChange(resp: any) { + // block until we're finished with a given run + if (running) { + return; } - #minilogo-canvas { - margin-top: 32px; + + // clear previous timeouts + if (timeout) { + clearTimeout(timeout); } - #page-wrapper { - min-height: auto; - } -} -/* for monaco */ -.wrapper { - display: flex; - flex-direction: column; - height: 100%; - width: 100%; -} + timeout = window.setTimeout(async () => { + running = true; + let commands = JSON.parse(resp.content).$commands; + await updateMiniLogoCanvas(commands); + running = false; -#monaco-editor-root { - flex-grow: 1; + }, 200); // delay of 200ms is arbitrary, choose what makes the most sense in your use case } ``` -{{< /tab >}} -{{< /tabs >}} +And now we have a nice delay where repeated updates are discarded, until we have about 200ms without a subsequent update. That allows us to take the commands we're working with, and start doing something with them. The semaphore will prevent following updates from overriding the current run, allowing it to finish before starting a new execution. -At this point, running `npm run build:web && npm run serve` should show Monaco on the left, an empty space on the right (this is the canvas), along with an "Update Canvas" button at the bottom. If you see this, then you can trust that the layout was updated correctly. +You may have also noticed we added `updateMiniLogoCanvas` as the action to perform with our commands. This will be implemented in the next step, where we interpret our drawing commands. -We'll also want to go into **setup.js** file, and add a small modification to the end. This change will create a global function on the window, giving us a callback that lets us execute our command to parse and generate data from the current program in Monaco. It's important that this goes into the same file as your Monaco setup code, as it directly interacts with the Monaco editor language client instance. +That's it for listening for notifications! Now that we have our commands extracted, we'll can actually perform a series of drawing actions on an HTML5 canvas. -```js -// modify your previous import to bring in the appropriate monaco-vscode-api version -import { vscode } from './monaco-editor-wrapper/index.js'; +## Interpreting Draw Commands (Drawing) -... - -const generateAndDisplay = (async () => { - console.info('generating & running current code...'); - const value = client.getEditor()?.getValue()!; - // parse & generate commands for drawing an image - // execute custom LSP command, and receive the response - const minilogoCmds = await vscode.commands.executeCommand('parseAndGenerate', value); - updateMiniLogoCanvas(minilogoCmds); -}); +If you've gotten to this point then you're on the final stretch! The last part we need to implement is the actual logic that takes our drawing commands and updates the canvas. This logic will be the content of the `updateMiniLogoCanvas` function, and we'll walk through each step here. -// Updates the mini-logo canvas -window.generateAndDisplay = generateAndDisplay; +First, let's get a handle on our canvas, as well as the associated 2D context. -// Takes generated MiniLogo commands, and draws on an HTML5 canvas -function updateMiniLogoCanvas(cmds) { - // print the commands out, so we can verify what we have received. - // TODO, will change in th next section... - alert(JSON.stringify(cmds)); +```ts +const canvas : HTMLCanvasElement | null = document.getElementById('minilogo-canvas') as HTMLCanvasElement | null; +if (!canvas) { + throw new Error('Unable to find canvas element!'); } -``` - -Running the build & serve workflow again, you should be able to now click "Update Canvas" and view an alert containing your generated commands corresponding with the current MiniLogo program in Monaco. Feel free to use the **examples/langium.logo** or **examples/test.logo** to try this out. - -## Interpreting Draw Commands -If you've gotten to this point then you're on the final stretch! The last part we need to implement is the actual logic that takes our drawing commands and updates the canvas. This logic will replace the existing contents of the `updateMiniLogoCanvas` function, and we'll walk through each step here. - -First, let's get a handle on our canvas, as well as the associated 2D context. - -```js -const canvas = document.getElementById('minilogo-canvas'); const context = canvas.getContext('2d'); +if (!context) { + throw new Error('Unable to get canvas context!'); +} ``` We'll also want to clean up the context, in case we already drew something there before. This will be relevant when we're updating the canvas multiple times with a new program. -```js +```ts context.clearRect(0, 0, canvas.width, canvas.height); ``` Next, we want to setup a background grid to display. It's not essential for drawing, but it looks nicer than an empty canvas. -```js +```ts context.beginPath(); context.strokeStyle = '#333'; for (let x = 0; x <= canvas.width; x+=(canvas.width / 10)) { @@ -405,38 +186,40 @@ context.stroke(); After drawing a grid, let's reset the stroke to a white color. -```js +```ts context.strokeStyle = 'white'; ``` Let's also setup some initial drawing state. This will be used to keep track of the pen state, and where we are on the canvas. -```js +```ts // maintain some state about our drawing context let drawing = false; let posX = 0; let posY = 0; ``` -And let's begin evaluating each of our commands. To do this, we'll setup an interval that repeatedly shifts the top element from our list of commands, evaluates it, and repeats. Once we're out of commands to evaluate, we'll clear the interval. Feel free to adjust the delay (or remove it entirely) in your version. - -```js -// use the command list to execute each command with a small delay -const id = setInterval(() => { - if (cmds.length > 0) { - // evaluate the next command in the current env/context - evalCmd(cmds.shift(), context); - } else { - // finish existing draw - if (drawing) { - context.stroke(); +And let's begin evaluating each of our commands. To do this, we'll setup an interval that repeatedly shifts the top element from our list of commands, evaluates it, and repeats. Once we're out of commands to evaluate, we'll clear the interval. The whole invocation will be wrapped in a promise, to make it easy to await later on. Feel free to adjust the delay (or remove it entirely) in your version. + +```ts +const doneDrawingPromise = new Promise((resolve) => { + // use the command list to execute each command with a small delay + const id = setInterval(() => { + if (cmds.length > 0) { + dispatchCommand(cmds.shift() as MiniLogoCommand, context); + } else { + // finish existing draw + if (drawing) { + context.stroke(); + } + clearInterval(id); + resolve(''); } - clearInterval(id); - } -}, 1); + }, 1); +}); ``` -The evaluate command itself only needs to handle 4 cases: +`dispatchCommand` itself only needs to handle 4 cases: - penUp - penDown @@ -447,11 +230,11 @@ Knowing this, and the details about what properties each command type can have, *Be sure to add this function inside the `updateMiniLogoCanvas` function, otherwise it will not have access to the necessary state!* -```js -// evaluate a single command in the current context -function evalCmd(cmd, context) { - if (cmd.cmd) { - switch (cmd.cmd) { +```ts +// dispatches a single command in the current context +function dispatchCommand(cmd: MiniLogoCommand, context: CanvasRenderingContext2D) { + if (cmd.name) { + switch (cmd.name) { // pen is lifted off the canvas case 'penUp': drawing = false; @@ -468,8 +251,8 @@ function evalCmd(cmd, context) { // move across the canvas // will draw only if the pen is 'down' case 'move': - const x = cmd.x; - const y = cmd.y; + const x = cmd.args.x; + const y = cmd.args.y; posX += x; posY += y; if (!drawing) { @@ -483,44 +266,37 @@ function evalCmd(cmd, context) { // set the color of the stroke case 'color': - if (cmd.color) { + if ((cmd.args as { color: string }).color) { // literal color or hex - context.strokeStyle = cmd.color; + context.strokeStyle = (cmd.args as { color: string }).color; } else { // literal r,g,b components - context.strokeStyle = `rgb(${cmd.r},${cmd.g},${cmd.b})`; + const args = cmd.args as { r: number, g: number, b: number }; + context.strokeStyle = `rgb(${args.r},${args.g},${args.b})`; } break; + // fallback in case we missed an instruction + default: + throw new Error('Unrecognized command received: ' + JSON.stringify(cmd)); + } } } ``` -Lastly, we want to view the page with some output on the canvas when our editor is finished starting, rather than an empty half of the screen to start. We can address this by setting the `generateAndDisplay` function to be called once the editor is finished loading. We can place this anywhere after our `startingPromise` has been created. +Now that we can interpret commands into drawing instructions, we're effectively done with setting up the last part of MiniLogo. Since we're listening to document updates, we don't need to do anything other than to just start it up and start with an example program. -*If you don't recall, this promise was returned by the previous call to `client.start({...})` in our last tutorial on Langium + Monaco. If you haven't read that, it would be good to double check it out now*. - -```js -startingPromise.then(() => { - generateAndDisplay(); -}); -``` - -That's it, we're all done writing up our JS file. We should now be able to run the following (assuming the generator script is also executed by `build:web`), and get our results in `localhost:3000`. +That's it, we're all done writing up our TS file. We should now be able to run the following (assuming the generator script is also executed by `build:web`), and get our results in `localhost:3000`. ```bash npm run build:web npm run serve ``` -If all went well, you should see a white diamond sketched out on the canvas when the page loads. If not, double check that you set the `code` value correctly in your `client.start({...})` configuration. To be specific, it's under `editorConfig.code`. If you didn't, you can still add it to the underlying editor directly, assuming you have access to the client wrapper: +If all went well, you should see a white diamond sketched out on the canvas when the page loads. If not, double check that you receive & use the `code` value correctly in your `createUserConfig` function. You can also add the program yourself from here: -```js -// where client is an instance of MonacoEditorLanguageClientWrapper -// retrieve the underlying editor, and set it's value -// this is implicitly the 'main' code -client.getEditor()?.setValue(` +```minilogo def test() { move(100, 0) pen(down) @@ -532,8 +308,6 @@ def test() { } color(white) test() - -`); ``` Once you have something drawing on the screen, you're all set, congratulations! You've just successfully written your own Langium-based language, deployed it in the web, and hooked up generation to boot. In fact, you've done *quite* a lot if you've gone through all of these tutorials so far. @@ -545,13 +319,13 @@ Once you have something drawing on the screen, you're all set, congratulations! - configuring code bundling - building an extension - setting up Langium + Monaco in the web -- adding a custom LSP command & handler -- using an LSP command to drive generation & draw images +- adding a document build phase listener +- listening for notifications in the client, and using the results And the concepts that we've gone over from the beginning to now are not just for MiniLogo of course, they can be easily generalized to work for your own language as well. As you've been going through these tutorials, we hope that you've been thinking about how you could have done things *differently* too. Whether a simple improvement, or another approach, we believe it's this creative kind of thinking that takes an idea of a language and really allows it to grow into something great. -One easy point to make is how the example code shown in these tutorials is designed to designed to be easy to demonstrate. However, it can improved with better error checking, better logic, generator optimizations, etc. +One easy note is how the example code shown in these tutorials was designed to be easy to demonstrate. It could definitely be improved with better error checking, better logic, generator optimizations, etc; something to keep in mind. -It is also easy to imagine how one could extend their generator to produce their own functionality, besides drawing. It's even possible to imagine that you might have multiple generator targets, as there is no requirement to have a single generator output form like we've done in these tutorials. You could add as many different outputs forms as you need for each specific target, and even share some functionality between generators. +It's also easy to imagine how one could extend their generator to produce their own functionality besides drawing. For example, imagine that you might have multiple generator targets, as there is no requirement to have a single generator output form like we've done in these tutorials. You could add as many different output forms as you need for each specific target, and even share some functionality between generators. We hope that these tutorials have given you a practical demonstration of how to construct a language in Langium, and facilitated further exploration into more advanced topics & customizations. If you're interested about learning more about Langium, you can continue through our other tutorials, reach out to us via discussions on Github, or continue working on your Langium-based language. diff --git a/hugo/content/tutorials/langium_and_monaco.md b/hugo/content/tutorials/langium_and_monaco.md index 171d71bb..48111921 100644 --- a/hugo/content/tutorials/langium_and_monaco.md +++ b/hugo/content/tutorials/langium_and_monaco.md @@ -5,43 +5,35 @@ weight: 6 {{< toc format=html >}} -*Updated on Aug. 2nd, 2023 for usage with monaco-editor-wrapper 2.1.1 & above.* +*Updated on Oct. 4th, 2023 for usage with monaco-editor-wrapper 3.1.0 & above, as well as Langium 2.0.2* In this tutorial we'll be talking about running Langium in the web with the Monaco editor. If you're not familiar with Monaco, it's the editor that powers VS Code. We're quite fond of it at TypeFox, so we've taken the time to write up this tutorial to explain how to integrate Langium in the web with Monaco, no backend required. -As a disclaimer, just because we are using Monaco in this tutorial does not mean that you cannot use another code editor of your choice. For example, you can use Code Mirror with Langium as well. Generally, if an editor has LSP support, it is very likely you can integrate it quite easily with Langium, since it's LSP compatible. +Although we're using Monaco in this tutorial, that does not mean that you cannot use another code editor of your choice. For example, you can use Code Mirror with Langium as well. Generally, if an editor has LSP support, it is very likely you can integrate it easily with Langium, since it's LSP compatible. -Without further ado, let's jump into getting your web-based Langium experience setup. +Without further ado, let's jump into getting your web-based Langium experience setup! -## Getting your Language Setup for the Web - -To begin, you're going to need a Langium-based language to work with. We have already written [MiniLogo](https://github.com/langium/langium-minilogo) in Langium as an example for deploying a language in the web. However, if you've been following along with these tutorials, you should be ready to move your own language into a web-based context. - -Per usual, we'll be using MiniLogo as the motivating example here. +## Technologies You'll Need -Once you have a language picked, you're going to want to add a script to your **package.json** to build a web worker for your language. The bundle this script produces will contain the language server for your language. +- [Langium](https://www.npmjs.com/package/langium) 2.0.2 or greater +- [Monaco Editor Wrapper](https://www.npmjs.com/package/monaco-editor-wrapper) 3.1.0 or greater +- [ESBuild](https://www.npmjs.com/package/esbuild) 0.18.20 or greater -```json -{ - ... - "build:worker": "esbuild --minify ./out/language/main.js --bundle --format=iife --outfile=./public/minilogo-server-worker.js" -} -``` - -Now, assuming `esbuild` is installed, if we try to invoke this it *won't succeed as expected*. Instead we'll get a warning back about some dependencies that couldn't be resolved. For example: +## Getting your Language Setup for the Web -> Could not resolve "fs" +To begin, you're going to need a Langium-based language to work with. We have already written [MiniLogo](https://github.com/langium/langium-minilogo) in Langium as an example for deploying a language in the web. However, if you've been following along with these tutorials so far, you should be ready to move your own language into a web-based context. -This makes sense since we're bundling for the web, and we can't depend on packages that rely on the usual environment with a filesystem. So, we need to update our language to make it compatible in a web-based context. +Per usual, we'll be using MiniLogo as the motivating example here. ## Factoring out File System Dependencies -First off, let's create a new entry point for our language server in **src/language/main-browser.ts**. This will mirror the regular entry point that we use to build already, but will allow us to target a web-based context instead. In this file, we'll have the following contents: +In order to build for the browser, we need to create a bundle that is free of any browser-incompatible modules. To do this, let's create a new entry point for our language server in **src/language-server/main-browser.ts**. This will mirror the regular entry point that we use to build already, but will target a browser-based context instead. We'll start with the following content: ```ts import { startLanguageServer, EmptyFileSystem } from 'langium'; -import { BrowserMessageReader, BrowserMessageWriter, createConnection } from 'vscode-languageserver/browser'; -import { createHelloWorldServices } from './hello-world-module'; +import { BrowserMessageReader, BrowserMessageWriter, createConnection } from 'vscode-languageserver/browser.js'; +// your services & module name may differ based on your language's name +import { createMiniLogoServices } from './minilogo-module.js'; declare const self: DedicatedWorkerGlobalScope; @@ -52,68 +44,117 @@ const messageWriter = new BrowserMessageWriter(self); const connection = createConnection(messageReader, messageWriter); // Inject the shared services and language-specific services -const { shared } = createHelloWorldServices({ connection, ...EmptyFileSystem }); +const { shared, MiniLogo } = createMiniLogoServices({connection, ...EmptyFileSystem }); // Start the language server with the shared services startLanguageServer(shared); ``` -Again, this is based on code that was originally produced by the yeoman generator, thus the **hello world** references. +Again, this is based on code that was originally produced by the yeoman generator, so it should look familiar. -Most of this is similar to what's in the **main.ts** file. The exceptions are the message readers & writers, and the notion of an `EmptyFileSystem` for the browser. There is a virtual file system API that we could utilize on most modern browsers, but for this tutorial we'll assume we aren't using any file system. Instead we'll have a single source 'file' located in our Monaco editor in memory. +Most of this is in line with what's contained in the **main.ts** file. The exceptions being the message readers & writers, and the notion of an `EmptyFileSystem` for the browser. There is a virtual file system API that we could utilize on most modern browsers, but for this tutorial we'll assume we aren't using any file system. Instead we'll have a single source 'file' located in memory. -We'll also need to include a library to resolve the missing `DedicatedWorkerGlobalScope`, which is normally not accessible until we update our **tsconfig.json** in our project root. We need to supplement the libs entry with `DOM` and `webworker`. From the yeoman generator example, the `lib` entry usually has just `["ESNext"]`. +We'll also need to include a library to resolve the missing `DedicatedWorkerGlobalScope`, which is normally not accessible until we update our **tsconfig.json** in our project root. We need to supplement the libs entry with `DOM` and `WebWorker`. From the yeoman generator example, the `lib` entry usually has just `["ESNext"]`. ```json { "compilerOptions": { ... - "lib": ["ESNext","DOM","webworker"] + "lib": ["ESNext", "DOM", "WebWorker"] } } ``` -Going back to the script we wrote before in our **package.json**, we're now ready to change **main.js** to **main-browser.js**: +Now that we have a new entry point for the browser, we need to add a script to our **package.json** to build a web worker for this language. The bundle this script produces will contain the language server for your language. The following script example is specific to MiniLogo, but should capture the general approach quite nicely: ```json { ... - "build:worker": "esbuild --minify ./out/language/main-browser.js --bundle --format=iife --outfile=./public/minilogo-server-worker.js" + "build:worker": "esbuild --minify ./out/language-server/main-browser.js --bundle --format=iife --outfile=./public/minilogo-server-worker.js", } ``` -Running `npm run build:worker` again, we should see the bundle is successfully generated without issue. If you're still having problems building the worker, double check that you're not coupled to `fs` or other file system dependent libraries in a related file. +Assuming `esbuild` is installed, and we've properly factored out any modules that are not suitable for a browser-based context, we should be good to go! + +Running `npm run build:worker` we should see the bundle is successfully generated without issue. If you're still having problems building the worker, double check that you're not coupled to `fs` or other file system dependent modules in a related file. Note that although our generator is still connected to using the file system, it's not relevant for the worker bundle to function. ## Setting up Monaco -Now we're going to setup Monaco, but not with Langium yet, as we want to be sure it's working first. +Now we're going to setup Monaco, but not with Langium yet, as we want to be sure it's working first before connecting the two. -For convenience, we're going to use two helper libraries from npm that wrap around some of Monaco's core functionality. +For convenience, we're going to use the Monaco Editor Wrapper (MER) to wrap around some of Monaco's core functionality, along with the Monaco Editor Workers package to assist. These packages are both maintained by TypeFox, and are designed to make it easier to use Monaco in a web-based context. We'll be using the following versions of these packages: -- [monaco-editor-wrapper](https://www.npmjs.com/package/monaco-editor-wrapper) -- [monaco-editor-workers](https://www.npmjs.com/package/monaco-editor-workers) +- [Monaco Editor Wrapper](https://www.npmjs.com/package/monaco-editor-wrapper) version **3.1.0** +- [monaco-editor-workers](https://www.npmjs.com/package/monaco-editor-workers) version **0.39.0** -Both these packages should be installed as dependencies for your language. In particular, this guide will assume that you're using version **2.1.1** or later of the monaco-editor-wrapper package, and version **0.39.0** of the monaco-editor-workers package. +Both these packages should be installed as dependencies for your language. In particular, this guide will assume that you're using version **3.1.0** or later of the monaco-editor-wrapper package, and version **0.39.0** of the monaco-editor-workers package. -Additionally, we'll want to add `express` as a development dependency (don't forget to also add `@types/express` too), since we'll be using that to run a local web server to test our standalone webpage. +Additionally, we'll want a way to serve this bundled language server. The choice of how you want to go about this is ultimately up to you. Previously we've recommended `express` as a development dependency (don't forget to also add `@types/express` too), as a powerful & lightweight NodeJS server framework. However, we'll be going with the built-in NodeJS support for standing up a web-server; however again the choice is yours here. -We'll also want to add some more scripts to our package.json to copy over the necessary files from the monaco-editor-wrapper & monaco-editor-worker into the **public** folder. We'll be referencing these library assets to setup the webpage for Langium and Monaco. +We'll also want to add some more scripts to our package.json to copy over the necessary files from the monaco-editor-wrapper & monaco-editor-worker into the **public** folder. We'll be referencing these library assets to setup the webpage for Langium + Monaco. ```json { ... - "prepare:public": "shx mkdir -p ./public && shx cp -fr ./src/static/* ./public/", - "copy:monaco-editor-wrapper": "shx cp -fr ./node_modules/monaco-editor-wrapper/bundle ./public/monaco-editor-wrapper", - "copy:monaco-workers": "shx cp -fr ./node_modules/monaco-editor-workers/dist/ ./public/monaco-editor-workers", - "build:web": "npm run build && npm run prepare:public && npm run build:worker && npm run copy:monaco-editor-wrapper && npm run copy:monaco-workers" + "prepare:public": "node scripts/prepare-public.mjs", + "build:web": "npm run build && npm run prepare:public && npm run build:worker && node scripts/copy-monaco-assets.mjs", } ``` -The last script is there to provide a convenient way to invoke all the intermediate build steps in sequence. However you'll want to wait before running the `build:web` script, as we still need to add our **static** assets to make that work. +Both scripts reference *mjs* files that need to be added as well into the scripts folder: + +**scripts/prepare-public.mjs** + +```js +import * as esbuild from 'esbuild' +import shell from 'shelljs' + +// setup & copy over css & html to public +shell.mkdir('-p', './public'); +shell.cp('-fr', './src/static/*.css', './public/'); +shell.cp('-fr', './src/static/*.html', './public'); + +// bundle minilogo.ts, and also copy to public +await esbuild.build({ + entryPoints: ['./src/static/minilogo.ts'], + minify: true, + sourcemap: true, + bundle: true, + outfile: './public/minilogo.js', +}); +``` + +**scripts/copy-monaco-assets.mjs** + +```js +import shell from 'shelljs' + +// copy workers to public +shell.mkdir('-p', './public/monaco-editor-workers/workers'); +shell.cp( + '-fr', + './node_modules/monaco-editor-workers/dist/index.js', + './public/monaco-editor-workers/index.js' +); +shell.cp( + '-fr', + './node_modules/monaco-editor-workers/dist/workers/editorWorker-es.js', + './public/monaco-editor-workers/workers/editorWorker-es.js' +); +shell.cp( + '-fr', + './node_modules/monaco-editor-workers/dist/workers/editorWorker-iife.js', + './public/monaco-editor-workers/workers/editorWorker-iife.js' +); +``` + +This saves us from writing these extra details into our package json, and focusing on the overall goal each step. -If you went with another editor you would want to make sure that the assets required for that editor will also be copied into **public** folder as part of your output. +The last script, `build:web` is there to provide a convenient way to invoke all the intermediate build steps in sequence. However you'll want to wait before running the `build:web` script, as we still need to add our **static** assets to make that work; which will come in the next step. + +As a quick note, if you went with another editor you would want to make sure that the assets required for that editor will also be copied into **public** folder as part of your output. ## Setting up a Static Page @@ -128,15 +169,30 @@ Here's the raw contents of the HTML content stored in **src/static/index.html**. - MiniLogo in Langium

MiniLogo in Langium

-
- -
+ + +
+ +
+
+
+
+
+ +
+ +
+
+ + +
+
+

@@ -144,7 +200,7 @@ Here's the raw contents of the HTML content stored in **src/static/index.html**. Langium
- + ``` @@ -157,9 +213,10 @@ html,body { font-family: 'Lucida Sans', 'Lucida Sans Regular', 'Lucida Grande', 'Lucida Sans Unicode', Geneva, Verdana, sans-serif; color: white; /* for monaco */ - margin: 8px auto; - width: 80%; - height: 80%; + margin: 0; + padding: 0; + width: 100%; + height: 100%; } h1 { text-align: center; @@ -169,6 +226,39 @@ h1 { margin: 8px auto; text-align: center; } +#page-wrapper { + display: flex; + max-width: 2000px; + margin: 4px auto; + padding: 4px; + min-height: 75vh; + justify-content: center; +} +#page-wrapper .half { + display: flex; + width: 40vw; +} +.build { + display: block; + margin: 8px auto; + width: 300px; + height: 30px; + background: none; + border: 2px #fff solid; + color: #fff; + transition: 0.3s; + font-size: 1.2rem; + border-radius: 4px; +} +.build:hover { + border-color: #6cf; + color: #6cf; + cursor: pointer; +} +.build:active { + color: #fff; + border-color: #fff; +} footer { text-align: center; color: #444; @@ -176,10 +266,21 @@ footer { margin-bottom: 16px; } @media(max-width: 1000px) { + #page-wrapper { + display: block; + } + #page-wrapper .half { + display: block; + width: auto; + } #minilogo-canvas { margin-top: 32px; } + #page-wrapper { + min-height: auto; + } } + /* for monaco */ .wrapper { display: flex; @@ -187,76 +288,85 @@ footer { height: 100%; width: 100%; } + #monaco-editor-root { flex-grow: 1; } + +#status-msg { + color: red; +} ``` -Finally, there's the actual Javascript setting up our Monaco instance (stored in **src/static/setup.js**), and for setting up Langium as well. This is the most complex part of setting up Langium + Monaco in the web, so we'll walk through the file in parts. +Finally, there's the actual Javascript setting up our Monaco instance (stored in **src/static/minilogo.ts**), and for setting up Langium as well. This is the most complex part of setting up Langium + Monaco in the web, so we'll walk through the file in parts. + +(*Update on Oct. 4th, 2023: Previously we wrote this as **src/static/setup.js**. This new file can be considered the same, but reworked into TypeScript & updated for the new versions of Langium & the MER.*) First, we need to import and setup the worker, as well as some language client wrapper configuration. -```js -import { MonacoEditorLanguageClientWrapper } from './monaco-editor-wrapper/index.js'; -import { buildWorkerDefinition } from "./monaco-editor-workers/index.js"; +```ts +import { MonacoEditorLanguageClientWrapper, UserConfig } from "monaco-editor-wrapper/bundle"; +import { buildWorkerDefinition } from "monaco-editor-workers"; +import { addMonacoStyles } from 'monaco-editor-wrapper/styles'; -buildWorkerDefinition('./monaco-editor-workers/workers', new URL('', window.location.href).href, false); +/** + * Setup Monaco's own workers and also incorporate the necessary styles for the monaco-editor + */ +function setup() { + buildWorkerDefinition( + './monaco-editor-workers/workers', + new URL('', window.location.href).href, + false + ); + addMonacoStyles('monaco-editor-styles'); +} ``` -Then, we'll want to instantiate our language client wrapper. In previous versions of the `monaco-editor-wrapper` package (before 2.0.0), configuration was performed by manually setting properties on the `MonacoEditorLanguageClientWrapper` instance. However, as of 2.1.1 (at the time of writing this), the constructor for `MonacoEditorLanguageClientWrapper` now takes a configuration object as its first argument. This configuration object allows us to set the same properties as before, but with more fine-grained control over all the properties that are set. - -The configuration itself can be quite large, so we're going to walk through the parts that will be used to build up this configuration first, and then joining the actual configuration object together afterwards. +Then, we'll want to instantiate our language client wrapper. In previous versions of the `monaco-editor-wrapper` package (before 2.0.0), configuration was performed by manually setting properties on the `MonacoEditorLanguageClientWrapper` instance. However, as of 3.1.0 (at the time of writing this), the constructor for `MonacoEditorLanguageClientWrapper` now takes a configuration object as its first argument. This configuration object allows us to set the same properties as before, but with more fine-grained control over all the properties that are set. -To start, let's mark our current language id as `minilogo`. This should match the id of the language that will be recognized by our language server. +We're going to walk through the parts that will be used to build up this configuration first, and then joining the actual configuration object together afterwards. -```js -const languageId = 'minilogo'; -``` +To start, let's keep in mind that our current language id will be `minilogo`. This should match the id of the language that will be recognized by our language server. Then, we'll want to add some static syntax highlighting. To do this we have a couple choices, using a TextMate or a [Monarch grammar](https://microsoft.github.io/monaco-editor/monarch.html). Both will provide us with the ability to parse our language, and apply styling to our tokens. However we have to choose one, we cannot use both simultaneously. This is related to how Monaco itself is configured with regards to whether we're using the VSCode API config, or the classic editor config. This makes sense to a degree, as we can only prepare the editor one way or the other. For MiniLogo, our monarch grammar will look like so: -```js -const monarchGrammar = { - // recognized keywords - keywords: [ - 'color','def','down','for','move','pen','to','up' - ], - // recognized operators - operators: [ - '-',',','*','/','+','=' - ], - // pattern for symbols we want to highlight - symbols: /-|,|\(|\)|\{|\}|\*|\/|\+|=/, - - // tokenizer itself, starts at the first 'state' (entry), which happens to be 'initial' - tokenizer: { - // initial tokenizer state - initial: [ - { regex: /#(\d|[a-fA-F])+/, action: {"token":"string"} }, - { regex: /[_a-zA-Z][\w_]*/, action: { cases: { '@keywords': {"token":"keyword"}, '@default': {"token":"string"} }} }, - { regex: /-?[0-9]+/, action: {"token":"number"} }, - // inject the rules for the 'whitespace' state here, effectively inlined - { include: '@whitespace' }, - { regex: /@symbols/, action: { cases: { '@operators': {"token":"operator"}, '@default': {"token":""} }} }, - ], - // state for parsing whitespace - whitespace: [ - { regex: /\s+/, action: {"token":"white"} }, - // for this rule, if we match, push up the next state as 'comment', advancing to the set of rules below - { regex: /\/\*/, action: {"token":"comment","next":"@comment"} }, - { regex: /\/\/[^\n\r]*/, action: {"token":"comment"} }, +```ts +/** + * Returns a Monarch grammar definition for MiniLogo + */ +function getMonarchGrammar() { + return { + keywords: [ + 'color','def','down','for','move','pen','to','up' ], - // state for parsing a comment - comment: [ - { regex: /[^\/\*]+/, action: {"token":"comment"} }, - // done with this comment, pop the current state & roll back to the previous one - { regex: /\*\//, action: {"token":"comment","next":"@pop"} }, - { regex: /[\/\*]/, action: {"token":"comment"} }, + operators: [ + '-',',','*','/','+','=' ], - } -}; + symbols: /-|,|\(|\)|\{|\}|\*|\/|\+|=/, + + tokenizer: { + initial: [ + { regex: /#(\d|[a-fA-F]){3,6}/, action: {"token":"string"} }, + { regex: /[_a-zA-Z][\w_]*/, action: { cases: { '@keywords': {"token":"keyword"}, '@default': {"token":"string"} }} }, + { regex: /(?:(?:-?[0-9]+)?\.[0-9]+)|-?[0-9]+/, action: {"token":"number"} }, + { include: '@whitespace' }, + { regex: /@symbols/, action: { cases: { '@operators': {"token":"operator"}, '@default': {"token":""} }} }, + ], + whitespace: [ + { regex: /\s+/, action: {"token":"white"} }, + { regex: /\/\*/, action: {"token":"comment","next":"@comment"} }, + { regex: /\/\/[^\n\r]*/, action: {"token":"comment"} }, + ], + comment: [ + { regex: /[^\/\*]+/, action: {"token":"comment"} }, + { regex: /\*\//, action: {"token":"comment","next":"@pop"} }, + { regex: /[\/\*]/, action: {"token":"comment"} }, + ], + } + }; +} ``` We can produce this Monarch grammar by updating our **langium-config.json** to produce a Monarch file as output. Note that although we're talking about MiniLogo here, we based this example off of the hello-world example produced by the yeoman generator. As such, we still have hello world names here and there, and for this tutorial we'll just use the same name again as for the TextMate grammar. @@ -264,138 +374,219 @@ We can produce this Monarch grammar by updating our **langium-config.json** to p ```json ... "textMate": { - "out": "syntaxes/hello-world.tmLanguage.json" + "out": "syntaxes/minilogo.tmLanguage.json" }, "monarch": { - "out": "syntaxes/hello-world.monarch.ts" + "out": "syntaxes/minilogo.monarch.ts" } ``` -To generate this file, run `npm run langium:generate`. You can then copy over the definition of the grammar from **syntaxes/hello-world.monarch.ts** (or whatever other name you have given this file) into the `setMonarchTokensProvider` function to setup that highlighting. Keep in mind that this generated monarch grammar is *very* simple. If you want more complex highlighting, we recommend writing your own custom monarch grammar, and storing it somewhere else to prevent it from being overridden. If you're interested, you can find more details about the [Monarch grammar highlighting language here](https://microsoft.github.io/monaco-editor/monarch.html). +To generate this file, run `npm run langium:generate`. You can then copy over the definition of the grammar from **syntaxes/hello-world.monarch.ts** (or whatever other name you have given this file). Keep in mind that this generated monarch grammar is *very* simple. If you want more complex highlighting, we recommend writing your own custom monarch grammar, and storing it somewhere else to prevent it from being overridden. If you're interested, you can find more details about the [Monarch grammar highlighting language here](https://microsoft.github.io/monaco-editor/monarch.html). Then, we want to setup the code that shows up by default. The following is a fixed MiniLogo program that should display a white diamond in the top left corner of the screen. -```js -const code = ` -def test() { - move(100, 0) - pen(down) - move(100, 100) - move(-100, 100) - move(-100, -100) - move(100, -100) - pen(up) +```ts +/** + * Retrieves the program code to display, either a default or from local storage + */ +function getMainCode() { + let mainCode = ` + def test() { + move(100, 0) + pen(down) + move(100, 100) + move(-100, 100) + move(-100, -100) + move(100, -100) + pen(up) + } + color(white) + test() + + `; + + // optionally: use local storage to save the code + // and seek to restore any previous code from our last session + if (window.localStorage) { + const storedCode = window.localStorage.getItem('mainCode'); + if (storedCode !== null) { + mainCode = storedCode; + } + } + + return mainCode; } -color(white) -test() -`; ``` -We'll want to setup the editor config as well. This will include configurations to setup the theme, languageId, main code, and some layout. +Since we're planning to use a language server with Monaco, we'll need to setup a language client config too. To do this we'll also need to generate a worker using our language server worker file, but that's fairly straightforward to setup here. Keep in mind that you'll need to have access to the bundle produced from your **main-browser.ts** from before. Here the built result is copied over as **public/minilogo-server-worker.js**. -```js -const editorConfig = { - languageId, - code, - useDiffEditor: false, - automaticLayout: true, - theme: 'vs-dark' -}; +```ts +/** + * Creates & returns a fresh worker using the MiniLogo language server + */ +function getWorker() { + const workerURL = new URL('minilogo-server-worker.js', window.location.href); + return new Worker(workerURL.href, { + type: 'module', + name: 'MiniLogoLS' + }); +} ``` -Since we're planning to use a language server with Monaco, we'll need to setup a language client config too. To do this we'll also need to generate a worker using our language server worker file, but that's fairly straightforward to setup here. +By creating the worker in advance, we give ourselves the ability to directly interact with the worker/LS independent of the wrapper itself, and to even pre-configure it before use. This can be hugely beneficial, especially if we expect to customize our LS on the fly. -```js -// configure the worker -const workerURL = new URL('./minilogo-server-worker.js', import.meta.url); -const lsWorker = new Worker(workerURL.href, { - type: 'classic', - name: 'minilogo-language-server-worker' -}); +Lastly, let's setup the user config, which will be used to startup the wrapper. -// setup the language client config with the worker -const languageClientConfig = { - enabled: true, - useWebSocket: false, - // can pass configuration data, or a pre-configured worker as well - // the later works better for us in this case - workerConfigOptions: lsWorker -}; -``` +```ts +type WorkerUrl = string; -We can also pass more explicit config options to `workerConfigOptions` if we don't want to create the worker ourselves. For example: +/** + * Classic configuration for the monaco editor (for use with a Monarch grammar) + */ +interface ClassicConfig { + code: string, + htmlElement: HTMLElement, + languageId: string, + worker: WorkerUrl | Worker, + monarchGrammar: any; +} -```js -{ - url: new URL('./minilogo-server-worker.js', import.meta.url), - type: 'classic', - name: 'minilogo-language-server-worker' +/** + * Generates a valid UserConfig for a given Langium example + * + * @param config An extended or classic editor config to generate a UserConfig from + * @returns A completed UserConfig + */ +function createUserConfig(config: ClassicConfig): UserConfig { + // setup urls for config & grammar + const id = config.languageId; + + // generate langium config + return { + htmlElement: config.htmlElement, + wrapperConfig: { + editorAppConfig: { + $type: 'classic', + languageId: id, + useDiffEditor: false, + code: config.code, + theme: 'vs-dark', + languageDef: config.monarchGrammar + }, + serviceConfig: { + enableModelService: true, + configureConfigurationService: { + defaultWorkspaceUri: '/tmp/' + }, + enableKeybindingsService: true, + enableLanguagesService: true, + debugLogging: false + } + }, + languageClientConfig: { + options: { + $type: 'WorkerDirect', + worker: config.worker as Worker, + name: `${id}-language-server-worker` + } + } + }; } ``` -Either case works, but by creating the worker in advance, we give ourselves the ability to directly interact with the worker/LS independent of the wrapper itself, and to even pre-configure it before use. This can hugely beneficial, especially if we expect to customize our LS on the fly. +This particular UserConfig will be for configuring a classic editor, rather than a VSCode extension-based editor. This is because we're using a Monarch grammar, which is not supported by the extension configuration. However, if we wanted to use a TextMate grammar, we could use the extension based configuration instead. -Lastly, let's put together the service config for our wrapper. Like the name suggests, this indicates what services should be enabled. It also influences whether we use the VSCode API config (with TextMate grammars) or the classic editor config (with Monarch grammars). - -```js -const serviceConfig = { - // the theme service & textmate services are dependent, they need to both be enabled or disabled together - // this explicitly disables the Monarch capabilities - // both are tied to whether we are using the VSCode config as well - enableThemeService: false, - enableTextmateService: false, - - enableModelService: true, - configureEditorOrViewsServiceConfig: { - enableViewsService: false, - useDefaultOpenEditorFunction: true - }, - configureConfigurationServiceConfig: { - defaultWorkspaceUri: '/tmp/' - }, - enableKeybindingsService: true, - enableLanguagesService: true, - // if you want debugging facilities, keep this on - debugLogging: true -}; +```json +editorAppConfig: { + $type: 'vscodeApi', + languageId: id, + useDiffEditor: false, + code: config.code, + ... +} ``` -Now, getting back to building our configuration. We have built all the parts we needed to make this work, so let's put them all together and start the wrapper. +You would just need to fill in the rest of the details for associating a TextMate grammar & such. [Here's an example from the monaco-components repo](https://github.com/TypeFox/monaco-components/blob/4f301445eca943b9775166704304637cf5e8bd00/packages/examples/src/langium/config/wrapperLangiumVscode.ts#L37). -```js -// create a client wrapper -const client = new MonacoEditorLanguageClientWrapper(); -// start the editor -// keep a reference to a promise for when the editor is finished starting, we'll use this to setup the canvas on load -const startingPromise = client.start({ - htmlElement: document.getElementById("monaco-editor-root"), - wrapperConfig: { - // setting this to false disables using the VSCode config, and instead favors - // the monaco editor config (classic editor) - useVscodeConfig: false, - serviceConfig, - // Editor config (classic) (for Monarch) - monacoEditorConfig: { - languageExtensionConfig: { id: languageId }, - languageDef: monarchGrammar - } - }, - editorConfig, - languageClientConfig -}); +Regardless of how the user config is setup, we can now invoke that helper function with a handful of configuration details, and have a working UserConfig to pass to the wrapper. + +```ts +// create a wrapper instance +const wrapper = new MonacoEditorLanguageClientWrapper(); + +// start up with a user config +await wrapper.start(createUserConfig({ + htmlElement: document.getElementById("monaco-editor-root")!, + languageId: 'minilogo', + code: getMainCode(), + worker: getWorker(), + monarchGrammar: getMonarchGrammar() +})); ``` That's it! Now if everything was configured correctly, we should have a valid wrapper that will display the code we want in our browser. -*Note the `startingPromise` that's returned from `startEditor`. We're not using this yet, but it will be important for our setup in the next tutorial.* +## Serving via NodeJS + +Now that we have our files all setup, and our build process prepared, we can put together a mini server application to make viewing our public assets easy. We'll do this by adding **src/web/app.ts** to our project, and giving it the following contents: + +```ts +/** + * Simple server app for serving generated examples locally + * Based on: https://developer.mozilla.org/en-US/docs/Learn/Server-side/Node_server_without_framework + */ +import * as fs from "node:fs"; +import * as http from "node:http"; +import * as path from "node:path"; + +const port = 3000; -## Serving via Express +const MIME_TYPES: Record = { + default: "application/octet-stream", + html: "text/html; charset=UTF-8", + js: "application/javascript", + css: "text/css", +}; + +const STATIC_PATH = path.join(process.cwd(), "./public"); + +const toBool = [() => true, () => false]; + +const prepareFile = async (url: string) => { + const paths = [STATIC_PATH, url]; + if (url.endsWith("/")) { + paths.push("index.html"); + } + const filePath = path.join(...paths); + const pathTraversal = !filePath.startsWith(STATIC_PATH); + const exists = await fs.promises.access(filePath).then(...toBool); + const found = !pathTraversal && exists; + // there's no 404, just redirect to index.html in all other cases + const streamPath = found ? filePath : STATIC_PATH + "/index.html"; + const ext = path.extname(streamPath).substring(1).toLowerCase(); + const stream = fs.createReadStream(streamPath); + return { found, ext, stream }; +}; + +http + .createServer(async (req, res) => { + const file = await prepareFile(req.url!); + const statusCode = file.found ? 200 : 404; + const mimeType: string = MIME_TYPES[file.ext] || MIME_TYPES.default; + res.writeHead(statusCode, { "Content-Type": mimeType }); + file.stream.pipe(res); + console.log(`${req.method} ${req.url} ${statusCode}`); + }) + .listen(port); + +console.log(`Server for MiniLogo assets listening on http://localhost:${port}`); +``` -Now that we have our files all setup, and our build process prepared, we can put together a mini express application to make viewing our public assets easy. We'll do this by adding **src/web/app.ts** to our project, and giving it the following contents: +If you would like to compact this, and don't mind adding additional deps to your project, you can include `express` and `@types/express` to your project, and use the following code instead: ```ts /** - * Simple app for serving generated examples + * Simple express app for serving generated examples */ import express from 'express'; @@ -407,7 +598,7 @@ console.log(`Server for MiniLogo assets listening on http://localhost:${port}`); }); ``` -And to invoke express, we need to add one more script to our package.json. +And to invoke the server, we need to add one more script to our package.json. ```json { @@ -423,6 +614,6 @@ npm run build:web npm run serve ``` -You should be greeted with a page that contains a working Monaco instance and a small MiniLogo program in the editor. This editor has the highlighting we would expect, and also is fully connected to the language server for our language. This means we have full LSP support for operations that we would expect to have in a desktop editor. +You should be greeted with a page that contains a working Monaco instance and a small MiniLogo program in the editor. This editor has the highlighting we would expect, and also is fully connected to the language server for our language. This means we have full LSP support for operations that we would expect to have in a native IDE, such as VSCode. And that's it, we have successfully implemented Langium + Monaco in the web for our language. It's not doing much at this time besides presenting us with an editor, but in the next tutorial we'll talk about [using the same setup to add generation in the web](/tutorials/generation_in_the_web). Since our generation has already been configured natively in prior tutorials, we can use what we've written to quickly implement a web application that translates MiniLogo programs into drawing instructions for an HTML5 canvas.