Programmatically control the Flok Editor? (Discussion)

[tmhglnd]

Hi!

I have the idea to control one user in Flok programmatically via OSC-messages. For example I would like to send a command which letter to type, what to backspace, or where to move the cursor to.

I still have to take some time to dive deeper into the code to see what the possibilities are. This is not really a feature request, but more a question if you would be able to point me towards some parts of the code where this idea could be achieved, or some resources that could be helpful to achieve this goal. If not no worries, I will dive into it myself for sure and can report back my findings!

[munshkr]

Hi! I wondered about this too a couple of times, and I think someone else asked something similar...

Have you thought about using a test automation tool like PlayWright or Pupeteer?
Good thing about this is that you don't have to modify Flok, you control the browser programmatically instead, like a user would do.

https://playwright.dev/docs/input#type-characters
https://playwright.dev/docs/input#keys-and-shortcuts

Maybe some minor adjustments would have to be made on Flok, like adding id attributes to the <textarea> elements, to easily find them.

[tmhglnd]

Ah interesting suggestions! Will definitely look into those options. Maybe this is a quicker option than hacking Flok to have this feature indeed.

One thing that I do need to look into is how I can get the text of the editor out of Flok, without using the evaluate button. I would like to be able to poll the current content of the editor(s) to see what text is in there. I think for this I would probably need to go into the Flok code and add some feature that responds to an osc message and returns the editors text.

[munshkr]

One thing that I do need to look into is how I can get the text of the editor out of Flok, without using the evaluate button. I would like to be able to poll the current content of the editor(s) to see what text is in there. I think for this I would probably need to go into the Flok code and add some feature that responds to an osc message and returns the editors text.

Take a look at the usage of postMessageParentWindow:

• flok/packages/web/src/routes/session.tsx

  Lines 224 to 235 in c74eb1a

  	newSession.on("change", (documents) => {
  	setDocuments(documents);
  	postMessageParentWindow({
  	event: "change",
  	documents: documents.map((doc: Document) => ({
  	id: doc.id,
  	target: doc.target,
  	content: doc.content,
  	})),
  	});
  	});

• flok/packages/web/src/routes/session.tsx

  Lines 267 to 275 in c74eb1a

  	newSession.on("message", ({ message }) => {
  	setMessages((messages) => [...messages, message as Message]);
  	setMessagesCount((count) => count + 1);
  	postMessageParentWindow({
  	event: "message",
  	message,
  	});
  	});

• flok/packages/web/src/routes/session.tsx

  Lines 289 to 296 in c74eb1a

  	newSession.on("eval", ({ docId, body, user }) => {
  	postMessageParentWindow({
  	event: "eval",
  	id: docId,
  	content: body,
  	user,
  	});
  	});

Flok posts messages to the parent window on these kinds of events (document changes, evaluation, REPL messages), in case you embed Flok with an <iframe>. Maybe you can send these via OSC or WebSockets from there? Or, you could embed Flok and handle these messages directly from within the browser.

[tmhglnd]

Great, thanks for the pointers! Will give this a try

[tmhglnd]

I've been able to get it working to typ in Flok via the puppeteer package, very nice, thanks for the tip!

However now I'm a bit stuck because I'm trying to extend Flok to be able to receive the code, cursor position, window panel ID every time some text is changed or the cursor of the user is moved.

I've been able to succesfully print these things to the console by adding the following code in the editor.tsx under the Editor extensions:

// Listen for changes in the editor
EditorView.updateListener.of((e) => {
  // If the cursor position changed or
  // if the code in the editor is updated
  if (e.selectionSet || e.docChanged){
    // set the character number for the cursor and line number
    let atChar = e.state.selection.main.head;
    let atLine = e.state.doc.lineAt(atChar).number;
    let line = e.state.doc.lineAt(atChar).text;

    // get the document id, target and content
    let id = document.id;
    let target = document.target;
    let code = document.content;
    
    let msg = { event: 'message', id: id, target: target, atChar: atChar, atLine: atLine, line: line, code: code };
    
    // print to the console (and later hopefully OSC-message output)
    console.log(msg)
  } 
})

But now, how do I get this as an OSC-message from the editor to the flok REPL? The ideal situation would be that I just can send a new osc message and parse it via the REPL and forward it to the port that Mercury already uses, 4880. Any pointers would be greatly appreciated :) I looked briefly at the postMessageParentWindow as well, but since I am also looking to get the cursor location I decided it might be "easier" to just create some new OSC message from within Flok, but maybe I'm mistaken haha.

Update:

I'm trying the iframe with the postMessageParentWindow combination, but I'm not seeing anything posted when I create an eventlistener for "eval" or "change". It does work if I create a custom window.parent.postMessage(msg, '*') and a window.addEventListener("message", (m) => console.log(m)).