Document how to use from Node.js/browser

[rrthomas]

I would like to use Topiary (compiled to WASM) as a library in a Node.js project.

This should clearly be possible, but it's not obvious how to separate Topiary from e.g. the demo web app.

Thanks for Topiary! I was easily able to add a formatter for a new language.

[yannham]

Hello,

I think you should be able to use the demo app as an example, as the playground is nothing special in the end. Bootstrap a simple React app (or whatnot). Then you can just import Topiary functions exported as WASM:

topiary/web-playground/src/App.tsx

Lines 5 to 9 in b4a8426

	import init, {
	topiaryInit,
	queryInit,
	format,
	} from "./wasm-app/topiary_playground.js";

.

You still need to make this wasm-app stuff available somewhere. To do so, you can see that the CI just builds the crate topiary-playground - a simple wrapper around topiary-core to re-export the interesting bits to WASM - and copy the result to a new wasm-app directory in the playground's webapp:

topiary/.github/workflows/ci.yml

Lines 48 to 56 in b4a8426

	- name: Build web playground Wasm app
	if: success() && matrix.os == 'ubuntu-latest'
	run: nix build .#topiary-playground
	- name: Copy web playground Wasm app into playground frontend
	if: success() && matrix.os == 'ubuntu-latest'
	run: |
	mkdir -p web-playground/src/wasm-app
	cp -r result/* web-playground/src/wasm-app/

.

The nix build .#topiary-playground command does some magic in addition to compiling the topiary-playground. If you really don't want or can't use Nix, you can also do it manually or using a script by looking at what the corresponding Nix part does:

topiary/default.nix

Lines 139 to 163 in b4a8426

	topiary-playground = craneLibWasm.buildPackage (commonArgs
	// {
	inherit cargoArtifacts;
	pname = "topiary-playground";
	cargoExtraArgs = "-p topiary-playground --no-default-features --target ${wasmTarget}";
	# Tests currently need to be run via `cargo wasi` which
	# isn't packaged in nixpkgs yet...
	doCheck = false;
	postInstall = ''
	echo 'Removing unneeded dir'
	rm -rf $out/lib
	echo 'Running wasm-bindgen'
	wasm-bindgen --version
	wasm-bindgen --target web --out-dir $out target/wasm32-unknown-unknown/release/topiary_playground.wasm;
	echo 'Running wasm-opt'
	wasm-opt --version
	wasm-opt -Oz -o $out/output.wasm $out/topiary_playground_bg.wasm
	echo 'Overwriting topiary_playground_bg.wasm with the optimized file'
	mv $out/output.wasm $out/topiary_playground_bg.wasm
	echo 'Extracting custom build outputs'
	export LANGUAGES_EXPORT="$(ls -t target/wasm32-unknown-unknown/release/build/topiary-playground-*/out/languages_export.ts | head -1)"
	cp $LANGUAGES_EXPORT $out/
	'';

.

There's no particular hidden coupling, to the best of my knowledge, between Topiary and the web playground, so it should indeed be reasonably straightforward to use Topiary from your own web app!

[rrthomas]

Thanks, I should have been more specific: my actual use-case is that I want to use Topiary as a library from a Node.js program. So you're quite right: I could probably work out how to use it in a web app from the example. But I felt that it would take a lot of effort to work out how to use it as a Node.js library. But I'll take a look at your explanation and see if I can transpose what you say from browser to Node.

[yannham]

I'm not well versed in the Node ecosystem, but then I think the important bit is that nix build .#topiary-playground should compile Topiary to WASM and pack it as a standard Node package (put under the local directory result), that you should then be able to use as any other NPM dependency (sometimes there are some gotchas with respect to loading WASM modules, but it depends on your precise setup)

[rrthomas]

Not expecting you to do anything @yannham, just noting I tried a non-nix build (as I'd like ideally to have this as part of a plain npm build) and had the following problem:

$ cargo build -p topiary-playground --no-default-features --target wasm32-unknown-unknown
  cargo:warning=/home/rrt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tree-sitter-0.22.6/src/./alloc.h:9:10: fatal error: 'stdio.h' file not found
  cargo:warning=    9 | #include <stdio.h>
  cargo:warning=      |          ^~~~~~~~~
  cargo:warning=1 error generated.

In other words, while building tree-sitter, the C compiler couldn't find stdio.h, which makes sense, as wasm32-unknown-unknown is a "bare-metal" target; but I have no idea what I'm supposed to do about it, or why it works in the nix build.

[yannham]

It's strange. From what I recall, topiary-playground shouldn't be using tree-sitter directly but rather tree-sitter-facade, which role is precisely to offer a reduced interface which also works when compiling to WASM. The Nix build doesn't seem to be doing much magic beside providing wasm-bindgen, binaryen and pkg-config in the PATH. Maybe Topiary devs would have some idea about that? @Xophmeister

[Xophmeister]

I'm afraid I'm not familiar with whatever magic Nix/Crane performs to build WASM targets. If you can get it to build with Nix, as @yannham suggested, at least that would allow you to move forward regarding Topiary and Node.js (which we're only presuming "just works" at the moment).

[rrthomas]

Thanks, both. Getting this working is not a high priority for me at present (I am content to invoke topiary via the command line from my Node program), but I will want to get it working at some point.