Support for alternative JavaScript runtimes

[JoshMock]

While Node.js is by far the most popular server-side JavaScript runtime, there have been a few alternative runtimes, or Node.js environments with subsets of the standard library available for use, that have started to gain traction. A few notable ones:

• Deno
• Bun
• Cloudflare workers
• Vercel edge functions

We should test how well the client runs in the most popular of these runtimes and establish what would need to be done to make the library compatible with each. And, for the environments where the effort is reasonable, we should do what we can to support compatibility.

A few related issues:

• Decouple elasticsearch-js from the Node.js core APIs #1742
• Support Cloudflare Workers / Vercel Edge Functions #2032
• Provide an ES Module entrypoint for @elastic/elasticsearch #1743
• Support For Deno? #1680

[JoshMock]

Initial report on Deno support

There are several barriers to getting the same codebase to run on Deno:

• Deno's import syntax for loading modules from relative file paths and Node's standard library is not identical to TypeScript's. The --unstable-sloppy-imports flag can be used to ease the pain of this, but it is not an ideal solution because it requires our users to also depend on that flag.
• Importing third-party dependencies from npm requires a Deno-specific specifier that does not work in the Node.js runtime.
• Deno does not provide full support for the Node.js standard library.
• For all of the above reasons, third-party dependencies are also not guaranteed to run on Deno.

After multiple days of work, exploring several avenues, I have not been able to run a Deno-compatible version of the client against our integration or unit test suites. Not to say that it is impossible, just that it's going to be a decent amount of work.

The best solution I have come up with so far:

• Use a script that transpiles Node.js code to Deno. There does not seem to be any available tool that already does this well, so we would have to build and maintain our own. It's not impossible, but it does introduce some complexity.
• Maintain a separate codebase for Deno (and perhaps other runtimes) and update our code generation to support outputting to both Node.js and Deno runtimes.

Both seem like viable options, but depending on the demand for Deno support, the effort may not yet be worth it. More discussion is necessary. cc @srikanthmanvi

More evaluations on other runtimes to come.

[JoshMock]

Initial report on Cloudflare worker support

Several Node.js standard library modules that the client or its dependencies use are not available at runtime:

• timers/promises
• worker_threads
• async_hooks
• diagnostics_channel
• perf_hooks
• util/types
• http2

Importing sub-paths from an npm package (e.g. @elastic/transport/package.json, @elastic/transport/lib/connection/BaseConnection) will require listing all sub-paths in the exports section of the package.json file.

Not done researching, but these are significant blockers right off the bat, because many of these things depend on upstream support from libraries this client depends on.

[JoshMock]

Initial report on Vercel edge functions

Vercel functions do already provide support for the full Node.js API. So there is a path to using this client on Vercel. What this report will cover, then, is whether this client can be run inside their Edge runtime, which is slimmer and more cost-effective. Since the Edge runtime is recommended for querying databases or other external services with slower request times, it would be ideal if this client could run there.

As of this writing, Vercel's docs note that they only support the following Node.js modules:

• async_hooks
• events
• buffer
• assert
• util

This leaves out several hard dependencies that this client and its transport depend on, so the path to supporting the Vercel Edge runtime would require a significant refactor.

[JoshMock]

Initial report on Bun

Since Bun supports the Node.js standard library more broadly, it appears to run the client just fine out of the box with zero modifications. I've added an action step to run our unit test suite against the Bun runtime in #2353.

That said, I don't want to advertise we're officially Bun-compatible quite yet, only to avoid creating a new category of support issues, until we're more confident and have seen it running well in some real-world use cases.

[JoshMock]

Conclusions

Right now, the amount of work to make this client broadly interoperable with other JS runtimes is non-trivial. More work needs to be done to understand which runtimes are gaining popularity in the Elasticsearch community, and when supporting them is going to become a necessity.

When the time comes, the most critical point to understand is that almost all of these runtimes are prioritizing support for the Fetch API—specifically the WinterCG fork—rather than Node.js's standard library http and https modules. It may be worthwhile to investigate if it would be feasible to refactor the client layer and/or provide an alternate transport layer that depends only on fetch and other web platform APIs. Some clients for other databases (Supabase, Planetscale, etc.) that support HTTP have chosen to depend on fetch rather than the Node.js standard library.

Maybe we need a browser-friendly client?

Another consideration is that it may be worth exploring the possibility of a separate JS client that specifically supports fetch and other web-standard APIs. As an example, Mongo's Atlas platform provides separate clients for Node.js and web, the latter of which uses fetch and is designed to run in a browser, but also supports many alternative runtimes as a result.

We have had many users come to us with questions, confusion and requests related to browser support over the years.¹ While the Search UI project was perhaps a partial solution to that, it has not been maintained, nor does does it expose the entire Elasticsearch API surface. This leads me to question whether we are missing an opportunity by not providing a browser option. The biggest concern there is security: any credentials embedded in a browser script are visible to the world, so these creds need to have very limited power. And there are likely some APIs that are either inherently unsafe to expose in a browser environment, or give too much power to the client to flood the server with slow requests.

There are a lot of considerations that would need to be taken into consideration here. I know @joemcelroy and other Search team engineers have worked on Search UI in the past, and they may have some thoughts.

9.0?

With Elasticsearch 9.0 coming up, we have the opportunity to introduce breaking changes to this client. This might be a good time to consider what changes, breaking or not, could be made to set us up to broaden our JavaScript support far beyond Node.js.

Footnotes

1. • https://github.com/elastic/elasticsearch-js/issues/905
   • https://github.com/elastic/elasticsearch-js/issues/1066
   • https://github.com/elastic/elasticsearch-js/issues/1317
   • https://github.com/elastic/elasticsearch-js/issues/1345
   • https://github.com/elastic/elasticsearch-js/issues/1397
   • https://github.com/elastic/elasticsearch-js/issues/1406
   • https://github.com/elastic/elasticsearch-js/issues/1449
   • https://github.com/elastic/elasticsearch-js/issues/1456
   • https://github.com/elastic/elasticsearch-js/issues/1582
   • https://github.com/elastic/elasticsearch-js/issues/1603
   • https://github.com/elastic/elasticsearch-js/issues/1687
   ↩

[JoshMock]

@JasonStoltz is another Search UI friend who is still around and might have thoughts.