Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does not work in node environment (upstream v3.42) #16

Closed
chuanqisun opened this issue Jun 25, 2023 · 8 comments
Closed

Does not work in node environment (upstream v3.42) #16

chuanqisun opened this issue Jun 25, 2023 · 8 comments

Comments

@chuanqisun
Copy link

Hi, I'm trying to use sqlite-wasm in both browser and node environment. The node environment is used for unit test. I don't expect OPFS to work so I switched to in-memory DB instead. When I run the sample code, it looks like the library is still expecting browser environment and throws an error:

Loading and initializing SQLite3 module...
Process exited with code 1
Exception loading sqlite3 module: TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11457:11) {cause: Error: not implemented... yet...
    at makeNe…or (node:internal/deps/undici/undici:6802:35)…, stack: 'TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11457:11)', message: 'fetch failed'}
Uncaught TypeError TypeError: fetch failed
    at fetch (internal/deps/undici/undici:11457:11)

Here is the screenshot of the setup. My node version is 20.0.0

image

I noticed that 3.42 ships with a sqlite-node.mjs file. So I hard coded the import path to it:
image

And got a self error seemingly related to #3

sqlite3 bootstrap initializer threw: ReferenceError: self is not defined
    at StructBinderFactory (/workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:7859:20)
    at /workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:9321:43
    at /workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:6686:13
    at Array.forEach (<anonymous>)
    at sqlite3ApiBootstrap (/workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:6685:44)
    at /workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:13243:32
    at callRuntimeCallbacks (/workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:681:26)
    at postRun (/workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:459:7)
    at doRun (/workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:5642:9)
    at run (file:///workspaces/tinykb/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-node.mjs:5654:9) {stack: 'ReferenceError: self is not defined
    at St…m/sqlite-wasm/jswasm/sqlite3-node.mjs:5654:9)', message: 'self is not defined'}

I understand if there is no official support for node.js runtime due to lack of dev resource. I'd still like to propose such a feature as it allows test driven development. In the js ecosystem, most unit test setups require node.js based runtime. Also, wasm is supposed to be the universal binary format that is agnostic of browser vs. node.js (or even deno), so it would be amazing for sqlite to work anywhere wasm runs.

@sgbeal
Copy link
Collaborator

sgbeal commented Jun 25, 2023

i'll fix the dangling "self" references tomorrow (it's late here). We don't officially/directly support node for the very simple reason that nobody in the sqlite project uses node in any capacity, and attempting to support a platform you don't actually use is a lost cause.

@sgbeal sgbeal closed this as completed Jun 26, 2023
@sgbeal
Copy link
Collaborator

sgbeal commented Jun 26, 2023

(Oooops - just deleted the closure comment because github showed it as duplicately posted when it wasn't. Rewriting...)

The "self" references have been replaced with "globalThis" upstream. That change will appear in the next 3.42.x and 3.43 releases.

As to fetch(), that comes from Emscripten-generated code. The sqlite3-node.mjs build variant blocks out that call when run from node:

  if (!binary &&
      typeof WebAssembly.instantiateStreaming == 'function' &&
      !isDataURI(binaryFile) &&
      // Avoid instantiateStreaming() on Node.js environment for now, as while
      // Node.js v18.1.0 implements it, it does not have a full fetch()
      // implementation yet.
      //
      // Reference:
      //   https://github.com/emscripten-core/emscripten/pull/16917
      !ENVIRONMENT_IS_NODE &&
      typeof fetch == 'function') {
    return fetch(binaryFile, { credentials: 'same-origin' }).then((response) => {

@irisjae
Copy link

irisjae commented May 9, 2024

I'm still seeing the use of self and the associated error on 3.45.3-build1; what's the plan for merging this?

@sgbeal
Copy link
Collaborator

sgbeal commented May 9, 2024

what's the plan for merging this?

Merging what, specifically? The code snippet shown above is from Emscripten-generated code which we have on influence over. There is precisely one "self" use in the current code, but it also comes from Emscripten and there's a relevant comment with it:

// Note that this includes Node.js workers when relevant (pthreads is enabled).
// Node.js workers are detected as a combination of ENVIRONMENT_IS_WORKER and
// ENVIRONMENT_IS_NODE.
if (ENVIRONMENT_IS_WEB || ENVIRONMENT_IS_WORKER) {
  if (ENVIRONMENT_IS_WORKER) { // Check worker, not web, since window could be polyfilled
    scriptDirectory = self.location.href;
  }
...
}

If your issue is that "self" use, please file a ticket upstream with Emscripten.

@irisjae
Copy link

irisjae commented May 9, 2024

(Oooops - just deleted the closure comment because github showed it as duplicately posted when it wasn't. Rewriting...)

The "self" references have been replaced with "globalThis" upstream. That change will appear in the next 3.42.x and 3.43 releases.

As to fetch(), that comes from Emscripten-generated code. The sqlite3-node.mjs build variant blocks out that call when run from node:

  if (!binary &&
      typeof WebAssembly.instantiateStreaming == 'function' &&
      !isDataURI(binaryFile) &&
      // Avoid instantiateStreaming() on Node.js environment for now, as while
      // Node.js v18.1.0 implements it, it does not have a full fetch()
      // implementation yet.
      //
      // Reference:
      //   https://github.com/emscripten-core/emscripten/pull/16917
      !ENVIRONMENT_IS_NODE &&
      typeof fetch == 'function') {
    return fetch(binaryFile, { credentials: 'same-origin' }).then((response) => {

Ah, thanks, I see. I interpreted your comment as saying that self had already been replaced in 3.42.x/3.43.x.

If this doesn't yet work in node, perhaps we can leave this issue open?

@sgbeal
Copy link
Collaborator

sgbeal commented May 9, 2024

I interpreted your comment as saying that self had already been replaced in 3.42.x/3.43.x.

The ones in our code have all bee changed - we're "self-free" since (IIRC) that change. We have no influence over the Emscripten-generated bits, though.

perhaps we can leave this issue open?

We don't specifically target node, and having this ticket open won't change that ;).

Nobody in the sqlite project uses node in any capacity except behind-the-scenes in the Emscripten build process, and claiming/attempting to support a platform we don't use would go poorly for us.

@kumavis
Copy link

kumavis commented Nov 8, 2024

heres what i needed to do to modify the node environment to get it to run

async function fetchFile(uri: string, mimeType: string = 'text/plain;charset=UTF-8'): Promise<Response> {
  if (uri.startsWith('file://')) {
    try {
      const path = new URL(uri).pathname;
      const data = await fs.promises.readFile(path);
      return new Response(data, { status: 200, statusText: 'OK', headers: { 'Content-Type': mimeType } });
    } catch (error) {
      console.error('Error reading file:', error);
      return new Response(null, { status: 404, statusText: 'File Not Found' });
    }
  }
  return new Response(null, { status: 500, statusText: 'Only file:// supported' });
}

(globalThis as any).self = globalThis;
const globalFetch = globalThis.fetch;
const wrappedFetchForSqliteWasm = async (url: string, options: RequestInit = {}): Promise<Response> => {
  if (url.endsWith('@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3.wasm')) {
    return fetchFile(url, 'application/wasm');
  }
  console.log(`fetching ${url}`);
  return globalFetch(url, options);
};
(globalThis as any).fetch = wrappedFetchForSqliteWasm;

@tomayac
Copy link
Collaborator

tomayac commented Nov 8, 2024

Did you see #48? Does your solution allow persisting the database?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants