WASM packaging should be possible #34

New Issue

scott · 2025-06-26T00:20:33Z

scott commented

2025-06-26 00:20:33 +00:00

@matt I fucking hate "modern Javascript." It promises things and is cruel when in the very subtle ways it doesn't deliver them.

Let me take you on a walk with me.

The tool I'm using to bundle up Rust is wasm-bindgen. You know we're in a world of hurt when it has multiple build targets.

For our purposes, the ones we're interested in are: web, nodejs, and bundler.

Are we `web` yet?

In an ideal world, the most straightforward version of this would be web. It produces a browser-ready ESM. Under the hood, it uses fetch to pull the wasm blob down. This, I think, maybe, could work if we do it out-of-band in terms of svelte & vite. I see you have p5.js-svg pulled down in a script tag in app.html. Perhaps we could do something similar with the wasm blob and just write some code to load the relevant functions on a global object.

Given that this is literally just a function that runs and nothing more exotic, this seems like a lot of weird, out-of-band ceremony. I'm worried about what happens when we need to do webworkers. Also, I have no idea how to automate a build here; we could download the files, I suppose. Or even use unpkg in place of hosting the thing ourselves. Thoughts?

Node says nope

Or, computer says no

So the next most obvious thing is to pack it up as a nodejs module. This plays nice with npm! And indeed, this works, too, and I can npm install @ludus/rudus just fine. I don't love it, since it's got more "building" going on to go from node modules to ESM, but ok.

The problem, here, is vite. Just dropping it in, we get a Type Error: TextEncoder is not a constructor error. From the best of my ability to determine, this is an issue in how vite handles platform functions. This issue gives some idea of what might be going on: https://github.com/vitejs/vite/discussions/12826. The solution there gave me a different error, which pointed me to this page in vite's documents: https://vite.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility.

I didn't follow that rabbit hole further. I could, I suppose.

By this point, I'm a `bundle` of nerves

So the last thing I tried was wasm-bindgen's most minimal build target: the bundler target. But bundler is very specifically targeted at webpack. But I thought, hey, okay, alright, we'll bundle the js on this side, too. But in this instance, they emit code that imports the wasm blob directly, which is... not a thing that is valid js. So we're stuck, then, with using webpack to try to gather things up. Like wasm-bindgen, the conceptual framing in webpack is that the thing that will eventually run the wasm is probably a web browser. I started playing with this, but didn't get too far. If we're going to use the browser, let's just use the web target. Webpack does have a node target, but it felt entirely too stupid to be webpacking things up only to then unpack them and repackage them with vite, and I had no hope that this would actually help.

What are we doing anyway?

wasm-bindgen does a few different things. The thing that's causing all the grief here is actually about how we load a wasm blob. The web version does it over the network, using fetch. The nodejs version does it locally, using node::fs functions. There's a deno version that does similar. The bundle packages it up according to some arcane js bundler magic.

I can't tell which the easiest path is going to be here. I need your input.

Also, fuck javascript.

@matt I fucking hate "modern Javascript." It promises things and is cruel when in the very subtle ways it doesn't deliver them. Let me take you on a walk with me. The tool I'm using to bundle up Rust is [wasm-bindgen](https://rustwasm.github.io/wasm-bindgen/). You know we're in a world of hurt when it has multiple build targets. For our purposes, the ones we're interested in are: `web`, `nodejs`, and `bundler`. ### Are we `web` yet? In an ideal world, the most straightforward version of this would be `web`. It produces a browser-ready ESM. Under the hood, it uses `fetch` to pull the wasm blob down. This, I think, maybe, could work if we do it out-of-band in terms of svelte & vite. I see you have `p5.js-svg` pulled down in a `script` tag in `app.html`. Perhaps we could do something similar with the wasm blob and just write some code to load the relevant functions on a global object. Given that this is literally just a function that runs and nothing more exotic, this seems like a lot of weird, out-of-band ceremony. I'm worried about what happens when we need to do webworkers. Also, I have no idea how to automate a build here; we could download the files, I suppose. Or even use unpkg in place of hosting the thing ourselves. Thoughts? ### Node says nope #### Or, computer says no So the next most obvious thing is to pack it up as a nodejs module. This plays nice with npm! And indeed, this works, too, and I can `npm install @ludus/rudus` just fine. I don't love it, since it's got more "building" going on to go from node modules to ESM, but ok. The problem, here, is vite. Just dropping it in, we get a `Type Error: TextEncoder is not a constructor` error. From the best of my ability to determine, this is an issue in how vite handles platform functions. This issue gives some idea of what might be going on: https://github.com/vitejs/vite/discussions/12826. The solution there gave me a different error, which pointed me to this page in vite's documents: https://vite.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility. I didn't follow that rabbit hole further. I could, I suppose. ### By this point, I'm a `bundle` of nerves So the last thing I tried was wasm-bindgen's most minimal build target: the `bundler` target. But `bundler` is very specifically targeted at webpack. But I thought, hey, okay, alright, we'll bundle the js on this side, too. But in this instance, they emit code that `import`s the wasm blob directly, which is... not a thing that is valid js. So we're stuck, then, with using webpack to try to gather things up. Like wasm-bindgen, the conceptual framing in webpack is that the thing that will eventually run the wasm is probably a web browser. I started playing with this, but didn't get too far. If we're going to use the browser, let's just use the `web` target. Webpack does have a `node` target, but it felt entirely too stupid to be webpacking things up only to then unpack them and repackage them with vite, and I had no hope that this would actually help. ### What are we doing anyway? wasm-bindgen does a few different things. The thing that's causing all the grief here is actually about how we load a wasm blob. The web version does it over the network, using `fetch`. The nodejs version does it locally, using `node::fs` functions. There's a deno version that does similar. The bundle packages it up according to some arcane js bundler magic. I can't tell which the easiest path is going to be here. I need your input. Also, fuck javascript.

scott self-assigned this 2025-06-26 00:20:33 +00:00

matt was assigned by scott

2025-06-26 00:20:33 +00:00

scott commented

2025-06-26 02:25:08 +00:00

Oh, I should add that web workers are going to add a whole other layer of complication with this, since node doesn't support web workers. So once again, we're tied to the web platform. I know that doesn't necessarily mean moving away from npm. I guess I just don't quite grok the model.

scott commented

2025-06-26 03:27:29 +00:00

Just leaving this here to communicate what all I have done.

I realized that one version of the thing that I could do, and probably ought to have tried before, is to use the web target and just throw the thing up on npm anyway. npm doesn't care the code it's bringing in runs on node or in the browser.

So I did that.

And I did test to make sure that it works in the browser, and yup, it sure does. In fact, it's so close to the Janet interpreter in this modality that the wrapper ludus.js file had maybe 3 lines changed in it?

BUT (and of course but): now svelte/vite is yelling about how fetch (a global browser function!) doesn't exist (which is what the web target uses to actually load up the wasm blob). I even tried to hard-code the uri for the script in the app.html file, but svelte so abstracts around actual web stuff that I couldn't tell what root the server serves stuff from.

Rather than go digging into the svelte documentation to figure that out, I yield. You have beaten me, js.

Just leaving this here to communicate what all I have done. I realized that one version of the thing that I could do, and probably ought to have tried before, is to use the `web` target and just throw the thing up on npm anyway. npm doesn't care the code it's bringing in runs on node or in the browser. So I did that. And I did test to make sure that it works in the browser, and yup, it sure does. In fact, it's so close to the Janet interpreter in this modality that the wrapper `ludus.js` file had maybe 3 lines changed in it? BUT (and of course but): now svelte/vite is yelling about how `fetch` (a global browser function!) doesn't exist (which is what the web target uses to actually load up the wasm blob). I even tried to hard-code the uri for the script in the `app.html` file, but svelte so abstracts around actual web stuff that I couldn't tell what root the server serves stuff from. Rather than go digging into the svelte documentation to figure that out, I yield. You have beaten me, js.

Sign in to join this conversation.