WASM: wire it up! #33

New Issue

I've successfully generated a .wasm bundle; I haven't wired it up to JS yet; it may not work. But it works as expected when compiled to native code on my Mac.

Okay. I figured out how to do this, more or less. Zig is great, but its build system is very powerful, very complicated and seriously under-documented. (See, e.g., https://ziggit.dev/t/building-wasm-with-dependencies/4362.) I've successfully generated a `.wasm` bundle; I haven't wired it up to JS yet; it may not work. But it works as expected when compiled to native code on my Mac.

scott self-assigned this 2024-05-18 21:48:26 +00:00

scott added a new dependency 2024-05-19 19:33:40 +00:00

#31 Stdin, out, err

scott added a new dependency 2024-05-19 19:33:46 +00:00

#30 Coroutines!

scott commented

2024-05-22 16:03:39 +00:00

Just to document here where I am and what I've done:

Current strategy has been to use Zig as the host language for WASM; Zig can compile C and has WASM as a first-class target.
Compiling a Zig WASM blob only compiles Zig files, so bringing jzignet in as a dependency only ever compiled the Zig, even with janet.c and janet.h as explicit dependencies.
Before I figured this out, jzignet's C dependencies were being requested as JS environment functions, so Node threw errors like, "janet_unwrap_string must be callable." I added all the "callable" functions to the JS/WASM environment and indeed, they were dutifully called. But notably not Janet's c-functions.
Also, in a charming move, dependencies actually require the correct target.
So, I put everything in the same repo: jzignet.zig, janet.h, janet.c, and compiled everything with a wasm32-freestanding (or, for that matter, wasm32-wasi) target.

Here's what I get:

$ zig build
install
└─ install wasm
   └─ zig build-exe wasm ReleaseSafe wasm32-wasi 2 errors
/opt/homebrew/Cellar/zig/0.12.0/lib/zig/libc/include/generic-musl/setjmp.h:10:10: error: 'bits/setjmp.h' file not found
#include <bits/setjmp.h>
         ^~~~~~~~~~~~~~~~
src/core/features.h:81:10: note: in file included from src/core/features.h:81:
/Users/scott/git/zanet/resources/janet.h:437:10: note: in file included from /Users/scott/git/zanet/resources/janet.h:437:
#include <setjmp.h>
         ^
resources/janet.c:1:1: error: expected type expression, found '/'
/* Amalgamated build - DO NOT EDIT */
^

So: Janet's interpreter depends on setjmp.h, apparently? And WASM doesn't like this. Apparently it's some kind of C-level iteration of jmp or longjmp assembly instructions which WASM doesn't have.

And thus: it looks like Zig's WASM integration won't/can't compile Janet.

Emscripten, however, will. That does mean using C/C++ instead of Zig as the host language.

So two paths forward at this point:

See if we can get Zig to do what we need it to.
Use Emscripten.

(1) is preferable in the long run; (2) is perhaps more tractable. IanTheHenry's https://toodle.studio/ uses Emscripten, and I think we can hack what's in that repo down to manageable size by deleting everything, getting a minimal compile, and then adding things back in until we have what we need for Ludus.

Just to document here where I am and what I've done: * Current strategy has been to use Zig as the host language for WASM; Zig can compile C and has WASM as a first-class target. * Compiling a Zig WASM blob only compiles Zig files, so bringing `jzignet` in as a dependency only ever compiled the Zig, even with `janet.c` and `janet.h` as explicit dependencies. * Before I figured this out, `jzignet`'s C dependencies were being requested as JS environment functions, so Node threw errors like, "`janet_unwrap_string` must be callable." I added all the "callable" functions to the JS/WASM environment and indeed, they were dutifully called. But notably *not* Janet's c-functions. * Also, in a charming move, dependencies actually require the correct target. * So, I put everything in the same repo: `jzignet.zig`, `janet.h`, `janet.c`, and compiled everything with a `wasm32-freestanding` (or, for that matter, `wasm32-wasi`) target. Here's what I get: ``` $ zig build install └─ install wasm └─ zig build-exe wasm ReleaseSafe wasm32-wasi 2 errors /opt/homebrew/Cellar/zig/0.12.0/lib/zig/libc/include/generic-musl/setjmp.h:10:10: error: 'bits/setjmp.h' file not found #include <bits/setjmp.h> ^~~~~~~~~~~~~~~~ src/core/features.h:81:10: note: in file included from src/core/features.h:81: /Users/scott/git/zanet/resources/janet.h:437:10: note: in file included from /Users/scott/git/zanet/resources/janet.h:437: #include <setjmp.h> ^ resources/janet.c:1:1: error: expected type expression, found '/' /* Amalgamated build - DO NOT EDIT */ ^ ``` So: Janet's interpreter depends on `setjmp.h`, apparently? And WASM doesn't like this. Apparently it's some kind of C-level iteration of `jmp` or `longjmp` assembly instructions which WASM doesn't have. And thus: it looks like Zig's WASM integration won't/can't compile Janet. Emscripten, however, will. That does mean using C/C++ instead of Zig as the host language. So two paths forward at this point: 1. See if we can get Zig to do what we need it to. 2. Use Emscripten. (1) is preferable in the long run; (2) is perhaps more tractable. IanTheHenry's https://toodle.studio/ uses Emscripten, and I think we can hack what's in that repo down to manageable size by deleting everything, getting a minimal compile, and then adding things back in until we have what we need for Ludus.

scott commented

2024-05-22 16:23:06 +00:00

A more direct way to arrive at the error above:

$ zig cc janet.c -target wasm32-wasi
In file included from src/core/features.h:81:
In file included from ./janet.h:437:
/opt/homebrew/Cellar/zig/0.12.0/lib/zig/libc/include/generic-musl/setjmp.h:10:10: fatal error: 'bits/setjmp.h' file not found
   10 | #include <bits/setjmp.h>
      |          ^~~~~~~~~~~~~~~
LLD Link... 1 error generated.

A more direct way to arrive at the error above: ``` $ zig cc janet.c -target wasm32-wasi In file included from src/core/features.h:81: In file included from ./janet.h:437: /opt/homebrew/Cellar/zig/0.12.0/lib/zig/libc/include/generic-musl/setjmp.h:10:10: fatal error: 'bits/setjmp.h' file not found 10 | #include <bits/setjmp.h> | ^~~~~~~~~~~~~~~ LLD Link... 1 error generated. ```

matt commented

2024-05-22 21:35:35 +00:00

Found another interesting example using emscripten: https://github.com/sogaiu/jaylib-wasm-demo/blob/master/main.c

scott commented

2024-05-22 23:37:59 +00:00

Taking notes on how to wire up a Janet+WASM environment, in broad strokes (repos with various stages forthcoming). Note that this allows passing Javascript strings into Janet and getting them back out without too much fuss.

Create your Janet program. These are all top-level bindings. No main or anything, just the bindings you want & need.
Compile the program with janet -c source.janet target.jimage
In the CPP driver file (ganked from IanTheHenry's Bauble/Toodle), you will need to accomplish the following:
- Create a static null pointer to a JanetFunction with a reasonable name
- Then, create a CPP function that calls the Janet function, with the appropriate arguments and return types
- Next, in the main function of the driver, read in and marshal the environment in question, that contains your Janet function
- Then, point the original JanetFunction pointer to the actual Janet function in your environment
- Make sure to add the function you want to keep around as a GC root in Janet, so it doesn't get collected
- And, finally, using Emscripten bindings, add the function with a name, a pointer, and allow_raw_pointers
Build the CPP file using Emscripten. Current invocation is:

emcc \
    # destination JS file; .mjs means you can us ES6 in Node without ceremony
    -o out.mjs \ 
    # you need both the Janet interpreter c file and the driver cpp file
    janet.c driver.cpp \ 
    # embed the jimage, give it a reasonable, identical name
    --embed-file image.jimage@image.jimage \
    # export the emscripten bindings so you can call wrapped Janet functions from JS
    -lembind \
    # export the main function from your cpp file
    -s "EXPORTED_FUNCTIONS=['_main']" \
    # ???
    -s MODULARIZE \
    # make it an ES6 module
    -s EXPORT_ES6

Then, you make a JS file that does the thing:

import init from "out.mjs" // init will be bound to the default export
const mod = await init() // runs the wasm, must be awaited
mod.hello_world() // now you can run the Janet function

Taking notes on how to wire up a Janet+WASM environment, in broad strokes (repos with various stages forthcoming). Note that this allows passing Javascript strings into Janet and getting them back out without too much fuss. * Create your Janet program. These are all top-level bindings. No main or anything, just the bindings you want & need. * Compile the program with `janet -c source.janet target.jimage` * In the CPP driver file (ganked from IanTheHenry's Bauble/Toodle), you will need to accomplish the following: - Create a static null pointer to a JanetFunction with a reasonable name - Then, create a CPP function that calls the Janet function, with the appropriate arguments and return types - Next, in the `main` function of the driver, read in and marshal the environment in question, that contains your Janet function - Then, point the original JanetFunction pointer to the actual Janet function in your environment - Make sure to add the function you want to keep around as a GC root in Janet, so it doesn't get collected - And, finally, using Emscripten bindings, add the function with a name, a pointer, and `allow_raw_pointers` * Build the CPP file using Emscripten. Current invocation is: ```shell emcc \ # destination JS file; .mjs means you can us ES6 in Node without ceremony -o out.mjs \ # you need both the Janet interpreter c file and the driver cpp file janet.c driver.cpp \ # embed the jimage, give it a reasonable, identical name --embed-file image.jimage@image.jimage \ # export the emscripten bindings so you can call wrapped Janet functions from JS -lembind \ # export the main function from your cpp file -s "EXPORTED_FUNCTIONS=['_main']" \ # ??? -s MODULARIZE \ # make it an ES6 module -s EXPORT_ES6 ``` * Then, you make a JS file that does the thing: ```javascript import init from "out.mjs" // init will be bound to the default export const mod = await init() // runs the wasm, must be awaited mod.hello_world() // now you can run the Janet function ```

scott commented

2024-05-23 00:33:26 +00:00

Added to the above: getting strings from Janet -> C++ -> JS is not trivial, either. But it's doable!

There may well be a more direct way of going about things, but this is what I got that works:

struct StringResult {
  string value;
};

StringResult string_result(const char* cstr) {
  return (StringResult) {.value = (string) cstr };
}

StringResult yo(string whom) {
  Janet result;
  const Janet args[1] = {janet_cstringv(whom.c_str())};
  call_fn(janet_yo, 1, args, &result);
  const char* cstr = janet_getcstring(&result, 0);  
  return string_result(cstr);
}

// And then, in the emcc bindings:

value_object<StringResult>("StringResult")
  .field("value", &StringResult::value);

In JS, then, simply access .value on any expected string result.

Added to the above: getting strings from Janet -> C++ -> JS is not trivial, either. But it's doable! There may well be a more direct way of going about things, but this is what I got that works: ```cpp struct StringResult { string value; }; StringResult string_result(const char* cstr) { return (StringResult) {.value = (string) cstr }; } StringResult yo(string whom) { Janet result; const Janet args[1] = {janet_cstringv(whom.c_str())}; call_fn(janet_yo, 1, args, &result); const char* cstr = janet_getcstring(&result, 0); return string_result(cstr); } // And then, in the emcc bindings: value_object<StringResult>("StringResult") .field("value", &StringResult::value); ``` In JS, then, simply access `.value` on any expected string result.

scott commented

2024-05-23 00:36:59 +00:00

Next up: make a repo with a standalone POC.

scott commented

2024-05-23 01:00:29 +00:00

It is all up in https://alea.ludus.dev/twc/emscranet

scott commented

2024-06-06 23:15:53 +00:00

Current state of play:

It works, but it's extremely titchy.
Current build cannot be done from the root of the project. cd into build and then just build.
Build artifacts are (somehow again this matters) built into the build directory, and cannot be sent somewhere else.
package.json is now updated to point to the new wasp-based interpreter.
Current version is now published to npm under 0.1.0-alpha.10.

Current state of play: * It works, but it's extremely titchy. * Current build cannot be done from the root of the project. `cd` into `build` and then `just build`. * Build artifacts are (somehow again this matters) built into the `build` directory, and cannot be sent somewhere else. * `package.json` is now updated to point to the new wasp-based interpreter. * Current version is now published to npm under `0.1.0-alpha.10`.

scott removed a dependency 2024-06-06 23:16:01 +00:00

#30 Coroutines!

scott removed a dependency 2024-06-06 23:16:05 +00:00

#31 Stdin, out, err

scott closed this issue

2024-06-06 23:16:14 +00:00

Sign in to join this conversation.