Language-independent testing #102

New Issue

scott · 2024-07-21T17:56:28Z

scott commented

2024-07-21 17:56:28 +00:00

We need a version of testing that is not tied to a specific Ludus implementation. This is because we'll need to test both a Janet-based tree-walk interpreter and a Zig-based bytecode interpreter. It would be wise not to have to duplicate testing efforts.

Two versions of this will exist:

(1) In-Ludus testing. This will be useful for a kind of integration testing. The Prelude should be extensively tested as the canonical place to ensure the language is behaving as expected. So we should be adding test exprs to Prelude extensively at some point.

(2) Shell-based testing. This will be useful for unit testing in the early phases of language development. The idea is that we will need some way of running Ludus scripts and capturing either outputs or error messages, and comparing them against expected values.

For both (1) and (2), we'll need some tooling. I'm seduced by Ian Henry's My Kind of REPL as a starting place. That way we write test cases, but we don't have to come up with the results of those cases.

More research is needed to figure out how to accomplish all of this.

We need a version of testing that is not tied to a specific Ludus implementation. This is because we'll need to test both a Janet-based tree-walk interpreter and a Zig-based bytecode interpreter. It would be wise not to have to duplicate testing efforts. Two versions of this will exist: (1) _In-Ludus testing_. This will be useful for a kind of integration testing. The Prelude should be extensively tested as the canonical place to ensure the language is behaving as expected. So we should be adding `test` exprs to Prelude extensively at some point. (2) _Shell-based testing_. This will be useful for unit testing in the early phases of language development. The idea is that we will need some way of running Ludus scripts and capturing either outputs or error messages, and comparing them against expected values. For both (1) and (2), we'll need some tooling. I'm seduced by Ian Henry's [My Kind of REPL](https://ianthehenry.com/posts/my-kind-of-repl/) as a starting place. That way we write test cases, but we don't have to come up with _the results of those cases_. More research is needed to figure out how to accomplish all of this.

scott added this to the Testing! project 2024-07-21 17:56:28 +00:00

scott added the

errors

infrastructure

labels 2024-07-21 17:56:45 +00:00

scott commented

2024-07-21 20:12:25 +00:00

I tried getting Janet to do this, but it won't work. Ian Henry's Judge's test-stdout only captures in-Janet stdout, not anything printed to the console. So (os-exec ["echo" "foobar"] :p), which will echo foobar to the console actually produces no output for Judge.

That suggests that the way to go about doing this is by using an environment that wraps Ludus (whether Janet– or Zig-based). There are two options for this:

shell
JS/Wasm

The tooling for Judge's "snapshot-based" testing in JS is absolutely more robust than shell-based testing, so that suggests JS-based testing.

A quick search in JS-testing-land shows that both Jest and Node-TAP have snapshot capabilities. Neither of these are inline. That said, I don't think we need inline snapshot testing for the language-agnostic version; we're not coding in JS, but executing code that JS sends to the interpreter.

So: Jest or Node-TAP? Normally I would prefer to use Node-TAP, since I'd prefer not to use a Facebook product. That said, Jest's snapshot based testing seems more robust than Node-TAP's.

@matt Do we have strong feelings about using a Facebook-originated product?

As for "inline" snapshot testing, which is what Ian Henry suggests: that's absolutely what we're going to do with Ludus.

Janet has (Judge)[https://github.com/ianthehenry/judge], which will be useful for sorting out the plumbing in the tree-walk interpreter. (I should start using it!)

Zig has ohsnap. (cf this post from the TigerBeetle folks/maktlad).

This suggests spinning three (or four) issues out of this one:

JS/Wasm-based testing.
Internal interpreter testing in Janet (and eventually Zig).
Inline snapshot-based testing in Ludus.

I tried getting Janet to do this, but it won't work. Ian Henry's Judge's `test-stdout` only captures in-Janet `stdout`, not anything printed to the console. So `(os-exec ["echo" "foobar"] :p)`, which _will_ echo `foobar` to the console actually produces no output for Judge. That suggests that the way to go about doing this is by using an environment that wraps Ludus (whether Janet– or Zig-based). There are two options for this: * shell * JS/Wasm The tooling for Judge's "snapshot-based" testing in JS is absolutely more robust than shell-based testing, so that suggests JS-based testing. A quick search in JS-testing-land shows that both Jest and Node-TAP have snapshot capabilities. Neither of these are inline. That said, I don't think we need _inline_ snapshot testing for the language-agnostic version; we're not _coding_ in JS, but executing code that JS sends to the interpreter. So: Jest or Node-TAP? Normally I would prefer to use Node-TAP, since I'd prefer not to use a Facebook product. That said, Jest's snapshot based testing seems more robust than Node-TAP's. @matt Do we have strong feelings about using a Facebook-originated product? As for "inline" snapshot testing, which is what Ian Henry suggests: that's _absolutely_ what we're going to do with Ludus. Janet has (Judge)[https://github.com/ianthehenry/judge], which will be useful for sorting out the plumbing in the tree-walk interpreter. (I should start using it!) Zig has [ohsnap](https://github.com/mnemnion/ohsnap). (cf [this post from the TigerBeetle folks/maktlad](https://tigerbeetle.com/blog/2024-05-14-snapshot-testing-for-the-masses)). This suggests spinning three (or four) issues out of this one: 1. JS/Wasm-based testing. 2. Internal interpreter testing in Janet (and eventually Zig). 3. Inline snapshot-based testing in Ludus.

Sign in to join this conversation.