# Catching back up
## May 2025

### Bugs

#### `match` is not popping things correctly
```
=== source code ===

let foo = match :foo with {
    :foo -> 1
    :bar -> 2
    :baz -> 3
}
foo

=== chunk: test ===
IDX | CODE           | INFO
0000: reset_match
0001: constant         0000: :foo
0003: match_constant   0000: :foo
0005: jump_if_no_match 0006
0007: constant         0001: 1
0009: store
0010: pop
0011: jump             0023
0013: match_constant   0002: :bar
0015: jump_if_no_match 0006
0017: constant         0003: 2
0019: store
0020: pop
0021: jump             0013
0023: match_constant   0004: :baz
0025: jump_if_no_match 0006
0027: constant         0005: 3
0029: store
0030: pop
0031: jump             0003
0033: panic_no_match
0034: load
0035: match_word
0036: panic_if_no_match
0037: push_binding     0000
0039: store
0040: pop
0041: pop
0042: load



=== vm run: test ===
0000: [] nil
0000: reset_match
0001: [] nil
0001: constant         0000: :foo
0003: [:10] nil
0003: match_constant   0000: :foo
0005: [:10] nil
0005: jump_if_no_match 0006
0007: [:10] nil
0007: constant         0001: 1
0009: [:10|1] nil
0009: store
0010: [:10|nil] 1
0010: pop
0011: [:10] 1
0011: jump             0023
0036: [:10] 1
0036: panic_if_no_match
0037: [:10] 1                <== Should "return" from match here
0037: push_binding     0000
0039: [:10|:10] 1
0039: store
0040: [:10|nil] :10
0040: pop
0041: [:10] :10
0041: pop
0042: [] :10
0042: load
:foo
```
Should return `1`.

Instruction `0037` is where it goes off the rails.

### Things left undone
Many. But things that were actively under development and in a state of unfinishedness:

1. Tuple patterns
2. Loops
3. Function calls

#### Tuple patterns
This is the blocking issue for loops, function calls, etc.
You need tuple pattern matching to get proper looping and function calls.
Here are some of the issues I'm having:
* Is it possible to represent tuples on the stack? Right now they're allocated on the heap, which isn't great for function calls.
* How to represent complex patterns? There are a few possibilities:
    - Hard-coded into the bytecode (this is probably the thing to do?)
    - Represented as a data structure, which itself would have to be allocated on the heap
    - Some hybrid of the two:
        * Easy scalar values are hard-coded: `nil`, `true`, `:foo`, `10` are all built into the bytecode + constants table
        * Perhaps dict patterns will have to be data structures

#### Patterns, generally
On reflection, I think the easiest (perhaps not simplest!) way to go is to model the patterns as separate datatypes stored per-chunk in a vec.
The idea is that we push a value onto the stack, and then have a `match` instruction that takes an index into the pattern vec.
We don't even need to store all pattern types in that vec: constants (which already get stored in the constant vec), interpolations, and compound patterns.
`nil`, `false`, etc. are singletons and can be handled like (but not exactly as) the `nil` and `false` _values_.
This also means we can outsource the pattern matching mechanics to Rust, which means we don't have to fuss with "efficient compiling of pattern matching" titchiness.
This also has the benefit, while probably being fast _enough_, of reflecting the conceptual domain of Ludus, in which patterns and values are different DSLs within the language.
So: model it that way.

### Now that we've got tuple patterns
#### May 23, 2025
A few thoughts:
* Patterns aren't _things_, they're complex conditional forms. I had not really learned that; note that the "compiling pattern matching efficiently" paper is about how to optimize those conditionals. The tuple pattern compilation more closely resembles an `if` or `when` form.
* Tuple patterns break the nice stack-based semantics of binding. So do other patterns. That means I had to separate out bindings and the stack. I did this by introducing a representation of the stack into the compiler (it's just a stack-depth counter).
    - This ended up being rather titchy. I think there's a lot of room to simplify things by avoiding manipulating this counter directly. My sense is that I should probably move a lot of the `emit_op` calls into methods that ensure all the bookkeeping happens automatically.
* `when` is much closer to `if` than `match`; remember that!
* Function calls should be different from tuple pattern matching. Tuples are currently (and maybe forever?) allocated on the heap. Function calls should *not* have to pass through the heap. The good news: `Arguments` is already a different AST node type than `Tuple`; we'll want an `ArgumentsPattern` pattern node type that's different from (and thus compiled differently than) `TuplePattern`. They'll be similar--the matching logic is the same, after all--but the arguments will be on the stack already, and won't need to be unpacked in the same way.
    - One difficulty will be matching against different arities? But actually, we should compile these arities as different functions.
    - Given splats, can we actually compile functions into different arities? Consider the following:
    ```ludus
      fn foo {
        (x) -> & arity 1
        (y, z) -> & arity 2
        (x, y, 2) -> & arity 3
        (...z) -> & arity 0+
      }
    ```
    `fn(1, 2, 3)` and `fn(1, 2, 4)` would invoke different "arities" of the function, 3 and 0+, respectively. I suspect the simpler thing really is to just compile each function as a singleton, and just keep track of the number of arguments you're matching.
* Before we get to function calls, `loop`/`recur` make sense as the starting ground. I had started that before, and there's some code in those branches of the compiler. But I ran into tuple pattern matching. That's now done, although actually, the `loop`/`recur` situation probably needs a rewrite from the ground up.
* Just remember: we're not aiming for *fast*, we're aiming for *fast enough*. And I don't have a ton of time. So the thing to do is to be as little clever as possible.
* I suspect the dominoes will fall reasonably quickly from here through the following:
    - [x] `list` and `dict` patterns
    - [x] updating `repeat`
    - [x] standing up `loop`/`recur`
    - [x] standing up functions
    - [x] more complex synthetic expressions
    - [x] `do` expressions
* That will get me a lot of the way there. What's left after that which might be challenging?
    - [x] string interpolation
    - [x] splats
    - [x] splatterns
    - [x] string patterns
    - [x] partial application
    - [x] tail calls
    - [-] stack traces in panics
    - [-] actually good lexing, parsing, and validation errors. I got some of the way there in the fall, but everything needs to be "good enough."
* After that, we're in integration hell: taking this thing and putting it together for Computer Class 1. Other things that I want (e.g., `test` forms) are for later on.
* There's then a whole host of things I'll need to get done for CC2:
    - some kind of actual parsing strategy (that's good enough for "Dissociated Press"/Markov chains)
    - actors
    - animation in the frontend
* In addition to a lot of this, I think I need some kind of testing solution. The Janet interpreter is pretty well-behaved.

### Now that we've got some additional opcodes and loop/recur working
#### 2025-05-27
The `loop` compilation is _almost_ the same as a function body. That said, the thing that's different is that we don't know the arity of the function that's called.

A few possibilities:
* Probably the best option: enforce a new requirement that splat patterns in function clauses *must* be longer than any explicit arity of the function. So, taking the above:
    ```ludus
      fn foo {
        (x) -> & arity 1
        (y, z) -> & arity 2
        (x, y, 2) -> & arity 3
        (...z) -> & arity 0+
      }
    ```
This would give you a validation error that splats must be longer than any other arity.
Similarly, we could enforce this:
```ludus
fn foo {
    (x) -> & arity 1
    (x, y) -> & arity 2
    (x, ...) & arity n > 1; error! too short
    (x, y, ...) & arity n > 2; ok!
}  
```
The algorithm for compiling functions ends up being a little bit titchy, because we'll have to store multiple functions (i.e. chunks) with different arities.
Each arity gets a different chunk.
And the call function opcode comes with a second argument that specifies the number of arguments, 0 to 7.
(That means we only get a max of 7 arguments, unless I decide to give the call opcode two bytes, and make the call/return register much bigger.)

Todo, then:
* [x] reduce the call/return register to 7
* [x] implement single-arity compilation
* [x] implement single-arity calls, but with two bytes
* [x] compile multiple-arity functions
* [x] add closures

### Some thoughts while chattin w/ MNL
On `or` and `and`: these should be reserved words with special casing in the parser.
You can't pass them as functions, because that would change their semantics.
So they *look* like functions, but they don't *behave* like functions.
In Clojure, if you try `(def foo or)`, you get an error that you "can't take the value of a macro."
I'll need to change Ludus so that `or` and `and` expressions actually generate different AST nodes, and then compile them from there.

AND: done.

### Implementing functions & calls, finally
#### 2025-06-01
Just to explain where I am to myself:
* I have a rough draft (probably not yet fully functional) of function compilation in the compiler.
* Now I have to implement two op codes: `Call` and `Return`.
* I now need to implement call frames.
* A frame in _Crafting Interpreters_ has: a pointer to a Lox function object, an ip, and an index into the value stack that indicates the stack bottom for this function call. Taking each of these in turn:
* The lifetime for the pointer to the function:
    - The pointer to the function object cannot, I think, be an explicit lifetime reference, since I don't think I know how to prove to the Rust borrow checker that the function will live long enough, especially since it's inside an `Rc`.
    - That suggests that I actually need the whole `Value::Fn` struct and not just the inner `LFn` struct, so I can borrow it.
* The ip and stack bottom are just placeholders and don't change.

### Partially applied functions
#### 2025-06-05
Partially applied functions are a little complicated, because they involve both the compiler and the VM.
Maybe.
My sense is that, perhaps, the way to do them is actually to create a different value branch.
They ....

### Jumping! Numbers! And giving up the grind
#### 2025-06-05
Ok! So.
This won't be ready for next week.
That's clear enough now--even though I've made swell progress!
One thing I just discovered, which, well, it feels silly I haven't found this before.
Jump instructions, all of them, need 16 bits, not 8.

That means some fancy bit shifting, and likely some refactoring of the compiler & vm to make them easier to work with.

For reference, here's the algorithm for munging u8s and u16s:

```rust
let a: u16 = 14261;
let b_high: u8 = (a >> 8) as u8;
let b_low: u8 = a as u8;
let c: u16 = ((b_high as u16) << 8) + b_low as u16;
println!("{a} // {b_high}/{b_low} // {c}");
```

To reiterate the punch list that *I would have needed for Computer Class 1*:
* [x] jump instructions need 16 bits of operand
    - Whew, that took longer than I expected
* [x] splatterns
    - [-] validator should ensure splatterns are the longest patterns in a form
* [-] improve validator
    - [-] Tuples may not be longer than n members
    - [-] Loops may not have splatterns
    - [-] Identify others
* [x] add guards to loop forms
* [x] check loop forms against function calls: do they still work the way we want them to?
* [x] tail call elimination
* [x] stack traces in panics

* [-] actually good error messages
    - [-] parsing
    - [-] my memory is that validator messages are already good?
    - [-] panics, esp. no match panics
* [-] getting to prelude
    - [-] `base` should load into Prelude
    - [-] prelude should run properly
    - [-] prelude should be loaded into every context
* [-] packaging things up
    - [-] add a `to_json` method for values
    - [-] teach Rudus to speak our protocols (stdout and turtle graphics)
    - [-] there should be a Rust function that takes Ludus source and returns valid Ludus status json
    - [-] compile Rust to WASM
    - [-] wire Rust-based WASM into JS
    - [-] FINALLY, test Rudus against Ludus test cases

So this is the work of the week of June 16, maybe?

Just trying to get a sense of what needs to happen for CC2:
* [x] Actor model (objects, Spacewar!)
* [-] Animation hooked into the web frontend (Spacewar!)
* [-] Text input (Spacewar!)
    - [-] Makey makey for alternate input?
* [-] Saving and loading data into Ludus (perceptrons, dissociated press)
* [-] Finding corpuses for Dissociated Press

### Final touches on semantics, or lots of bugs
#### 2025-06-19
* Main code is fucking up bindings in functions
* Why?
* In the middle of doing TCO, looks like it works for `do` forms based on the bytecode, but I ran into these weird binding problems while trying to test that in the vm
* Once that's put to bed, I believe TCO fully works

_Edited to add: all the above is, I think, fixed._

* But then I need to test it actually works in its intended use case: recursive calls
* Which means I need to test recursive calls
* And, once that's done, I think I have a COMPLETE SEMANTICALLY CORRECT INTERPRETER.
* After that, jesus, it's time for base > prelude > test cases

#### Later
So this is my near-term TODO:

* [x] recursive calls
    - [x] direct recursive calls
        * [x] stash a function declaration before compiling the function, hang onto that declaration
        * [x] update that declaration to a definition after compiling the function
    - [x] mutual recursive
        * [x] check to make sure a function has already been declared, hang onto that declaration
        * [x] update that declaration to a definition after compiling the function...
        * [x] but BEFORE we close over any upvalues, so the function will be an upvalue for itself
    - I suspect this can be done not using anything other than an index into a chunk's `constants` vec--no fancy memory swapping or anything; and also--done in the compiler rather than the VM.
* [x] getting to prelude
    - [x] `base` should load into Prelude
    - [x] write a mock prelude with a few key functions from real prelude
    - [x] a prelude should be loaded into every context
    - [?] the full prelude should run properly
* [x] packaging things up
    - [x] add a `to_json` method for values
    - [x] teach Rudus to speak our protocols (stdout and turtle graphics)
    - [x] there should be a Rust function that takes Ludus source and returns valid Ludus status json
    - [x] compile Rust to WASM
    - [x] wire Rust-based WASM into JS
    - [-] FINALLY, test Rudus against Ludus test cases

And then: quality of life improvements:
* [-] refactor messes
    - [x] The compiler should abstract over some of the very titchy bytecode instruction code
    - [x] Pull apart some gargantuan modules into smaller chunks: e.g., `Op` and `Chunk`  should be their own modules
    - [x] Identify code smells
    - [x] Fix some of them
* [-] improve validator
    - [-] Tuples may not be longer than n members
    - [-] Loops may not have splatterns
    - [-] Identify others
    - [-] Splats in functions must be the same arity, and greater than any explicit arity
* [-] actually good error messages
    - [-] parsing
    - [-] my memory is that validator messages are already good?
    - [-] panics, esp. no match panics
        * [-] panics should be able to refernce the line number where they fail
        * [-] that suggests that we need a mapping from bytecodes to AST nodes
        * The way I had been planning on doing this is having a vec that moves in lockstep with bytecode that's just references to ast nodes, which are `'static`, so that shouldn't be too bad. But this is per-chunk, which means we need a reference to that vec in the VM. My sense is that what we want is actually a separate data structure that holds the AST nodes--we'll only need them in the sad path, which can be slow. 

### Bugs discovered while trying to compile prelude
#### 2025-06-20
Consider the following code:
```ludus
fn one {
    (x as :number) -> {
        fn two () -> :number
        two
    }
    (x as :bool) -> {
        fn two () -> :bool
        two
    }
}

```
The second clause causes a panic in the compiler.
I'm not entirely sure what's going on.
That said, I'm pretty sure the root of it is that the fact that `two` was already bound in the first clause means that it's in the constants vector in that chunk under the name "two."

...

So that's fixed!

But:
Prelude functions are not properly closing over their upvalues.
Which is not right.
We get: `get_upvalue 000` and a panic: `index out of bounds: the len is 0 but the index is 0`.
The vec in question is the LFn::Defined.closed.

...

Just patched up the `if` alternative branch unconditional jump, which was jumping too far.

Now it really is just some pretty systematic testing of prelude functions, including the problems with upvalues, which I haven't yet been able to recreate.

### On proceeding from here
#### 2025-06-021
Rather than doing everything by hand right now, I think the way to go about things is to figure out how to do as much automated bugfixing as possible.
That means reprioritizing some things.
So here's a short punch list of things to do in that register:
* [x] Hook validator back in to both source AND prelude code
    - [x] Validator should know about the environment for global/prelude function
    - [x] Run validator on current prelude to fix current known errors
* [x] Do what it takes to compile this interpreter into Ludus's JS environment
    - [x] JSONify Ludus values
    - [x] Write a function that's source code to JSON result
    - [x] Expose this to a WASM compiler
    - [x] Patch this into a JS file
    - [-] Automate this build process
* [-] Start testing against the cases in `ludus-test`
* [-] Systematically debug prelude
    - [-] Bring it in function by function, testing each in turn

***
I've started working on systematically going through the Prelude.
I've found a closure error.
This is in `map`/`mapping`.
What's happening is that the inner function, `mapping`, is closing over values directly from a stack that no longer exists.
What we need to have happen is that if a function is closing over values _inside_ a function, it needs to capture not the upvalues directly from the stack, but _from the enclosing closure_.
I think I need to consult Uncle Bob Nystrom to get a sense of what to do here.
***
So I found the minimal test case:
```ludus
let foo = {
    let thing = :thing
    let bar = :bar
    let baz = :baz
    fn quux () -> {
        fn frobulate () -> (bar, baz)
    }
}

foo ()
```
`frobulate` is closed over when `quux` is called, when the stack looks completely different than it does when `quux` is defined.
If you remove line 2, binding `thing`, then you don't get a panic, but when you call `foo () ()`, the result is `(fn quux, fn frobulate)`.
The problem is that `frobulate`'s upvalues are indexes into the stack, rather than having some relation to `quux`'s upvalues.
What needs to happen is that an enclosing function needs to capture, define, and pass down the upvalues for its enclosed functions.

I'm having an exact problem that Uncle Bob is describing at 
https://craftinginterpreters.com/closures.html#flattening-upvalues. 
I need to study and adapt this exact set of problems.
I believe I need to take the strategy he uses with closures being different from functions, etc.
So: rework the closures strategy here.

***

Closures strategy mostly unfucked.
Now I'm having some difficulty with bindings, again?
Current situation: still trying to get `map` and `fold` to work properly.
The bindings inside of non-trivial functions goes weird.
The scope depths are all out of whack.
And the stack positions on stuff are also totally weird.
One thing: the way we are now resolving upvalues means that nothing should ever reach back further in the stack than the stack base in the vm.
So now we'll do all bindings relative to the stack base.
UGH.

***

So now I have a bug that I'm dying to figure out.
But it's time to go to bed.

When `recur` is in the alternatve branch of an if, it thinks there's one more value on the stack than there really is.
To the best of my ability to tell, `if` has proper stack behaviour.
So the question is what's happening in the interaction between the `jump_if_false` instruction and `recur`.

To wit, the following code works just fine:
```ludus
fn not {
    (false) -> true
    (nil) -> true
    (_) -> false
}
    
let test = 2
loop ([1, 2, 3]) with {
	([]) -> false
	([x]) -> eq? (x, test)
	([x, ...xs]) -> if not(eq? (x, test))
		then recur (xs)
		else true
}
```

But the following code does not:
```ludus
let test = 2
loop ([1, 2, 3]) with {
	([]) -> false
	([x]) -> eq? (x, test)
	([x, ...xs]) -> if eq? (x, test)
		then true
		else recur (xs)
}
```

Meanwhile, other `loop`/`recur` forms seem to work okay for me.
So: ugh.

### Grinding and grinding
#### 2025-06-22
Got 'er done.
Fixed `loop`/`recur` and many other stack shananigans.
I don't believe I've fixed everything yet.
I may be surprised, though.

Currently fixing little bugs in prelude.
Here's a list of things that need doing:
* [-] Escape characters in strings: \n, \t, and \{,  \}.
* [-] `doc!` needs to print the patterns of a function.
* [-] I need to return to the question of whether/how strings are ordered; do we use `at`, or do we want `char_at`? etc.
* [-] Original implementation of `butlast` is breaking stack discipline; I don't know why. It ends up returning from evaluating one of the arguments straight into a `load` instruction. Something about tail calls and ternary synthetic expressions and base functions. (For now, I can call `slice` instead of `base :slice` and it works.)
    - Original version of `update` also had this same problem with `assoc`; fixed it by calling the Ludus, rather than Rust, function.
    - I need this fixed for optimization reasons.
    - I _think_ I just fixed this by fixing tail position tracking in collections
    - [-] test this
    - I did not fix it.
* [x] Dict patterns are giving me stack discipline grief. Why is stack discipline so hard?
* [x] This is in the service of getting turtle graphics working
* Other forms in the language need help:
    * xx] repeat needs its stack discipline updated, it currently crashes the compiler

### More closure problems
#### 2025-06-23
My solution to closures wasn't quite right.
I can't use Uncle Bob's strategy of the recursive call, since Rust's ownership semantics make this onerous at best.
My solution: introduce the concept of a "compiler depth," with 0 being the global scope.
If the compiler's at 0 depth, we can pull it out of the environment.
If the compiler's at a depth > 0, then we can ask the enclosing compiler to stash the upvalue.
And thus we get what we need.

But: some functions in prelude aren't properly getting their closures, and I don't know why, since they *are* getting them properly in user scripts.
Take `apply_command`.

Next step: sort out if any other functions aren't getting things closed over properly.

PROBLEM: forward-declared functions weren't at the top of the stack when `Op::SetUpvalue` was called.
So all of `apply_command`'s upvalues were being attached to the function declared before it (which was sitting right there at the top of the stack.)

SOLUTION: test to see if the function has been forward-declared, and if it has, bring it to the top fo the stack.

NEW PROBLEM: a lot of instructions in the VM don't properly offset from the call frame's stack base, which leads to weirdness when doing things inside function calls.

NEW SOLUTION: create a function that does the offset properly, and replace everywhere we directly access the stack.
This is the thing I am about to do

### I think the interpreter, uh, works?
#### 2025-06-24
I'm sure I'll find some small problems.
But right now the thing works.
At the moment, I'm thinking about how to get good error messages.
Panics are difficult.
And I'm worried about ariadne as the error reporting crate.
Since it writes to stdout, it has all kinds of escape codes.
I need a plain ass string, at least for the web frontend.
So.

Current task, however, is how to get reasonable panic error messages.
Let's simplify the problem.

First, let's get tracebacks and line numbers.
We use Chumsky spanned ASTs.
The span is just a range into an str.
What I can do is pretty stupidly just generate line numbers from the spans in the compiler, and from there, get a reasonable traceback.
So instead of using Ariadne's fancy report builder, let's just do something more what we already have here:

```
Ludus panicked! no match
  on line 1 in input, 
  calling: add
  with arguments: ("foo")
  expected match with one of:
    ()
    (x as :number)
    (x as :number, y as :number)
    (x, y, ...zs)
    ((x1, y1), (x2, y2))
  >>> add ("foo")
......^
```
We need:
* the location of the function call in terms of the line number
* the arguments used
* the patterns expected to match (special cases: `let` vs `match` vs `fn`)

That means, for bookkeeping, we need:
* In the compiler, line number
* In the VM, the arguments
* In the VM, the pattern AST node.

In Janet-Ludus, there are only a few types of panic:
* `fn-no-match`: no match against a function call
* `let-no-match`: no match against a `let` binding
* `match-no-match`: no match against a `match` form
* `generic-panic`: everything else
* `runtime-error`: an internal Ludus error

The first three are simply formatting differences.
There are no tracebacks.

Tracebacks should be easy enough, although there's some fiddly bits.
While it's nice to have the carret, the brutalist attempt here should be just to give us the line--since the carret isn't exactly precise in the Janet interpereter.
And the traceback should look something like:

```
    calling foo with (:bar, :baz)
        at line 12 in input
    calling bar with ()  
        at line 34 in prelude
    calling baz with (1, 2, 3)
        at line 12 in input
```

Which means, again: function names, ip->line conversion, and arguments.

The runtime needs a representation of the patterns in _any_ matching form.
The speed is so much greater now that I'm not so concerned about little optimizations.
So: a chunk needs a vec of patterns-representations. (I'm thinking simply a `Vec<String>`.)
So does a function, for `doc!`.
Same same re: `Vec<String>`.
A VM needs a register for the scrutinee (which with function calls is just the arguments, already captured).
A VM also needs a register for the pattern.
So when there's a no match, we just yank the pattern and the scrutinee out of these registers.

This all seems very straightforward compared to the compiling & VM stuff.

Here's some stub code I wrote for dealing with ranges, source, line numbers:

```rust
let str = "foo bar baz\nquux frob\nthing thing thing";
let range = 0..4;

println!("{}", str.get(range).unwrap());

let lines: Vec<&str> = str.split_terminator("\n").collect();

println!("{:?}", lines);

let idx = 20;

let mut line_no = 1;
for i in 0..idx {
    if str.chars().nth(i).unwrap() == '\n' {
        line_no += 1;
    }
}

println!("line {line_no}: {}", lines[line_no - 1]);
```

This is the thing I am about to do.

### I think the interpreter, uh, works?
#### 2025-06-24
I'm sure I'll find some small problems.
But right now the thing works.
At the moment, I'm thinking about how to get good error messages.
Panics are difficult.
And I'm worried about ariadne as the error reporting crate.
Since it writes to stdout, it has all kinds of escape codes.
I need a plain ass string, at least for the web frontend.
So.

Current task, however, is how to get reasonable panic error messages.
Let's simplify the problem.

First, let's get tracebacks and line numbers.
We use Chumsky spanned ASTs.
The span is just a range into an str.
What I can do is pretty stupidly just generate line numbers from the spans in the compiler, and from there, get a reasonable traceback.
So instead of using Ariadne's fancy report builder, let's just do something more what we already have here:

```
Ludus panicked! no match
  on line 1 in input, 
  calling: add
  with arguments: ("foo")
  expected match with one of:
    ()
    (x as :number)
    (x as :number, y as :number)
    (x, y, ...zs)
    ((x1, y1), (x2, y2))
  >>> add ("foo")
......^
```
We need:
* the location of the function call in terms of the line number
* the arguments used
* the patterns expected to match (special cases: `let` vs `match` vs `fn`)

That means, for bookkeeping, we need:
* In the compiler, line number
* In the VM, the arguments
* In the VM, the pattern AST node.

In Janet-Ludus, there are only a few types of panic:
* `fn-no-match`: no match against a function call
* `let-no-match`: no match against a `let` binding
* `match-no-match`: no match against a `match` form
* `generic-panic`: everything else
* `runtime-error`: an internal Ludus error

The first three are simply formatting differences.
There are no tracebacks.

Tracebacks should be easy enough, although there's some fiddly bits.
While it's nice to have the carret, the brutalist attempt here should be just to give us the line--since the carret isn't exactly precise in the Janet interpereter.
And the traceback should look something like:

```
    calling foo with (:bar, :baz)
        at line 12 in input
    calling bar with ()  
        at line 34 in prelude
    calling baz with (1, 2, 3)
        at line 12 in input
```

Which means, again: function names, ip->line conversion, and arguments.

The runtime needs a representation of the patterns in _any_ matching form.
The speed is so much greater now that I'm not so concerned about little optimizations.
So: a chunk needs a vec of patterns-representations. (I'm thinking simply a `Vec<String>`.)
So does a function, for `doc!`.
Same same re: `Vec<String>`.
A VM needs a register for the scrutinee (which with function calls is just the arguments, already captured).
A VM also needs a register for the pattern.
So when there's a no match, we just yank the pattern and the scrutinee out of these registers.

This all seems very straightforward compared to the compiling & VM stuff.

Here's some stub code I wrote for dealing with ranges, source, line numbers:

```rust
let str = "foo bar baz\nquux frob\nthing thing thing";
let range = 0..4;

println!("{}", str.get(range).unwrap());

let lines: Vec<&str> = str.split_terminator("\n").collect();

println!("{:?}", lines);

let idx = 20;

let mut line_no = 1;
for i in 0..idx {
    if str.chars().nth(i).unwrap() == '\n' {
        line_no += 1;
    }
}

println!("line {line_no}: {}", lines[line_no - 1]);
```

### Integration meeting with mnl
#### 2025-06-25
* Web workers
* My javascript wrapper needs to execute WASM in its own thread (ugh)
    - [-] is this a thing that can be done easily in a platform-independent way (node vs. bun vs. browser)?
* Top priorities:
    - [-] Get a node package out
    - [-] Stand up actors + threads, etc.
    - [-] How to model keyboard input from p5?
        * [-] Model after the p5 keyboard input API
        * [-] ludus keyboard API: `key_is_down(), key_pressed(), key_released()`, key code values (use a dict) 
    - Assets:
        * We don't (for now) need to worry about serialization formats, since we're not doing perceptrons
        * We do need to read from URLs, which need in a *.ludus.dev.
        * Users can create their own (public) repos and put stuff in there.
        * We still want saving text output from web Ludus
        * Later, with perceptrons & the book, we'll need additional solutions.

#### Integration hell
As predicted, Javascript is the devil.

wasm-pack has several targets:
* nodejs -> this should be what we want
* web -> this could be what we want
* bundler -> webpack confuses me

The simplest, shortest route should be to create a viable nodejs library.
It works.
I can wire up the wasm-pack output with a package.json, pull it down from npm, and it work.
However, because of course, vite, which svelte uses, doesn't like this.
We get an error that `TextEncoder is not a constructor`.
This, apparently, has something to do with the way that vite packages up node libraries?
See https://github.com/vitejs/vite/discussions/12826.

Web, in some ways, is even more straightforward.
It produces an ESM that just works in the browser.
Put the rest of my thinking up in an issue on alea.

### Actor/Model
#### 2025-06-26
Okay, even though we're not fully hooked up with wasm yet, I've started working on the actor model.
I believe I have things broadly right.
The thing that I'm thinking about right now is how to connect ludus to the world and to processes.
It's not clear how to expose that plumbing.

The most obvious way to do this would be to make all the process stuff special forms.

But first, before I get there, this is what Elixir does.

* `receive` is a special form. It is strictly equivalent to `match`, but with the caveats that (a) if there are no messages to receive, the process yields, and (b) if there's no match for the first message, it matches against the second, and so on (keeping messages in the mailbox), and (c) if no messages match, the process yields as well.

Everything else is functions:
* `spawn/1` takes a function to execute
* `send/3` takes a pid and a message (the third argument is for options, which we won't have in Ludus), returning a pid
* `self/0` returns the current processes' pid
* `exit/2` takes a pid and a message, and kills the process
* `sleep/1` takes a duration in milliseconds and sleeps the process for a time. This isn't strictly speaking necessary, but it is a nice-to-have.

In Ludus, `receive` will need to be a special form.
We could make `self` a reserved word, emitting the instruction to get the current pid.
`spawn` I like as a special form: whatever the expression that comes after spawn is just deferred into the new process.
So in place of Elixir's `spawn(fn -> :something end)` in Ludus, we'd have `spawn :something`.
`send`, `exit`, and `sleep` are a little more interesting (difficult).

Each could be like `and` and `or`: special forms that _look_ like functions, but really aren't--implemented in the compiler.

Alternately, I could attempt to find a way to expose the vm to base.
That seems, frankly, rather more daunting in Rust.

But we could also just build out syntax?
For example, I had proposed the following desugaring:
`foo ::bar (baz)` -> `send (foo, (:bar, baz))`
I don't mind the sugar, but it feels like it's actually really important conceptually for Ludus that everything really is functions.
(Then again, the sugar feels weird, because `foo` is just a name bound to a keyword, so it's possible to do: `:foo ::bar (baz)`, which makes sense and is weird.).

The only disadvantage with making `send` a special form is that you can't pass it as a higher-order function.
`and` and `or` must be special forms, and don't make sense as higher-order functions because they have a different evaluation order than regular functions.
But that's not true of all the things here.

So how do we connect Ludus function calls to handles in the rust interpreter?
In place of base functions, we need something like messages to the process, that aren't mailbox processes but something else.
So, like, a special sort of Ludus value, only available in base, that represents the VM.
Call it `base :process`.
We call it like a function, but the VM instead responds to the various prompts.
Thus we can use it to communicate with the process.
`receive` will be tricky.
But everything else? Seems pretty straightforward.


#### Some time later...
I've implemented what I decribe above. It works! I'm low-key astonished.
It perfectly handles an infinitely recurring process! What the fuck.
Anyway, things left to do:
* [x] `receive` forms are the big one: they require threading through the whole interpreter
* [x] implement the missing process functions at the end of prelude
* [-] research how Elixir/Erlang's abstractions over processes work (I'm especially interested in how to make synchronous-looking calls); take a look especially at https://medium.com/qixxit-development/build-your-own-genserver-in-49-lines-of-code-1a9db07b6f13
* [-] write some examples just using these simple tools (not even GenServer, etc.) to see how they work, and to start building curriculum
* [-] develop a design for how to deal with asynchronous io with js

```ludus
fn agent/get (pid) -> {
    send (pid, (:get, self ()))
    receive {
        (:response, value) -> value
    }
}

fn agent/store (pid, x) -> {
    send (pid, (:store, x))
    :ok
}

fn agent/update (pix, f) -> {
    send (pid, (:update, f))
}

fn agent (state) -> receive {
    (:get, pid) -> {
        send (pid, (:response, state))
        agent (state)
    }
    (:update, pid, f) -> {
        agent (f (state))
    }
    (:store, pid, x) -> {
        agent (x)
    }
}
```

Two things that pop out to me:
* The way this works is just to yield immediately. This actually makes a lot of sense. If we put them next to one another, there's no risk that there'll be backlogged `(:response, x)` messages in the mbx, right? But that makes me a little queasy.
* The way `gen_server` works is pretty deep inversion of control; you effectively write callbacks for the `gen_server` to call. I'm not sure that's how we want to do things in Ludus; it's a handy pattern, and easy. But it's not simple. But also worth investigating. In any event, it's the foundation of all the other process patterns Elixir has developed. I need an intuiation around it.

### Rethinking reception
#### 2025-06-27
So one thing that's stuck with me is that in Elixir, ~`receive` isn't a special form: it's a function that takes a block~ (**EDIT**: it is indeed a special form in Elixir, and it has to be on in Ludus.).
It may be a macro, but it's still mostly normalish, and doesn't invovle compiler shenanigans.

So, this is what I want to write:

```ludus
fn receiver () -> receive {
    :foo -> :bar
    :bar -> :baz
    :baz -> :quux
}  
```
There's a way to model this without the magic of receive.
Imagine instead a function that just receives a message and matches against it:
```ludus
fn receive_msg (msg) -> match msg with {
    :foo -> :bar
    :bar -> :baz
    :baz -> :quux
}
```
But if there's no matching message clause, we get a panic.
And panics stop the world.
Meanwhile, we need to know whether the message matched.
So this desugaring goes a little further:
```ludus
fn receive_msg (msg) -> match msg with {
    :foo -> :bar
    :bar -> :baz
    :baz -> :quux
    _ -> :does_not_understand
}
```
This way we avoid a panic when there's no matching message.
There's an easy wrapping function which looks like this:
```ludus
fn receive (receiver) -> {
    let my_msgs = msgs ()
    loop (my_msgs, 0) with {
        ([], _) -> yield! ()
        (xs, i) -> match receiver(first (xs)) with {
            :does_not_understand -> recur (rest (xs), inc (i))
            x -> {
                flush_n! (i)
                x
            }
        }
    }
}

receive (receive_msg)
```
There's a thing I both like and don't like about this.
The fact that we use a magic keyword, `:does_not_understand`, means it's actually easy to hook into the behaviour of not understanding.
I don't know if we should panic on a process receiving a message it doesn't understand.
Maybe we do that for now?
Crash early and often; thanks Erlang.
And so we do have to worry about the last clause.

At an implementation level, it's worth noting that we're not optimizing for scanning through the messages--we'll scan through any messages we don't understand every time we call `receive`.
That's probably fine!

So now the thing, without sugar, is just:
```ludus
fn agent (x) -> receive (fn (msg) {
    (:get, pid) -> {
        send (pid, (:response, x))
        agent (x)
    }
    (:set, y) -> agent(y)
    (:update, f) -> agent (f (x))
    _ -> :does_not_understand
})
```
So I don't need any sugar to make `receive` work?
And it doesn't even need to hook into the vm?
What?

#### some time later
It turns out I do.
The problem is that the `flush_i` instruction never gets called from Ludus in a thing like an agent, because of the recursive call.
So the flushing would need to happen in the receiver.


A few things that are wrong right now:
* [-] `loop`/`recur` is still giving me a very headache. It breaks stack discipline to avoid packing tuples into a heap-allocated vec. There may be a way to fix this in the current compiler scheme around how I do and don't handle arguments--make recur stupider and don't bother loading anything. A different solution would be to desugar loop into an anonymous function call. A final solution would be just to use a heap-allocated tuple.

Minimal failing case:
```ludus
match (nil) with {
    (x) -> match x with {
        (y) -> recur (y)
    }
}
```
* [x] The amount of sugar needed to get to `receive` without making a special form is too great.

In particular, function arities need to be changed, flushes need to be inserted, anonymous lambdas need to be created (which can't be multi-clause), etc.

_This is now implemented._

~* [-] There was another bug that I was going to write down and fix, but I forgot what it was. Something with processes.~
* [-] I remembered: I got some weird behaviour when `MAX_REDUCTIONS` was set to 100; I've increased it to 1000, but now need to test what happens when we yield because of reductions.

* [-] Also: the `butlast` bug is still outstanding: `base :slice` causes a panic in that function, but not the call to the Ludus function. Still have to investigate that one.

* [x] In testing this, it's looking like `match` is misbehaving; none of the matches that *should* happen in my fully sugarless `receive` testing are matching how they ought.
    - That is not what was happening. The mailbox 

I haven't got to a minimal case, but here's what's not working:
```ludus
 fn receive (receiver) -> {
  fn looper {
   ([], _) -> yield! ()
   (xs, i) -> {
    print!("looping through messages:", xs)
    match receiver (first (xs), i) with {
     :does_not_understand -> looper (rest (xs), inc (i))
     x -> x
   }}
 }
 print! ("receiving in", self (), "with messages", msgs())
 looper (msgs (), 0)
}

fn agent (x) -> receive (fn (msg, i) -> {
print!("received msg in agent: ", msg)
match msg with {
  (:get, pid) -> {
    flush_i! (i)
    print!("getted from {pid}")
    send (pid, (:response, x))
    agent (x)
  }
  (:set, y) -> {flush_i!(i); print!("setted! {y}"); agent (y)}
  (:update, f) -> {flush_i!(i);print!("updated: {f}"); agent (f (x))}
  y -> {print!("no agent reception match!!!! {y}");:does_not_understand}
}
})

fn agent/get (pid) -> {
  send (pid, (:get, self ()))
  yield! ()
  receive (fn (msg, i) -> match msg with {
    (:response, x) -> {flush_i! (i); x}
  })
}

fn agent/set (pid, val) -> send (pid, (:set, val))

fn agent/update (pid, f) -> send (pid, (:update, f))

let counter = spawn! (fn () -> agent (0))
agent/set (counter, 12)
agent/update (counter, inc)
agent/update (counter, mult(_, 3))
agent/get (counter)
```

_I haven't been able to recreate this, and the code above is subtle enough that the Ludus may be behaving as expected; the following works as expected:_

```ludus
fn receive (receiver) -> {
  print! ("receiving in", self (), "with msgs", msgs())
  if empty? (msgs ())
    then {yield! (); receive (receiver)}
    else do msgs () > first > receiver
}

fn foo? (val) -> receive (fn (msg) -> match report!("scrutinee is", msg) with {
  (:report) -> {
    print! ("LUDUS SAYS ==> value is {val}")
    flush! ()
    foo? (val)
  }
  (:set, x) -> {
    print! ("LUDUS SAYS ==> foo! was {val}, now is {x}")
    flush! ()
    foo? (x)
  }
  (:get, pid) -> {
    print! ("LUDUS SAYS ==> value is {val}")
    send (pid, (:response, val))
    flush! ()
    foo? (val)
  }
  x -> print! ("LUDUS SAYS ==> no match, got {x}")
})

let foo = spawn! (fn () -> foo? (42))
print! (foo)
send (foo, (:set, 23))
yield! ()
send (foo, (:get, self ()))
yield! ()
fn id (x) -> x
receive(id)
```

#### some time later
I've started work on `receive`.
I've got all the stuff wired up, and it seems to all work (and was pretty straightforward!).
EXCEPT: I've got a difficult off-by-one error.
The problem is being in a receive in a tight loop/tail call, where the ip doesn't advance past the tail call back to the top of the function.
Jumping back to the beginning of the loop advances the message counter by one.
Basically, after the first time we've matched, we keep skipping item 0.
So: what I need to do is to figure out the right order of operations for.
This is just stepwise logic, and some titchy state management.

So the thing that's worth noting is that entering the receive afresh with the whole message queue and entering it with the next message in the queue are _different_ behaviours. This may involve mucking with the instruction pointer on a yield.
This is subtle but will give me the feeling of "oh, why didn't I see that immediately" as soon as I get it.

### Step-by-step
#### 2025-06-28
Here's some pseudobytecode to get us to where we need to be:

010 reset the message counter
020 load current message
025 if no more messages, jump to 010 THEN yield
030 test the message against a pattern
040 if no match jump to 090
050 reset message counter
060 delete current message
070 execute body
    # this may be the last instruction executed
    # recursive tail calls will jump to 010
080 jump to 100
085 increase the message counter
090 jump to 025 (not really in bytecode; this will be unrolled)
100 receive end

#### a short time later
Well, that worked! The real issue was the jump back if we're out of messages.

That leaves the following list:
* [a] research how Elixir/Erlang's abstractions over processes work (I'm especially interested in how to make synchronous-looking calls); take a look especially at https://medium.com/qixxit-development/build-your-own-genserver-in-49-lines-of-code-1a9db07b6f13
* [ ] write some examples just using these simple tools (not even GenServer, etc.) to see how they work, and to start building curriculum
* [a] develop a design for how to deal with asynchronous io with js
* [a] I got some weird behaviour when `MAX_REDUCTIONS` was set to 100; I've increased it to 1000, but now need to test what happens when we yield because of reductions.
* [a] Also: the `butlast` bug is still outstanding: `base :slice` causes a panic in that function, but not the call to the Ludus function. Still have to investigate that one.
    - Original version of `update` also had this same problem with `assoc`; fixed it by calling the Ludus, rather than Rust, function.
* [a] `loop`/`recur` is still giving me a very headache. It breaks stack discipline to avoid packing tuples into a heap-allocated vec. There may be a way to fix this in the current compiler scheme around how I do and don't handle arguments--make recur stupider and don't bother loading anything. A different solution would be to desugar loop into an anonymous function call. A final solution would be just to use a heap-allocated tuple.
* My javascript wrapper needs to execute WASM in its own thread (ugh)
    - [x] is this a thing that can be done easily in a platform-independent way (node vs. bun vs. browser)?
    - No, no it is not. I will need to build a separate node version for using at the command line (and, like, for testing with our test harness.)
* Top priorities:
    - [-] Get a node package out
    - [x] Stand up actors + threads, etc.
    - [ ] How to model keyboard input from p5?
        * [ ] Model after the p5 keyboard input API
        * [ ] ludus keyboard API: `key_down?(), key_pressed?(), key_released?()`, key code values (use a dict) 
* [a] Escape characters in strings: \n, \t, and \{,  \}.
* [a] `doc!` needs to print the patterns of a function.
* [a] I need to return to the question of whether/how strings are ordered; do we use `at`, or do we want `char_at`? etc.
    - MNL and I decided: yes, stings are indexable
    - [ ] implement `slice` and `at` and others for strings
* [x] Automate this build process
* [ ] Start testing against the cases in `ludus-test`
* [ ] Systematically debug prelude
* [ ] Bring it in function by function, testing each in turn
* [x] Animation hooked into the web frontend (Spacewar!)
* [ ] Text input (Spacewar!)
* [ ] ~Makey makey for alternate input?~
* [x] Saving and loading data into Ludus (perceptrons, dissociated press)
* [ ] Finding corpuses for Dissociated Press
* [a] improve validator
    - [a] Tuples may not be longer than n members
    - [a] Loops may not have splatterns
    - [ ] Identify others
    - [a] Splats in functions must be the same arity, and greater than any explicit arity
* [a] actually good error messages
    - [a] parsing
    - [ ] my memory is that validator messages are already good?
    - [a] panics, esp. no match panics
        * [ ] panics should be able to refernce the line number where they fail
        * [ ] that suggests that we need a mapping from bytecodes to AST nodes
        * The way I had been planning on doing this is having a vec that moves in lockstep with bytecode that's just references to ast nodes, which are `'static`, so that shouldn't be too bad. But this is per-chunk, which means we need a reference to that vec in the VM. My sense is that what we want is actually a separate data structure that holds the AST nodes--we'll only need them in the sad path, which can be slow. 

### Next steps in integration hell
#### 2025-06-29
* [x] improve build process for rudus+wasm_pack
    - [x] delete generated .gitignore
    - [x] edit first line of rudus.js to import the local `ludus.js`
    - On this, made a justfile, but I needed to like actually try the build and figure out what happens. I got carried away touching the js. Too many things at once.
* [x] design & implement asynchronous i/o+runtime
    - [x] use `box`es for i/o: they can be reified in rust: making actors available is rather more complex (i.e. require message passing between the ludus and rust)
        * We also then don't have to have prelude run in the vm; that's good
        * We... maybe or maybe don't need processes in prelude, since we need to read and write from the boxes; we might be able to do that with closures and functions that call `spawn!` themselves
    - [x] start with ludus->rust->js pipeline
        * [x] console
        * [ ] turtle graphics
        * [x] completion
    - [ ] then js->rust->ludus
        * [x] kill
        * [x] text input
        * [ ] keypresses
    - [x] then ludus->rust->js->rust->ludus
        * [x] slurp
    - For the above, I've started hammering out a situation. I ought to have followed my instinct here: do a little at a time. I ended up doing all the things in one place all at once.
    - What I've done is work on a bespoke `to_json` method for values; and using serde deserialization to read a string delivered from js. I think this is easier and more straightforward than using `wasm_bindgen`. Or easier; I have no idea what the plumbing looks like.
    - Just to catch myself up, some additional decisions & thoughts:
        * No need to send a run event: we'll just start things with with a call to `run`, which we expose to JS.
        * One thing I hadn't quite grokked before is that we need to have a way of running the i/o events. Perhaps the simplest way to do this is to just to do it every so often, regardless of how long the ludus event loop is taking. That way even if things are getting weird in the VM, i/o still happens regularly.
        * The return to a `slurp` call is interesting.
        * I think the thing to do is to write to a slurp buffer/box as well.

### Finishing integration?
#### 2025-07-01
Happy Canada day!

After a really rough evening, I seem to have the actor model not only working in Ludus, but reasonably debugged in Rust.
We've got one bug to address in Firefox before I continue:
* [x] the event loop isn't returning once something is done, which makes no sense
    - What seems to be happening is that the javascript behaviour is subtly different
    - Current situation is that synchronous scripts work just fine
    - But async scripts work ONCE, and then not again
    - In FF, `do_io` doesn't return after `complete_main` in the `world` loop the second time.
    - Which is to say, that last call to `io` isn't completing.
    - Do I hack around this or do I try to find the source of the problem?

After that:
* [ ] implement other verbs beside `console`:
    - [x] `command`
    - [x] `input`
        * [x] js->rust->ludus buffer (in Rust code)
        * [x] ludus abstractions around this buffer (in Ludus code)
    - [x] `fetch`--request & response
        * [x] request: ludus->rust->js->net
        * [x] response: js->rust->ludus
    - [ ] `keyboard`
        * [ ] still working on how to represent this
* [x] hook this up to `web.ludus.dev`
* [x] do some integration testing
    - [x] do synchronous programs still work?
    - [x] animations?
    - [x] read inputs?
    - [x] load url text?

### Following links
#### 2025-07-05
Doing some last minute design work on linking processes.
Here's the erlang way:
* `link` is `link/1`: we always link _from a process_.
* links are always bidirectional
* [ ] update the link API to reflect this
* There's the concept of an exit signal
    - erlang has three ways of doing this (ish): `:normal` (returning), `:kill` (panic), or a reason (which is a value).
    - you can "trap" an exit signal that's `:normal` or a reason, but not if it's `:kill`, which will bring down the monitoring process
* I think in ludus we want...

Well? What do we want?

Erlang and Elixir have designs that are meant for massively concurrent processes.
In addition, I've been thinking about how the willy-nilly, anything goes, messages getting passed around all kinds of places, actually cuts against the spirit of Ludus.
What sort of process management, abstraction, modelling, and mental-modelling do we want in ludus?
My sense is that, actually, I have to build the things we're going to build before I have a sense of what needs doing.
`link` may not even be a thing we need?

Anyway, here's the tl;dr for basic process linking:
* `link/1` should establish a bidirectional link, bringing one process down if the other exits for any reason, causing a panic if the linked process exits unexpectedly, even on a normal return
* `monitor/1` means we can monitor exit messages, these will be `(:exit, pid, (:ok, value))` in the case of successful return, or `(:exit, pid, (:err, msg))` in the case of a panic. These are sent as normal messages to the monitoring process.

Erlang/Elixir has a lot more, but that's basically it for Ludus for now.
I don't think we need more than the binary: `:ok`/`:err`, or return/panic as how processes exit.
`monitor/1` makes it easy to write an `await`.
We would need a way of killing processes remotely if indeed we wanted to write robust distributed applications.
But I'm less interested in that than in the cognitive modelling here.

As I'm thinking about this, I actually want to roll back `spawn` as a special form.
I think it's actually a good cognitive shift to understand that functions are reified, deferred computations.
And that processes take this a long way.
We can revisit the idea of a special spawn form later.

=> done

And now, we build capacity for building projects.

* [ ] allow for multiple turtles
    - [x] rework p5 function in `ludus.js` to properly parse commands targeted at different turtles
    - [x] write ludus code to spawn & manipulate turtle actors
    - [ ] write linking logic so we can `await` turtles
* [ ] do some sample multiturtle sketches
