"Literate" Ludus #98

New Issue

scott · 2024-07-13T04:48:57Z

scott commented

2024-07-13 04:48:57 +00:00

The idea here is that there should be some kind of document that includes prose, code, and some other stuff. The prose should be readable, the code should be runnable, and the other stuff should do what we need it to.

The idea is, perhaps, something like Observable, like a Jupyter notebook, or like Literate Scribble.

A few thoughts come to mind immediately:

This should have a canonical textual format, as a hybrid of some markup textual format and code.
There should be an interactive interface to this textual format, which allows for the editing and running of code.
The textual format and interactive interface ought to meet the affordances of the different modalities we're working in. That means chapters, sections, code blocks, footnotes, references, links, turtle graphics, and images (at least).
The interactive interface ought to be extremely usable and lovely: people will be reading a book-length thing in it.
The interactive interface ought to enhance book reading: not just straight reproduction of prose, plus code, but excellent structured navigation, text searchability, affordance discoverability, etc.
The interactive interface ought to be web-based, not proprietary software.

A few open questions:

It is not yet clear whether the interactive interface should allow actual editing of the underlying textual document, or whether it is functionally equivalent to the current Ludus web interface, where changes are limited to a particular instance, locally in a web browser (or similar). (In other words, is the interactive interface effectively a reader for a notebook-style document, or is it a document editing tool.)
It is not yet clear how integrated into Ludus the literate format should be. In other words, how much does Ludus need or want to know, vs. a web frontend? Does the interface split a single file into markdown and code, and Ludus itself only needs to know about the code.
It is not at all clear to me whether it's within our powers to actually build this system from scratch. I want to be able to, but there are perhaps integrated systems that can be bent to our will (Replit, StackBlitz). That said, the less the interface does, the less complex it will be, so there's a balance between capability and use.

From a design perspective, the obvious first place is a canonical, textual document that can be written, put under source control, etc.

Three possibilities come to mind right away:

First, just use a (very lightly modified?) markdown format & file:

This is _markdown_ **text**.
* first
* [second](./second.md)
* ![alt text](./an-image.png)

```block-name
& this is Ludus code
let foo = :bar
```

This is more markdown.

This has a few advantages: you can just parse the thing with your usual markdown parser; markdown is readable as text; it's a widely-known standard, and thus highly searchable; it's text-forward, which is to say, you just start writing text, and code becomes the special case, which is nice for authoring things.

Additionally, this allows for different blocks to have names. This may or may not be an advantage.

Second, you could modify Ludus to include markdown in an ergonomic way:

&&&
# A document title
_This_ is **markdown** text.
&&&
let foo = :bar & this is Ludus code
&&&
This is more `markdown`.

This also has a few advantages. With the "block comment" &&&-fencing strategy, all Ludus notebooks are just Ludus files. This comes with many of the same benefits as the first "just markdown" strategy.

These have one major difference between them. In the first, you can see the named Ludus block. The idea here is that there's a single markdown file (the "toplevel" is textual, as it were), which includes multiple Ludus blocks, which presumably each get their own scope. In the second, the toplevel is Ludus: all the code is a single script, and thus executed all at once.

It's not at all clear to me which is better. Clearly the "separate code blocks" is more powerful in some sense, although it does require an engine of some kind beyond a Ludus interpreter to manage the relationships between blocks. Meanwhile, the "single script" keeps things less abstracted and complex.

Third, there's the possibility of creating a system from scratch, maybe something like Racket's Scribble (linked above). This, of course, allows for a more flexible system, that may well make things like references or marginal comments, or similarly not-web-standard textual factors far easier to deal or work with.

Clearly, the design space is huge. We could go anywhere from a few extensions to markdown to full-on DSL territory. What would a Ludus-ish DSL look like in this case? What would the principles be?

One simple version could be something akin to Clojure's hiccup, but with semantic-ish meanings rather than mapping to HTML. So, e.g.,

[:title My title]
[:sub My subtitle]
This is plain text. This is [:italic excitingly] [:bold strongly] formatted text.

[:ludus #{:id :one}
let foo = :bar
let bar = :baz
]

[:ludus #{:id :two}
let quux = 23
let frob = 42
]

[:drawing #{:id :my_drawing :height 23 :width 42}]

[:image #{:url https://myurl.ca/img.png :height 42 :width 23}]

Here is more text, lorem ipsum and so on.

You get the gist. These could be "tuples" or "lists" (parens or brackets; probably brackets, since they're less common in prose), but the thought is that something like this is extremely straightforward to parse, and extremely straightforward to render into HTML (which will be our target).

In other words, while there's a set of design problems here, to look into (using Scribble, Hiccup, Markdown as having solved related problems), the technical issues for this textual format are actually extremely straightforward: parsing is easy, outputting HTML is easy, CSS formatting is easy, the set of things we'd want & need are domain specific and fairly restricted.

(Also, there's a big benefit here to the idea that, by swapping out bare text with strings, we'd have something like a basic HTML library for Ludus that models markup with lists and keywords and dicts. That's a nice touch.)

That said (and still in other words), the more difficult design issues is the mapping between an HTML front-end that renders the textual format (say, .ldn, for Ludus notebook), and the actual textual back-end. Where does interactivity lie? Do we try to make the different code blocks hot-reloading? Or do we follow Ludus's extremely strict approach to mutation and change? How do we think about and model, in code and on the page, different turtle graphics drawings? Different consoles? Or do we have a code-and-prose pane on the left, and drawings-and-console pane on the right, very much like we have now?

The idea here is that there should be some kind of document that includes prose, code, and some other stuff. The prose should be readable, the code should be runnable, and the other stuff should do what we need it to. The idea is, perhaps, something like [Observable](https://observablehq.com), like a [Jupyter](https://jupyter.org) notebook, or like [Literate Scribble](https://docs.racket-lang.org/scribble/lp.html). A few thoughts come to mind immediately: 1. This should have a canonical textual format, as a hybrid of some markup textual format and code. 2. There should be an interactive interface to this textual format, which allows for the editing and running of code. 3. The textual format and interactive interface ought to meet the affordances of the different modalities we're working in. That means chapters, sections, code blocks, footnotes, references, links, turtle graphics, and images (at least). 4. The interactive interface ought to be extremely usable and lovely: people will be reading a book-length thing in it. 5. The interactive interface ought to _enhance_ book reading: not just straight reproduction of prose, plus code, but excellent structured navigation, text searchability, affordance discoverability, etc. 6. The interactive interface ought to be web-based, not proprietary software. A few open questions: 1. It is not yet clear whether the interactive interface should allow actual editing of the underlying textual document, or whether it is functionally equivalent to the current Ludus web interface, where changes are limited to a particular instance, locally in a web browser (or similar). (In other words, is the interactive interface effectively a _reader_ for a notebook-style document, or is it a document editing tool.) 2. It is not yet clear how integrated into Ludus the literate format should be. In other words, how much does Ludus need or want to know, vs. a web frontend? Does the interface split a single file into markdown and code, and Ludus itself only needs to know about the code. 3. It is not at all clear to me whether it's within our powers to actually build this system from scratch. I want to be able to, but there are perhaps integrated systems that can be bent to our will (Replit, StackBlitz). That said, the less the interface does, the less complex it will be, so there's a balance between capability and use. --- From a design perspective, the obvious first place is a canonical, textual document that can be written, put under source control, etc. Three possibilities come to mind right away: **First,** just use a (very lightly modified?) markdown format & file: ```` This is _markdown_ **text**. * first * [second](./second.md) * ![alt text](./an-image.png) ```block-name & this is Ludus code let foo = :bar ``` This is more markdown. ```` This has a few advantages: you can just parse the thing with your usual markdown parser; markdown is readable as text; it's a widely-known standard, and thus highly searchable; it's text-forward, which is to say, you just start writing text, and code becomes the special case, which is nice for authoring things. Additionally, this allows for different blocks to have names. This may or may not be an advantage. **Second,** you could modify Ludus to include markdown in an ergonomic way: ``` &&& # A document title _This_ is **markdown** text. &&& let foo = :bar & this is Ludus code &&& This is more `markdown`. ``` This also has a few advantages. With the "block comment" `&&&`-fencing strategy, all Ludus notebooks are just Ludus files. This comes with many of the same benefits as the first "just markdown" strategy. These have one major difference between them. In the first, you can see the named Ludus block. The idea here is that there's a single markdown file (the "toplevel" is textual, as it were), which includes multiple Ludus blocks, which presumably each get their own scope. In the second, the toplevel is Ludus: all the code is a single script, and thus executed all at once. It's not at all clear to me which is better. Clearly the "separate code blocks" is more powerful in some sense, although it does require an engine of some kind beyond a Ludus interpreter to manage the relationships between blocks. Meanwhile, the "single script" keeps things less abstracted and complex. **Third,** there's the possibility of creating a system from scratch, maybe something like Racket's Scribble (linked above). This, of course, allows for a more flexible system, that may well make things like references or marginal comments, or similarly not-web-standard textual factors far easier to deal or work with. Clearly, the design space is huge. We could go anywhere from a few extensions to markdown to full-on DSL territory. What would a Ludus-ish DSL look like in this case? What would the principles be? One simple version could be something akin to Clojure's [hiccup](https://github.com/weavejester/hiccup), but with semantic-ish meanings rather than mapping to HTML. So, e.g., ``` [:title My title] [:sub My subtitle] This is plain text. This is [:italic excitingly] [:bold strongly] formatted text. [:ludus #{:id :one} let foo = :bar let bar = :baz ] [:ludus #{:id :two} let quux = 23 let frob = 42 ] [:drawing #{:id :my_drawing :height 23 :width 42}] [:image #{:url https://myurl.ca/img.png :height 42 :width 23}] Here is more text, lorem ipsum and so on. ``` You get the gist. These could be "tuples" or "lists" (parens or brackets; probably brackets, since they're less common in prose), but the thought is that something like this is extremely straightforward to parse, and extremely straightforward to render into HTML (which will be our target). --- In other words, while there's a set of design problems here, to look into (using Scribble, Hiccup, Markdown as having solved related problems), the technical issues for this textual format are actually extremely straightforward: parsing is easy, outputting HTML is easy, CSS formatting is easy, the set of things we'd want & need are domain specific and fairly restricted. (Also, there's a big benefit here to the idea that, by swapping out bare text with strings, we'd have something like a basic HTML library for Ludus that models markup with lists and keywords and dicts. That's a nice touch.) That said (and still in other words), the more difficult design issues is the mapping between an HTML front-end that renders the textual format (say, `.ldn`, for Ludus notebook), and the actual textual back-end. Where does interactivity lie? Do we try to make the different code blocks hot-reloading? Or do we follow Ludus's extremely strict approach to mutation and change? How do we think about and model, in code and on the page, different turtle graphics drawings? Different consoles? Or do we have a code-and-prose pane on the left, and drawings-and-console pane on the right, very much like we have now?

scott added the

enhancement

research

ux

labels 2024-07-13 04:49:25 +00:00

scott commented

2024-07-19 14:43:32 +00:00

Conversation with @matt: Looks like the first version of "Ludus as a host in markdown" is the most promising approach here: it will already work with all the parsers.

Also, perhaps we have a version of Ludus shebang in a markdown file that means that Ludus can know this is executable as Ludus.

Conversation with @matt: Looks like the first version of "Ludus as a host in markdown" is the most promising approach here: it will already work with all the parsers. Also, perhaps we have a version of Ludus shebang in a markdown file that means that Ludus can know this is executable as Ludus.

matt commented

2024-07-19 14:52:21 +00:00

For reference: https://observablehq.com/framework/

Observable's open source SSG based on markdown

For reference: https://observablehq.com/framework/ Observable's open source SSG based on markdown

scott commented

2024-07-19 14:59:13 +00:00

Next steps:

Turtle graphics protocol v. 0.1.
Specification of a "Literate Ludus" markdown format. (YAML/TOML front matter is probably the way to go here.)
Development of various parsers/frontends for textual .md Ludus notebooks:
a. Command line execution
b. web.ludus.dev notebook format
c. Test against pandoc output

Next steps: 1. Turtle graphics protocol v. 0.1. 2. Specification of a "Literate Ludus" markdown format. (YAML/TOML front matter is probably the way to go here.) 3. Development of various parsers/frontends for textual .md Ludus notebooks: a. Command line execution b. web.ludus.dev notebook format c. Test against pandoc output

scott added this to the Literate Ludus project 2024-07-21 20:19:23 +00:00

Sign in to join this conversation.