The chain

Learn tells you what you can rely on. This section tells you why, and in what order. None of it is needed to write a program: it is here for when you want to predict what the language will do.

1. source textwhat you wrote
2. lexthe language's symbols
3. tokenswords and symbols, no meaning yet
4. parse↳ desugarsymbols into op calls
5. raw programbindings and outputs, untyped
7. composethe language and the port layers
8. raw program + descriptorand everything it may use
9. analyse↳ prunewhat no output reaches
10. core programtyped, checked, pruned
11. evaluatethe input values
12. valuesone per output

One program, carried the whole way. Every artefact below is the real one, printed by the four scripts in packages/core/examples/3-(code)/, which run this program stage by stage:

let passing  = $score >= 60
let spare    = 99
output grade = If(passing, "Pass", "Fail")

Lex: characters become tokens

1. source textwhat you wrote
2. lexthe language's symbols
3. tokenswords and symbols, no meaning yet

ident:let  ident:passing  punct:=  punct:$  ident:score  punct:>=  number:60
ident:let  ident:spare  punct:=  number:99
ident:output  ident:grade  punct:=  ident:If  punct:(  ident:passing  punct:,
string:Pass  punct:,  string:Fail  punct:)  eof:

Two things in there are worth stopping on.

>= is one token. The lexer does not know what symbols exist; it is handed the symbol vocabulary of the language it is lexing, taken straight from the grammar’s registered symbols. Add a symbol and the lexer needs no edit. Feed the same text to a language with no symbols and >= comes back as two unknown characters.

$ and score are separate tokens. The sigil is structural punctuation, and the parser joins them. This is why $ cannot appear in a name and why an input is recognisable without consulting anything.

let, output and If are all plain ident tokens. There are no keywords at this level.

What this step reports: unknown_character and unterminated_string, and two warnings, unterminated_comment and invalid_escape. They are listed under Parse on Every diagnostic, because the lexer runs inside parseSource: a host never calls it on its own.

Parse: tokens become a raw program

1. tokenswords and symbols, no meaning yet
2. parse↳ desugarsymbols into op calls
3. raw programbindings and outputs, untyped

The result is a RawProgram: two maps, bindings and outputs, name to AST node. No types, no resolution, no checking.

let passing  -> operation
let spare    -> literal
output grade -> operation

Note "output": { "kind": "name", "name": "any" } on those nodes in the full dump: a raw node carries no inferred type. any is the placeholder, not a decision.

What this stage may not do: resolve a name, look up an op, or decide a type. It does not know what exists.

What it reports: syntax_error, unexpected_token, unexpected_end and duplicate_binding, and the warning deprecated_syntax, all under Parse.

Desugar: symbols become op calls

1. tokenswords and symbols, no meaning yet
2. parse↳ desugarsymbols into op calls
3. raw programbindings and outputs, untyped

Desugaring is not a pass of its own. It happens as the parser reads, which is why it leaves no artefact of its own behind: the raw program the parser hands on is already desugared.

passing being an operation is the interesting part. Here is its node:

{ "kind": "operation", "op": "Not",
  "inputs": { "a": { "kind": "operation", "op": "LessThan",
    "inputs": { "a": { "kind": "input", "name": "score" },
                "b": { "kind": "literal", "value": 60 } } } } }

The symbol is already gone. $score >= 60 became Not(LessThan($score, 60)) during parsing, because that is what the standard library registered >= to mean. There is no symbol node kind, and nothing after this point can tell that a symbol was written. It also explains why one symbol can expand into two op calls.

A template desugars the same way. `n = {count}` becomes Join(["n = ", count]), one operation node whose parts is a list of the text and the holes, and nothing downstream can tell that backticks were written. The safe cast is the exception that shows the rule: a type is not a value an op can take, so $x as number cannot become an op call, and the parser keeps it as a node kind of its own, cast, holding the value and the target type.

Desugaring reports nothing of its own. A symbol used on the wrong values is reported later, by the analyser, as a mistake in a call to the op it became.

Compose: the descriptor is assembled

1. raw programbindings and outputs, untyped
2. composethe language and the port layers
3. raw program + descriptorand everything it may use

The analyser checks a program against a descriptor: every op, type, input and output the program may use. Composing builds it, and it is the one step that does not read the program at all. It starts from the language’s vocabulary and adds the port layers in order, the host’s and then the document’s, checking each against everything before it. The same layers give the same descriptor for every program.

For our program that is one layer declaring $score as a number and grade as an output. The scripts do not print it: it is a map of the whole vocabulary, not something this program made.

What it reports: the eight kinds under Compose, from invalid_name to invalid_convert_input. Ports and layers has the rules it enforces.

Analyse: the raw program becomes a core program

1. raw program + descriptorand everything it may use
2. analyse↳ prunewhat no output reaches
3. core programtyped, checked, pruned

This is where the work is. Seven passes, in order:

1. Build the reference graph: which binding mentions which.
2. Topologically sort it, which is also where a cycle is found.
3. Compute reachability per output: which bindings can each one actually reach.
4. Analyse the bindings in dependency order, resolving types and checking every op call.
5. Validate the outputs against what the descriptor says they must be.
6. Prune the bindings no output reaches.
7. Warn about the ones that were pruned.

For our program the result is:

passing: boolean       dependsOn: [score]
output grade: string   dependsOn: [score]
warnings: unused_binding

Two things happened there, and a third that is the next section.

Types were resolved. passing is a boolean, worked out from Not’s output type, not from an annotation. grade is a string because If narrows to its branch type when both branches agree.

dependsOn was computed. Both carry [score], the set of inputs each reaches through the whole tree beneath it. This set is the entire basis of incremental evaluation, and it is computed once, here, statically.

Two rules this stage keeps:

• Errors are collected, never thrown. A failed binding is recorded in failedBindings so the names reading it fail cleanly rather than cascading into noise.
• Soundness is per output. Pass 5 decides output by output. One broken output is dropped; ok comes back false only if a required one was lost, which is the single case where the whole program fails.

The stage needs one thing besides the program: the descriptor that compose assembled.

What it reports: everything under Analyse, the largest group by far, from unknown_op to implicit_any_cast.

Prune: what no output reaches is dropped

1. raw program + descriptorand everything it may use
2. analyse↳ prunewhat no output reaches
3. core programtyped, checked, pruned

Pruning is the analyser’s last two passes, not a step after it, so the core program comes out already pruned.

spare is gone. No output can reach it, so pass 6 dropped it from the program the evaluator will see and pass 7 warned. The program that runs is not the program you wrote; it is the part of it that matters.

What it reports: unused_binding, a warning, once per binding it dropped. Nothing is an error here: a binding nobody reads cannot break anything.

Evaluate: the core program becomes values

1. core programtyped, checked, pruned
2. evaluatethe input values
3. valuesone per output

score=72 -> { "grade": "Pass" }
score=45 -> { "grade": "Fail" }

Outputs pull. Asking for grade walks down from its node, and each node consults a cache before computing. A node recomputes when the set of changed inputs intersects its dependsOn and it has no cache entry. Evaluation has the details.

What this stage may not do: report a type error. Everything checkable was checked. What reaches it is a runtime failure (a host evaluator throwing, a value that was not the shape its type claimed), and those arrive on the outputs, not as diagnostics. They are listed under Evaluate.

Why it is separate artefacts and not one pass

Each one is a thing you can hold, store, or inspect, which is what makes the language embeddable:

• The source is what gets saved. Not the AST (see persistence).
• The raw program is form-agnostic. A graph editor can produce one without any text.
• The core program is what runs, and it is disposable: it is rebuilt from the raw program whenever the descriptor changes, which is how a host adding an op makes old programs re-check rather than silently drift.
• The values are the only thing a host acts on.