Module highlight 

Colouring source the way the compiler reads it, and a disassembly the way its printer writes it.

Why there is no tokenizer in the page

The obvious way to highlight a <textarea> is a regular expression and a list of keywords in JavaScript. It is also the way that goes wrong silently: the list is a second, informal specification of the language, nothing compares it against the first, and the day opaque or ..< is added the page keeps colouring the old language and no test anywhere has an opinion. A playground that lied about what Cove is would be worse than one that showed plain text.

The module a page already loads contains the whole front end, so the lexer is there. paint calls it. The keywords are cove_syntax::token::Keyword’s, the numbers are the ones cove_syntax::lexer accepts including 500ms, and a token kind the language grows arrives here as a token kind. Agreement is by construction rather than by discipline.

What comes out

A tiling: a list of pieces that between them cover every UTF-16 code unit of the source exactly once, in order. The page renders it by walking the list and slicing the text — no offsets to reconcile, no gaps to guess, and a check that the pieces tile is a check the whole thing is sound.

UTF-16 and not bytes because the consumer is JavaScript, where a string is indexed in UTF-16 code units. Two of the shipped samples contain an em dash, so this is not a hypothetical: byte offsets would have misaligned every colour after the first —.

Six categories for source, and the two that are not token kinds

Kind is deliberately short. A playground wants a reader to see the shape of a program, and twenty colours is a wall rather than a shape. Six of its seven are what source is cut into; the seventh, slot, belongs to the disassembly below and is argued there.

Two of the six do not come from a TokenKind, and both are named here because they are the parts a reader should check rather than trust:

• type is an identifier that begins with an uppercase letter. The lexer does not know what a type is — it answers TokenKind::Ident for Int and for total alike. Uppercase is not only a convention in Cove, though: the parser’s parse_pattern decides that Ok(value) is a variant and other is a binding on exactly this rule, so the page is colouring by something the grammar already reads. It is still a heuristic about names, and a struct someone called Total is coloured as a type because it is one.

• comment for a // or /* */ comment, which is not a token at all: the lexer discards them. They are recovered from the gaps between tokens, which in a source that lexes hold nothing else. The rule and its one wrong answer are written out where it is applied.

Source that does not lex

Constantly, because the reader is typing. One open quote and the file has a lexical error, and that is the state a string literal is in for as long as it takes to write one.

cove_syntax::lexer::lex_recovered exists for this: the tokens before the error are real tokens and are answered, so the colouring is of the text that is actually in the box rather than a stale picture of the text that was there two keystrokes ago. ok reports whether the source lexed cleanly, and it is the page’s text either way — nothing here ever answers a tiling of something the caller did not send.

The other text on the page: a disassembly

disassembly colours what cove_ir::print emits, and it exists for the reason above rather than in spite of it. The page shows that text already; a wall of one colour is what it was. The obvious way to fix that would have been a regular expression in the page, and it would have been the same mistake: an informal second reader of a format, drifting from the format with nothing watching.

What is honest to say is that this is a second reader. There is no lexer for a disassembly to borrow, and making the printer emit spans would mean rewriting two hundred lines of format! in cove-ir for a colour on a playground pane. So the reader is here, in Rust, next to the crate that prints the text and inside the module the page already takes tilings from, and the drift is caught by a test rather than by discipline: web/check.mjs colours the real disassembly of all nine shipped samples and fails the build if a single line of any of them is one this reader does not recognise. That is what ok is for here — not “the text is well formed”, which it always is, but “every line of it was a line shape cove_ir::print documents”.

It reads the six line shapes that module writes and nothing about any particular instruction: a header, a frame, a capture, a local, a blank line, and pc opcode operands. Inside an instruction it goes by the shape of each token — s3:int is a slot, "…" is a literal, digits are a number, a name before ( is a callee and any other name is a layout. Adding an instruction to the language therefore needs nothing here; changing how operands are written does, and that is what the nine samples are asserted for.

Structs

Painting

A tiling of one text, and whether the thing that read it had a complaint.

Piece

One run of text that is all one colour.

Enums

Kind

What one piece of a text is coloured as.

Functions

disassembly

Colours a disassembly and answers a colour for every part of it.

paint

Lexes source and answers a colour for every part of it.