pub fn lex_recovered(
sources: &SourceMap,
file: FileId,
) -> (Vec<Token>, Vec<Diagnostic>)Expand description
Lexes file and answers both halves of what the lexer found: the tokens it
recovered and everything it complained about.
lex is this function with the tokens thrown away whenever there is a
diagnostic, which is the right contract for a compiler — a parser handed a
token stream with a hole in it reports a second, invented error at the
hole. It is the wrong contract for a reader looking at the text. The
recovery this exposes is not new: the lexer has always skipped past a
lexical error and carried on so that one call reports every problem in a
file, and these are the tokens that pass produced.
The caller is what makes the difference. crates/cove-wasm’s highlighter
colours source that is being typed, which is a state that does not lex for
most of the time a string literal is being written, and a highlighter that
had nothing to say about a file with one open quote in it would have
nothing to say most of the time.
The tokens are the same tokens lex would have answered had there been
no diagnostic; nothing is invented to fill a gap. Text the lexer skipped is
simply absent from the stream, so a caller that cares what is between two
tokens must look at the source.