pub fn lex(
sources: &SourceMap,
file: FileId,
) -> Result<Vec<Token>, Vec<Diagnostic>>Expand description
Lexes file out of sources into a token stream.
The returned stream always ends with exactly one TokenKind::Eof whose
span is the empty range at end of file. On any lexical error, every error
found in the file is collected and returned; no tokens are produced in
that case.