Lexer

Noun · Development

Definitions

  1. The first stage of a compiler or interpreter that converts raw source code text into a stream of tokens (keywords, identifiers, operators, literals). Also called a tokenizer or scanner. Handles whitespace, comments, and string escaping. The output feeds into the parser which builds the AST.

    In plain English: The part of a compiler that breaks source code text into meaningful chunks (tokens) like keywords and numbers.

    Example: "The lexer turns if (x > 5) into tokens: [IF, LPAREN, IDENT(x), GT, NUMBER(5), RPAREN] for the parser to consume."

Related Terms