Info Catalog wisent: Grammar styles wisent: Wisent Semantic

wisent: Wisent Lex

 
 4.2 The Wisent Lex lexer
 ========================
 
 The lexical analysis step of Semantic is performed by the general
 function ‘semantic-lex’.  For more information, 
 (semantic-langdev)Writing Lexers.
 
    ‘semantic-lex’ produces lexical tokens of the form:
 
      (TOKEN-CLASS START . END)
 
 TOKEN-CLASS
      Is a symbol that identifies a lexical token class, like ‘symbol’,
      ‘string’, ‘number’, or ‘PAREN_BLOCK’.
 
 START
 END
      Are the start and end positions of mapped data in the input buffer.
 
    The Wisent’s parser doesn’t depend on the nature of analyzed input
 stream (buffer, string, etc.), and requires that lexical tokens have a
 different form (Writing a lexer):
 
      (TOKEN-CLASS VALUE [START . END])
 
    ‘wisent-lex’ is the default Wisent’s lexer used in Semantic.
 
  -- Function: wisent-lex
      Return the next available lexical token in Wisent’s form.
 
      The variable ‘wisent-lex-istream’ contains the list of lexical
      tokens produced by ‘semantic-lex’.  Pop the next token available
      and convert it to a form suitable for the Wisent’s parser.
 
    Mapping of lexical tokens as produced by ‘semantic-lex’ into
 equivalent Wisent lexical tokens is straightforward:
 
      (TOKEN-CLASS START . END)
           ⇒ (TOKEN-CLASS VALUE START . END)
 
    VALUE is the input ‘buffer-substring’ from START to END.
 

Info Catalog wisent: Grammar styles wisent: Wisent Semantic