Lexical Analyzer


Lexical Analyzer

A Lexical Analyzer:
  1. Ignores comments
  2. Ignores spaces, tabs, line returns and line feeds
  3. Identifies reserved words (keywords)
  4. Identifies separators such as: (, [, {, etc.
  5. Identifies values: numeric, strings, etc.
The Wintempla lexical analyzer can be used to separate code in tokens. The Cpl::LexicalAnalyzer::Token structure holds information about each token. Wintempla has three lexical analyzers:
  1. C/C++
  2. SQL/PLSQL (Sql::LexicalAnalyzer)
  3. HTML (Web::LexicalAnalyzer)
You can use the Cpl::LexicalAnalizer class as a base class to create your own lexical analyzer using inheritance. You may need to override the virtual methods. As a matter of fact: the SQL lexical analyzer is derived from the Cpl::LexicalAnalizer class.

LexicalAnalyzer

Token Type

There are several token types. You can inspect the Cpl::LexicalAnalyzer::GetTokenText function to learn about the lexical analyzer. Token types are defined in the Wintempla.h file, they start with LEX_The Wintempla lexical analyzer supports:
  • Most data types, such as: double, int, etc.
  • Neural Lab data types, such as: LayerNet, ProbNet, etc,
  • Most control sentences, such as: if, if-else, for, while, do-while, break, continue, (it does not support switch, but it can be implemented using if-else)

SQL Lexical Analyzer

It is a lexical analyzer specialized in SQL code. You can find the Wintempla SQL lexical analyzer in the SQL namespace. It supports most SQL data types and keywords, such as: SELECT, FROM, DELETE, etc.

Debug

To debug the Wintempla lexical analyzer, you must use the Cpl::LexicalAnalyzer::Debug function. You can use the debug function to find errors and understand how the lexical analyzer operates. To debug the lexical analyzer, you must:
  1. Call Cpl::LexicalAnalyzer::Create
  2. Call Cpl::LexicalAnalyzer::Debug

Running the lexical analyzer

To run the Wintempla lexical analyzer, you must use the Cpl::LexicalAnalyzer::GetNextToken function. The code below shows how to run the lexical analyzer. To run lexical analyzer you must:
  1. Call Cpl::LexicalAnalyzer::Create
  2. Call Cpl::LexicalAnalyzer::GetNextToken

Program.cpp
void Program::Window_Open(Win::Event& e)
{
     Cpl::LexicalAnalyzer::Token lookahead;
     Cpl::LexicalAnalyzer lex;

     if (lex.Create(tbxInput.Text.c_str())==false)
     {
          // Display error
     }
     do
     {
          lex.GetNextToken(lookahead);
          // Do something with lookahead
     }
     while (lookahead.type == LEX_END_OF_FILE);
}


Problem 1
Create a Wintempla dialog application called MyLexical to test the lexical analyzer. Using Wintempla add two textboxes (tbxInput and tbxOutput) with the multi-line property, vertical scrollbars and horizontal scrollbar. Set the event of "Change" in the tbxInput textbox. Set also the "Want return" property in the tbxInput textbox.

tbxInputProperties

Scrollbars

ChangeEvent

MyLexical.cpp
...

void Program::Window_Open(Win::Event& e)
{
}

void MyLexical::tbxInput_Change(Win::Event& e)
{
     Cpl::LexicalAnalyzer lex;
     lex.Create(tbxInput.Text.c_str());
     //
     wstring output = L"Line\tstring\t\tdouble\t\tint/bool \t\ttype\r\n";
     Cpl::LexicalAnalyzer::Token token;
     wstring text;
     while (true)
     {
          lex.GetNextToken(token);
          if (token.type == LEX_END_OF_FILE) break;
          //
          Sys::Format(text, L"%d\t%s\t\t%g\t\t%d\t\t%s\r\n",
               token.line_number, token.string_value, token.double_value, token.int_value, Cpl::LexicalAnalyzer::GetTokenConstant(token.type));
          output += text;
     }
     tbxOutput.Text = output;
}

MyLexicalRun

Inserting tabs in a textbox

By default a textbox does not accept tabs. To learn how to setup a textbox to allow the input of tabs see Wintempla>OOP>Sub-classing.
Por defecto una caja de texto no acepta tabulaciones. Para aprender como configurar una caja de texto para que permita la entrada de tabulaciones vea Wintempla>OOP>Sub-classing.

Problem 2
Create a Wintempla dialog application called HtmlAnalysis to test the HTML lexical analyzer. Using Wintempla add two textboxes (tbxInput and tbxOutput) with the multi-line property, vertical scrollbars and horizontal scrollbar. Set the event of "Change" in the tbxInput textbox. Set also the "Want return" property in the tbxInput textbox.

HtmlAnalysis.cpp
...
void HtmlAnalysis::tbxInput_Change(Win::Event& e)
{
     Web::LexicalAnalyzer lex;
     lex.Create(tbxInput.Text.c_str());
     //
     wstring output = L"Line\tstring\t\tdouble\t\tint/bool \t\ttype\r\n";
     Cpl::LexicalAnalyzer::Token token;
     wstring text;
     while (true)
     {
          lex.GetNextToken(token);
          if (token.type == LEX_END_OF_FILE) break;
          //
          Sys::Format(text, L"%d\t%s\t\t%g\t\t%d\t\t%s\r\n",
               token.line_number, token.string_value, token.double_value, token.int_value, Cpl::LexicalAnalyzer::GetTokenConstant(token.type));
          output += text;
     }
     tbxOutput.Text = output;
}

HtmlAnalysisRun

Problem 3
Create a Wintempla dialog application called SqlAnalysis to test the SQL lexical analyzer. Using Wintempla add two textboxes (tbxInput and tbxOutput) with the multi-line property, vertical scrollbars and horizontal scrollbar. Set the event of "Change" in the tbxInput textbox. Set also the "Want return" property in the tbxInput textbox.

SqlAnalysis.cpp
...
void SqlAnalysis::tbxInput_Change(Win::Event& e)
{
     Sql::LexicalAnalyzer lex;
     lex.Create(tbxInput.Text.c_str());
     //
     wstring output = L"Line\tstring\t\tdouble\t\tint/bool \t\ttype\r\n";
     Cpl::LexicalAnalyzer::Token token;
     wstring text;
     while (true)
     {
          lex.GetNextToken(token);
          if (token.type == LEX_END_OF_FILE) break;
          //
          Sys::Format(text, L"%d\t%s\t\t%g\t\t%d\t\t%s\r\n",
               token.line_number, token.string_value, token.double_value, token.int_value, Cpl::LexicalAnalyzer::GetTokenConstant(token.type));
          output += text;
     }
     tbxOutput.Text = output;
}


SqlAnalysisRun

© Copyright 2000-2019 Wintempla selo. All Rights Reserved. Sep 05 2019. Home