Java Code: For Lexer.java Make a HashMap of in your Lex class. Below is a list of the keywords that you need. Make token types and populate the hash map in your constructor (I would make a helper method that the constructor calls). while, if, do, for, break, continue, else, return, BEGIN, END, print, printf, next, in, delete, getline, exit, nextfile, function Modify “ProcessWord” so that it checks the hash map for known words and makes a token specific to the word with no value if the word is in the hash map, but WORD otherwise. For example, Input: for while hello do Output: FOR WHILE WORD(hello) DO Make a token type for string literals. In Lex, when you encounter a “, call a new method (I called it HandleStringLiteral() ) that reads up through the matching “ and creates a string literal token ( STRINGLITERAL(hello world) ). Be careful of two things: make sure that an empty string literal ( “” ) works and make sure to deal with escaped “ (String quote = “She said, \”hello there\” and then she left.”;) Make a new token type, a new method (HandlePattern) and call it from Lex when you encounter a backtick. The last thing that we need to deal with in our lexer is symbols. Most of these will be familiar from Java, but a few I will detail a bit more. We will be using two different hash maps – one for two-character symbols (like ==, &&, ++) and one for one character symbols (like +, -, $). Why? Well, some two-character symbols start with characters that are also symbols (for example, + and +=). We need to prioritize the += and only match + if it is not a +=. Create a method called “ProcessSymbol” – it should use PeekString to get 2 characters and look them up in the two-character hash map. If it exists, make the appropriate token and return it. Otherwise, use PeekString to get a 1 character string. Look that up in the one-character hash map. If it exists, create the appropriate token and return it. Don’t forget to update the position in the line. If no symbol is found, return null. Call ProcessSymbol in your lex() method. If it returns a value, add the token to the token list. Make sure all the functionality of the unit test are tested. Below are lexer.java and token.java. Make sure to show full code with screenshot of the output. Attached is checklist.  Lexer.java import java.util.LinkedList;   public class Lexer { private StringHandler stringHandler;     private int lineNumber;     private int charPosition;              public Lexer(String input) {         stringHandler = new StringHandler(input);         lineNumber = 1;         charPosition = 0;     }       public LinkedList lex() {         LinkedList tokens = new LinkedList<>();                        while (!stringHandler.isDone()) {             char c = stringHandler.peek(0);               if (c == ' ' || c == '\t') {                 stringHandler.swallow(1);                 charPosition++;             } else if (c == '\n') {                 tokens.add(new Token(TokenType.SEPARATOR, lineNumber, charPosition));                 stringHandler.swallow(1);                 lineNumber++;                 charPosition = 0;                             } else if (Character.isLetter(c)) {                 tokens.add(processWord());             } else if (Character.isDigit(c)) {                 tokens.add(processNumber());             } else {                 throw new RuntimeException("Unrecognized character: " + c);             }         }           return tokens;     }       private Token processWord() {         StringBuilder value = new StringBuilder();         while (!stringHandler.isDone() && (Character.isLetterOrDigit(stringHandler.peek(0)) || stringHandler.peek(0) == '_' || stringHandler.peek(0) == ',')) {             value.append(stringHandler.getChar());             charPosition++;         }         return new Token(TokenType.WORD, value.toString(), lineNumber, charPosition - value.length());     }           private Token processNumber() {         StringBuilder value = new StringBuilder();         while (!stringHandler.isDone() && (Character.isDigit(stringHandler.peek(0)) || stringHandler.peek(0) == '.')) {             value.append(stringHandler.getChar());             charPosition++;         }         return new Token(TokenType.NUMBER, value.toString(), lineNumber, charPosition - value.length());     } }         Token.java   enum TokenType {     WORD, NUMBER, SEPARATOR } public class Token { private TokenType type;     private String value;     private int lineNumber;     private int charPosition;         public Token(TokenType type, int lineNumber, int charPosition) {         this.type = type;         this.lineNumber = lineNumber;         this.charPosition = charPosition;     }         public Token(TokenType type, String value, int lineNumber, int charPosition) {         this(type, lineNumber, charPosition);         this.value = value;     }       public String toString() {         if (value == null) {             return type.toString();         } else {             return type.toString() + "(" + value + ")";         }     } }

Database System Concepts
7th Edition
ISBN:9780078022159
Author:Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Publisher:Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Chapter1: Introduction
Section: Chapter Questions
Problem 1PE
icon
Related questions
Question

Java Code: For Lexer.java

Make a HashMap of <String, TokenType> in your Lex class. Below is a list of the keywords that you need. Make token types and populate the hash map in your constructor (I would make a helper method that the constructor calls).

while, if, do, for, break, continue, else, return, BEGIN, END, print, printf, next, in, delete, getline, exit, nextfile, function

Modify “ProcessWord” so that it checks the hash map for known words and makes a token specific to the word with no value if the word is in the hash map, but WORD otherwise.

For example,

Input: for while hello do

Output: FOR WHILE WORD(hello) DO

Make a token type for string literals. In Lex, when you encounter a “, call a new method (I called it HandleStringLiteral() ) that reads up through the matching “ and creates a string literal token ( STRINGLITERAL(hello world) ). Be careful of two things: make sure that an empty string literal ( “” ) works and make sure to deal with escaped “ (String quote = “She said, \”hello there\” and then she left.”;)

Make a new token type, a new method (HandlePattern) and call it from Lex when you encounter a backtick.

The last thing that we need to deal with in our lexer is symbols. Most of these will be familiar from Java, but a few I will detail a bit more. We will be using two different hash maps – one for two-character symbols (like ==, &&, ++) and one for one character symbols (like +, -, $). Why? Well, some two-character symbols start with characters that are also symbols (for example, + and +=). We need to prioritize the += and only match + if it is not a +=.

Create a method called “ProcessSymbol” – it should use PeekString to get 2 characters and look them up in the two-character hash map. If it exists, make the appropriate token and return it. Otherwise, use PeekString to get a 1 character string. Look that up in the one-character hash map. If it exists, create the appropriate token and return it. Don’t forget to update the position in the line. If no symbol is found, return null. Call ProcessSymbol in your lex() method. If it returns a value, add the token to the token list.

Make sure all the functionality of the unit test are tested. Below are lexer.java and token.java. Make sure to show full code with screenshot of the output. Attached is checklist. 

Lexer.java

import java.util.LinkedList;
 
public class Lexer {
private StringHandler stringHandler;
    private int lineNumber;
    private int charPosition;
    
 
 
    public Lexer(String input) {
        stringHandler = new StringHandler(input);
        lineNumber = 1;
        charPosition = 0;
    }
 
    public LinkedList<Token> lex() {
        LinkedList<Token> tokens = new LinkedList<>();
        
     
        while (!stringHandler.isDone()) {
            char c = stringHandler.peek(0);
 
            if (c == ' ' || c == '\t') {
                stringHandler.swallow(1);
                charPosition++;
            } else if (c == '\n') {
                tokens.add(new Token(TokenType.SEPARATOR, lineNumber, charPosition));
                stringHandler.swallow(1);
                lineNumber++;
                charPosition = 0;
               
            } else if (Character.isLetter(c)) {
                tokens.add(processWord());
            } else if (Character.isDigit(c)) {
                tokens.add(processNumber());
            } else {
                throw new RuntimeException("Unrecognized character: " + c);
            }
        }
 
        return tokens;
    }
 
    private Token processWord() {
        StringBuilder value = new StringBuilder();
        while (!stringHandler.isDone() && (Character.isLetterOrDigit(stringHandler.peek(0)) || stringHandler.peek(0) == '_' || stringHandler.peek(0) == ',')) {
            value.append(stringHandler.getChar());
            charPosition++;
        }
        return new Token(TokenType.WORD, value.toString(), lineNumber, charPosition - value.length());
    }
 
   
    private Token processNumber() {
        StringBuilder value = new StringBuilder();
        while (!stringHandler.isDone() && (Character.isDigit(stringHandler.peek(0)) || stringHandler.peek(0) == '.')) {
            value.append(stringHandler.getChar());
            charPosition++;
        }
        return new Token(TokenType.NUMBER, value.toString(), lineNumber, charPosition - value.length());
    }
}
 
 
 
 
Token.java
 
enum TokenType {
    WORD, NUMBER, SEPARATOR
}
public class Token {
private TokenType type;
    private String value;
    private int lineNumber;
    private int charPosition;
 
 
    public Token(TokenType type, int lineNumber, int charPosition) {
        this.type = type;
        this.lineNumber = lineNumber;
        this.charPosition = charPosition;
    }
 
 
    public Token(TokenType type, String value, int lineNumber, int charPosition) {
        this(type, lineNumber, charPosition);
        this.value = value;
    }
 
    public String toString() {
        if (value == null) {
            return type.toString();
        } else {
            return type.toString() + "(" + value + ")";
        }
    }
}
 
Input: for while hello do
Output: FOR WHILE WORD (hello) DO
Transcribed Image Text:Input: for while hello do Output: FOR WHILE WORD (hello) DO
Code Style
Unit Tests
Keywords
Comments
StringLiteral
Patterns
Symbol - One
Character
Symbol - Two
Character
ProcessSymbol
Some good
naming, some
necessary
comments (3)
Don't exist (0) At least one (6)
Few
comments,
bad names (0)
Unmodified
(0)
Not skipped
(0)
Not processed
(0)
Processed ad
hoc (3)
No hashmap
(0)
No hashmap
(0)
None (0)
Token type
exists (3)
Not processed Token type
(0)
exists (3)
Mostly good
naming, most
necessary
comments (6)
Missing tests
(12)
Hashmap
created and
populated (6)
Token type
exists and
Method exists
and is called (6)
Token type
exists and
Method exists
and is called (6)
Hash map exists Most symbols
(3)
exist (6)
Hash map exists Most symbols
(3)
exist (6)
Exists, looks in
one map (3)
Exists, looks in
both maps (6)
Good naming, non-trivial
methods well commented,
static only when necessary,
private members (10)
All functionality tested (20)
Keywords properly
recognized and proper
tokens created (15)
Rest of line ignored (5)
String literals properly
processed into their own
tokens; line count and
column updated (10)
Patterns properly
processed into their own
tokens; line count and
column updated (10)
All symbols exist with
reasonable token types (10)
All symbols exist with
reasonable token types (10)
Exists, looks in twoChar,
then oneChar, updates
StringManager and position
correctly (10)
Transcribed Image Text:Code Style Unit Tests Keywords Comments StringLiteral Patterns Symbol - One Character Symbol - Two Character ProcessSymbol Some good naming, some necessary comments (3) Don't exist (0) At least one (6) Few comments, bad names (0) Unmodified (0) Not skipped (0) Not processed (0) Processed ad hoc (3) No hashmap (0) No hashmap (0) None (0) Token type exists (3) Not processed Token type (0) exists (3) Mostly good naming, most necessary comments (6) Missing tests (12) Hashmap created and populated (6) Token type exists and Method exists and is called (6) Token type exists and Method exists and is called (6) Hash map exists Most symbols (3) exist (6) Hash map exists Most symbols (3) exist (6) Exists, looks in one map (3) Exists, looks in both maps (6) Good naming, non-trivial methods well commented, static only when necessary, private members (10) All functionality tested (20) Keywords properly recognized and proper tokens created (15) Rest of line ignored (5) String literals properly processed into their own tokens; line count and column updated (10) Patterns properly processed into their own tokens; line count and column updated (10) All symbols exist with reasonable token types (10) All symbols exist with reasonable token types (10) Exists, looks in twoChar, then oneChar, updates StringManager and position correctly (10)
Expert Solution
trending now

Trending now

This is a popular solution!

steps

Step by step

Solved in 6 steps with 1 images

Blurred answer
Follow-up Questions
Read through expert solutions to related follow-up questions below.
Follow-up Question

Where are the unit test? I need to see various junit test cases being tested. 

Solution
Bartleby Expert
SEE SOLUTION
Knowledge Booster
Map
Learn more about
Need a deep-dive on the concept behind this application? Look no further. Learn more about this topic, computer-science and related others by exploring similar questions and additional content below.
Similar questions
Recommended textbooks for you
Database System Concepts
Database System Concepts
Computer Science
ISBN:
9780078022159
Author:
Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Publisher:
McGraw-Hill Education
Starting Out with Python (4th Edition)
Starting Out with Python (4th Edition)
Computer Science
ISBN:
9780134444321
Author:
Tony Gaddis
Publisher:
PEARSON
Digital Fundamentals (11th Edition)
Digital Fundamentals (11th Edition)
Computer Science
ISBN:
9780132737968
Author:
Thomas L. Floyd
Publisher:
PEARSON
C How to Program (8th Edition)
C How to Program (8th Edition)
Computer Science
ISBN:
9780133976892
Author:
Paul J. Deitel, Harvey Deitel
Publisher:
PEARSON
Database Systems: Design, Implementation, & Manag…
Database Systems: Design, Implementation, & Manag…
Computer Science
ISBN:
9781337627900
Author:
Carlos Coronel, Steven Morris
Publisher:
Cengage Learning
Programmable Logic Controllers
Programmable Logic Controllers
Computer Science
ISBN:
9780073373843
Author:
Frank D. Petruzella
Publisher:
McGraw-Hill Education