Homework Help is Here – Start Your Trial Now!
Engineering
Computer Science
In the next few problems, we will work with a text file that contains the complete works of William Shakespeare. The data file using for this problem is located at: /FileStore/tables/shakespeare_complete.txt. We will begin by loading and processing the file and tokenizing the lines into individual words. Complete the following steps in a single code cell: 2. 1. Read the contents of the file shakespeare_complete.txt into an RDD named ws_lines. Create an RDD named ws_words by applying the transformations described below. This will require several uses of map() and flatMap() and a single call to filter (). Try to chain together the transformations together to complete all of these steps with a single statement (that will likely span multiple lines). • Tokenize the strings in ws_lines by splitting them on the 8 characters in the following list: ['' :', '|', '\t'] The resulting RDD should consist of strings rather than lists of strings. This will require multiple separate uses of flatMap () and split(). . Use the Python string method strip() with the punctuation string to remove common punctuation symbols from the start and end of the tokens. Then use strip() again with the string '0123456789' to remove numbers from the start and end of the tokens

In the next few problems, we will work with a text file that contains the complete works of William Shakespeare. The data file using for this problem is located at: /FileStore/tables/shakespeare_complete.txt. We will begin by loading and processing the file and tokenizing the lines into individual words. Complete the following steps in a single code cell: 2. 1. Read the contents of the file shakespeare_complete.txt into an RDD named ws_lines. Create an RDD named ws_words by applying the transformations described below. This will require several uses of map() and flatMap() and a single call to filter (). Try to chain together the transformations together to complete all of these steps with a single statement (that will likely span multiple lines). • Tokenize the strings in ws_lines by splitting them on the 8 characters in the following list: ['' :', '|', '\t'] The resulting RDD should consist of strings rather than lists of strings. This will require multiple separate uses of flatMap () and split(). . Use the Python string method strip() with the punctuation string to remove common punctuation symbols from the start and end of the tokens. Then use strip() again with the string '0123456789' to remove numbers from the start and end of the tokens

Database System Concepts
Database System Concepts
7th Edition
ISBN:9780078022159
Author:Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Publisher:Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Chapter1: Introduction
Section: Chapter Questions
Problem 1PE
See similar textbooks
icon
Related questions
Question
In the next few problems, we will work with a text file that contains the complete works of William Shakespeare. The data file using for this problem is located at: /FileStore/tables/shakespeare_complete.txt. We will begin by loading and processing the file and tokenizing the lines into individual words. Complete the following steps in a single code cell: 2. 1. Read the contents of the file shakespeare_complete.txt into an RDD named ws_lines. Create an RDD named ws_words by applying the transformations described below. This will require several uses of map() and flatMap() and a single call to filter (). Try to chain together the transformations together to complete all of these steps with a single statement (that will likely span multiple lines). Tokenize the strings in ws_lines by splitting them on the 8 characters in the following list: [' :', '|', '\t'] The resulting RDD should consist of strings rather than lists of strings. This will require multiple separate uses of flatMap() and split(). • Use the Python string method strip() with the punctuation string to remove common punctuation symbols from the start and end of the tokens. Then use strip() again with the string '0123456789' to remove numbers from the start and end of the tokens.
Transcribed Image Text:In the next few problems, we will work with a text file that contains the complete works of William Shakespeare. The data file using for this problem is located at: /FileStore/tables/shakespeare_complete.txt. We will begin by loading and processing the file and tokenizing the lines into individual words. Complete the following steps in a single code cell: 2. 1. Read the contents of the file shakespeare_complete.txt into an RDD named ws_lines. Create an RDD named ws_words by applying the transformations described below. This will require several uses of map() and flatMap() and a single call to filter (). Try to chain together the transformations together to complete all of these steps with a single statement (that will likely span multiple lines). Tokenize the strings in ws_lines by splitting them on the 8 characters in the following list: [' :', '|', '\t'] The resulting RDD should consist of strings rather than lists of strings. This will require multiple separate uses of flatMap() and split(). • Use the Python string method strip() with the punctuation string to remove common punctuation symbols from the start and end of the tokens. Then use strip() again with the string '0123456789' to remove numbers from the start and end of the tokens.
• • Use the Python string method replace() to replaces instances of the single quote/apostrophe "*" with the empty string ''. Convert all strings to lower case using the lower () string method. The steps above will create some empty strings of the form within the RDD. Filter out these empty strings. 3. Create a second RDD named dist_words that contains only one copy of each word found in ws_words. 4. Print the number of words in ws_words and the number of distinct words using the format shown below. Add spacing so that the numbers are left-aligned. Total Number of Words: Number of Distinct Words: XXXX xxxx We will now use sample() to get a sense as to the types of words found in ws_words. Draw a sample from ws_words using the arguments with Replacement=False and fraction=0.0001. Collect and print the results.
Transcribed Image Text:• • Use the Python string method replace() to replaces instances of the single quote/apostrophe "*" with the empty string ''. Convert all strings to lower case using the lower () string method. The steps above will create some empty strings of the form within the RDD. Filter out these empty strings. 3. Create a second RDD named dist_words that contains only one copy of each word found in ws_words. 4. Print the number of words in ws_words and the number of distinct words using the format shown below. Add spacing so that the numbers are left-aligned. Total Number of Words: Number of Distinct Words: XXXX xxxx We will now use sample() to get a sense as to the types of words found in ws_words. Draw a sample from ws_words using the arguments with Replacement=False and fraction=0.0001. Collect and print the results.
Expert Solution
steps

Step by step

Solved in 4 steps with 2 images

SEE SOLUTIONCheck out a sample Q&A here
Blurred answer
Knowledge Booster
Files and Directory
Learn more about
Need a deep-dive on the concept behind this application? Look no further. Learn more about this topic, computer-science and related others by exploring similar questions and additional content below.
Similar questions
Recommended textbooks for you
Database System Concepts
Database System Concepts
Computer Science
ISBN:
9780078022159
Author:
Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Publisher:
McGraw-Hill Education
Starting Out with Python (4th Edition)
Starting Out with Python (4th Edition)
Computer Science
ISBN:
9780134444321
Author:
Tony Gaddis
Publisher:
PEARSON
Digital Fundamentals (11th Edition)
Digital Fundamentals (11th Edition)
Computer Science
ISBN:
9780132737968
Author:
Thomas L. Floyd
Publisher:
PEARSON
C How to Program (8th Edition)
C How to Program (8th Edition)
Computer Science
ISBN:
9780133976892
Author:
Paul J. Deitel, Harvey Deitel
Publisher:
PEARSON
Database Systems: Design, Implementation, & Manag…
Database Systems: Design, Implementation, & Manag…
Computer Science
ISBN:
9781337627900
Author:
Carlos Coronel, Steven Morris
Publisher:
Cengage Learning
Programmable Logic Controllers
Programmable Logic Controllers
Computer Science
ISBN:
9780073373843
Author:
Frank D. Petruzella
Publisher:
McGraw-Hill Education
SEE MORE TEXTBOOKS