1(WEB SCRAPING WITH THE REQUESTS AND BEAUTIFUL SOUP LIBRARIES) Web pages are excellent sources of text to use in NLP tasks. In the following IPython session, you’ll use the requests library to download the www.python.org home page’s content. This is called web scraping. You’ll then use the Beautiful Soup library37 to extract only the text from the page. Eliminate the stop words in the resulting text, then use the wordcloud module to create a word cloud based on the text. Its module name is bs4for Beautiful Soup 4. In [1]: importrequests In [2]: response = requests.get('https://www.python.org') In [3]: response.content # gives back the page's HTML In [4]: frombs4 importBeautifulSoup In [5]: soup = BeautifulSoup(response.content, 'html5lib') In [6]: text = soup.get_text(strip=True) # text without tags In the preceding code, snippets [1]–[3] get a web page. The get functionreceives a URL as an argument and returns the corresponding web page as a Response object. The Response’s content property contains the web page’s content. Snippets [4]–[6] get only the web page’s text. Snippet [5] creates a BeautifulSoup object to process the text in response.content. BeautifulSoup method get_text with the keyword argument strip=Truereturns just the text of the web page without its structural information that your web browser uses to display the web page.

Database System Concepts

7th Edition

ISBN:9780078022159

Author:Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan

Publisher:Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan

Chapter1: Introduction

Section: Chapter Questions

Problem 1PE

See similar textbooks

Related questions

Question

1. 1(WEB SCRAPING WITH THE REQUESTS AND BEAUTIFUL SOUP LIBRARIES) Web pages are excellent sources of text to use in NLP tasks. In the following IPython session, you’ll use the requests library to download the www.python.org home page’s content. This is called web scraping. You’ll then use the Beautiful Soup library³⁷ to extract only the text from the page. Eliminate the stop words in the resulting text, then use the wordcloud module to create a word cloud based on the text.
2. Its module name is bs4for Beautiful Soup 4.
In [1]: importrequests

In [2]: response = requests.get('https://www.python.org')

In [3]: response.content # gives back the page's HTML

In [4]: frombs4 importBeautifulSoup

In [5]: soup = BeautifulSoup(response.content, 'html5lib')

In [6]: text = soup.get_text(strip=True) # text without tags

In the preceding code, snippets [1]–[3] get a web page. The get functionreceives a URL as an argument and returns the corresponding web page as a Response object. The Response’s content property contains the web page’s content. Snippets [4]–[6] get only the web page’s text. Snippet [5] creates a BeautifulSoup object to process the text in response.content. BeautifulSoup method get_text with the keyword argument strip=Truereturns just the text of the web page without its structural information that your web browser uses to display the web page.

12.17 (PROJECT: SIMILARITY DETECTION) One way to measure similarity between two documents is to compare frequency counts of the parts of speech used in each. Build dictionaries of the parts of speech frequencies for two Project Gutenberg books from the same author and two from different authors and compare the results.

Expert Solution

Trending now

This is a popular solution!

Step by step

Solved in 3 steps with 2 images

SEE SOLUTION Check out a sample Q&A here

Knowledge Booster

Learn more about

Need a deep-dive on the concept behind this application? Look no further. Learn more about this topic, computer-science and related others by exploring similar questions and additional content below.

Similar questions