Dropping stopwords and stemming/lemmatization process on a corpus (a set of text based documents) produce the following document space: “alone, animation, best, cold, enjoy, ever, filled, film, he, horror, i, love, most, movie, night, one, outside, overrated, put, plot, really, seen, story, scary, suspense, toy, twist, unexpected, watching”. If the corresponding set of text based documents consist of: D1 = I enjoy watching movies when it’s cold outside; D2 = Toy story is the best animation movie ever; D3 = Watching horror movies alone at night is really scary; D4 = He loves film filled with suspense and unexpected plot twists D5 = This is one of the most overrated movie I’ve ever seen.

Database System Concepts
7th Edition
ISBN:9780078022159
Author:Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Publisher:Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Chapter1: Introduction
Section: Chapter Questions
Problem 1PE
icon
Related questions
Question
100%

Dropping stopwords and stemming/lemmatization process on a corpus (a set of
text based documents) produce the following document space:
“alone, animation, best, cold, enjoy, ever, filled, film, he, horror, i, love, most,
movie, night, one, outside, overrated, put, plot, really, seen, story, scary,
suspense, toy, twist, unexpected, watching”.


If the corresponding set of text based documents consist of:


D1 = I enjoy watching movies when it’s cold outside;
D2 = Toy story is the best animation movie ever;
D3 = Watching horror movies alone at night is really scary;
D4 = He loves film filled with suspense and unexpected plot twists
D5 = This is one of the most overrated movie I’ve ever seen.


and if the query document Q contains key terms watching best animation movies, perform hand calculation
to determine the rank of every single document above if the retrieval process is performed using BM25
approach. Please use k1 = 1.5, b = 0,75, and to avoid IDF with negative values please slightly modify IDF
equation (AIMA 3rd Edition, p.868-869) to become: IDF(qi)=loge(1+(N−DF(qi)+0.5)/(DF(qi)+0.5)) where N is
the number of documents and DF(qi) is the number of documents in the corpus that contain term qi.

Expert Solution
steps

Step by step

Solved in 2 steps

Blurred answer
Knowledge Booster
Intelligent Machines
Learn more about
Need a deep-dive on the concept behind this application? Look no further. Learn more about this topic, computer-science and related others by exploring similar questions and additional content below.
Similar questions
  • SEE MORE QUESTIONS
Recommended textbooks for you
Database System Concepts
Database System Concepts
Computer Science
ISBN:
9780078022159
Author:
Abraham Silberschatz Professor, Henry F. Korth, S. Sudarshan
Publisher:
McGraw-Hill Education
Starting Out with Python (4th Edition)
Starting Out with Python (4th Edition)
Computer Science
ISBN:
9780134444321
Author:
Tony Gaddis
Publisher:
PEARSON
Digital Fundamentals (11th Edition)
Digital Fundamentals (11th Edition)
Computer Science
ISBN:
9780132737968
Author:
Thomas L. Floyd
Publisher:
PEARSON
C How to Program (8th Edition)
C How to Program (8th Edition)
Computer Science
ISBN:
9780133976892
Author:
Paul J. Deitel, Harvey Deitel
Publisher:
PEARSON
Database Systems: Design, Implementation, & Manag…
Database Systems: Design, Implementation, & Manag…
Computer Science
ISBN:
9781337627900
Author:
Carlos Coronel, Steven Morris
Publisher:
Cengage Learning
Programmable Logic Controllers
Programmable Logic Controllers
Computer Science
ISBN:
9780073373843
Author:
Frank D. Petruzella
Publisher:
McGraw-Hill Education