o compute the conditional probabilities you need to determine unigram and bigram counts first (you can do this in a single pass through a file if you do things carefully) and store them in a Binary Search Tree (BST). After that, you can compute the conditional probabilities. Input files Test files can be found on (http://www.gutenberg.org/ebooks/). For example, search for “Mark Twain.” Then click on any of his books. Next download the “Plain Text UTF-8” format. In addition, you should test your program on other input files as well, for which you can hand-compute the correct answer. Output files
To compute the conditional probabilities you need to determine unigram and
bigram counts first (you can do this in a single pass through a file if you do things
carefully) and store them in a Binary Search Tree (BST). After that, you can compute
the conditional probabilities.
Input files
Test files can be found on (http://www.gutenberg.org/ebooks/). For example,
search for “Mark Twain.” Then click on any of his books. Next download the “Plain
Text UTF-8” format.
In addition, you should test your program on other input files as well, for which you
can hand-compute the correct answer.
Output files
Your program must accept the name of an input file as a command line argument.
Let's call the file name of this file fn. Your program must then produce as output the
following set of files:
• Your program must write the unigram counts to a file named fn.uni in which
each unigram is listed on a separate line, and each line contains just the
unigram and its count (an integer), separated by a single space.
• Your program must write the bigram counts to a file named fn.bi in which
each bigram is listed on a separate line, and each line contains just the
bigram and its count (an integer), separated by a single space.
• Your program must write the conditional probabilities to a file named fn.cp,
reported in the form P(WORD(k)|WORD(k-1)) = p, where p is the conditional
probability of WORD(k) given WORD(k-1).
Notes
• You may use any BST implementation found online at your own risk
(provided the source of this code is cited properly). 50 marks bonus would
be given if you implemented the BST yourself (only the functionalities
needed to complete your project).
• Your program should accept file name(s) as command line argument(s) (no
hard-coded file names in your code).
• Your code will be tested using the latest version of the GNU C compiler on a
Unix-based
may want to use Windows Subsystem or a Virtual Box under your own
responsibility!
• Your code must be well commented. When writing your comments, you
should focus on what the code does at a high level; for example, describe the
main steps of an
• A ReadMe.txt file (including instructions on how to compile and how to run
your program along with any known problems) must be submitted
Step by step
Solved in 2 steps