This is the README file for A0135817B and A0147995H's submission

Email address:
e0011848@u.nus.edu
e0012667@u.nus.edu

== Python Version ==

We're using Python Version 2.7.14 for this assignment.

== General Notes about this assignment ==

## Indexing:

Procedure:

  1. For every training file, load the whole content into the system and tokenize into words
  2. Remove invalid characters of the tokenized word
  3. Count the term in a dictionary mapping document id to a term frequency counter
  4. Write postings into the posting file
  5. Write terms into dictionary file with offset to the corresponding postings.

Notes:

  - We choose not to use sentence tokenizer because we are handle invalid punctuation by keeping only
     digits, alphabets, white spaces and dashes

Searching:
 - The entire dictionary is read into memory
 - Each query is tokenized using the same method as for indexing
 - For each query term, we calculate their weights using 1 + log(tf)
 - For every document that any of the query terms appear in, we calculate the total score using the query weight
   and the document tf-idf
 - We collect the top 10 documents using a priority queue and return the result

== Files included with this submission ==

common.py       - includes the tokenization function which is used for both index and search
index.py        - indexing program that will be run to index all the training files.
search.py       - searching program that will be used to execute queries in a specific file and give output.
dictionary.txt  - a dictionary mapping terms to their location in the postings file
postings.txt    - includes document IDs for each term

== Statement of individual work ==

Please initial one of the following statements.

[X] We, A0135817B and A0147995H, certify that I have followed the CS 3245 Information
Retrieval class guidelines for homework assignments.  In particular, I
expressly vow that I have followed the Facebook rule in discussing
with others in doing the assignment and did not take notes (digital or
printed) from the discussions.  

[ ] I, A0000000X, did not follow the class rules regarding homework
assignment, because of the following reason:

<Please fill in>

I suggest that I should be graded as follows:

<Please fill in>

== References ==

- tf-idf