Python Artificial Intelligence Projects for Beginners

term frequency inverse document frequency
 (
TF-IDF
), which
measures how common a word or term is in the document. 

---

Applications for Comment Classification
Chapter 3
[ 51 ]
We use logarithms to ensure that long documents with many words are very similar to
short documents with similar words. TF-IDF has two components that multiply, that is
when TF is high, the result is high but IDF measures how common the word is among all
the documents and that will affect the common words. So, a word that is common in other
documents will have a low score, regardless of how many times it appeared.
If a document has a low score which means the word appeared rarely and if the score is
high it means the word appears frequently in the document. But if the word is quite
common in all the documents then it becomes irrelevant to score on this document. It is
anyhow considered to have low score. This shows that the formula for TF-IDF exhibits in a
way we want our model to be. The following graph explains our theory:
We will be using the bag-of-words method to detect whether YouTube comments are spam
or .