But for a machine: if a machine can only see a list of words that appear in a document and how often they appear, can it reasonably guess what the document is about?
If you handed someone a blank piece of paper and the only things written on it were words and word frequencies, could they guess what the article was about?
An article about knife sharpening is pretty predictable. The original text of this word frequency table comes from a manual for sharpening kitchen knives.
What if the words "step" and "how" were on the list? Would people still think the article was about knife sharpening? Would they be able to tell if the article was about sharpening kitchen knives or pocket knives?
If we can't guess what the article is about based on the senegal mobile database words it uses, then it doesn't meet BSoPt's word frequency criteria.
Can we still use word frequencies in BERT?
Early natural language processing NLP methods used by search engines used statistical analysis of word frequency and word co-occurrence to determine the content of a page. They ignored the order and parts of speech of the words in our content and simply treated our pages as a tool for storing text.
The tool we use to optimize this NLP compares our content to our competitors’ word usage frequency and tells us where the gaps in word usage are. Hypothetically, if we add these words to our content, our pages will rank higher or at least give search engines a better understanding of our content.
These tools still exist: MarketMuse, SEMRush, seobility, Ryte, and others that have word frequency or TD-IDF gap analysis capabilities. I've been using a free word frequency detection tool called Online Text Comparator , and it works pretty well. Are they still useful now that search engines have made progress with NLP methods like BERT? I think so, but it's not as simple as just thanking more words and getting better rankings.