MLS-C01 · Question #349
A news company is developing an article search tool for its editors. The search tool should look for the articles that are most relevant and representative for particular words that are queried…
The correct answer is B. Build a term frequency for each word in the articles that is weighted with the article's length. Build. To identify the most relevant and important words in documents for search, a method that considers both local frequency and global rarity is required.
Question
A news company is developing an article search tool for its editors. The search tool should look for the articles that are most relevant and representative for particular words that are queried among a corpus of historical news documents. The editors test the first version of the tool and report that the tool seems to look for word matches in general. The editors have to spend additional time to filter the results to look for the articles where the queried words are most important. A group of data scientists must redesign the tool so that it isolates the most frequently used words in a document. The tool also must capture the relevance and importance of words for each document in the corpus. Which solution meets these requirements?
Options
- AExtract the topics from each article by using Latent Dirichlet Allocation (LDA) topic modeling.
- BBuild a term frequency for each word in the articles that is weighted with the article's length. Build
- CDownload a pretrained word-embedding lookup table. Create a titles-embedding table by
- DBuild a term frequency score table for each word in each article of the corpus. Assign a score of
How the community answered
(30 responses)- A7% (2)
- B77% (23)
- C3% (1)
- D13% (4)
Why each option
To identify the most relevant and important words in documents for search, a method that considers both local frequency and global rarity is required.
Latent Dirichlet Allocation (LDA) is a topic modeling technique used to discover abstract topics in a collection of documents, but it doesn't directly provide word importance scores for specific queries in the same way TF-IDF does.
Building a Term Frequency-Inverse Document Frequency (TF-IDF) score for each word in each article directly addresses the requirements. TF-IDF measures the importance of a word in a document relative to a corpus by weighting its frequency within the document (Term Frequency) against its inverse frequency across all documents (Inverse Document Frequency), thus isolating important and relevant words.
Word embeddings capture semantic relationships between words and are useful for similarity, but they don't inherently quantify the 'relevance and importance' of a word within a specific document relative to a corpus for keyword search in the manner TF-IDF does.
Building a simple term frequency table or assigning a score of 1 to each word only captures presence or raw frequency, which is what the editors found insufficient, as it doesn't account for the importance or rarity of words across the entire document corpus.
Concept tested: TF-IDF for Document Relevance
Source: https://en.wikipedia.org/wiki/Tf-idf
Topics
Community Discussion
No community discussion yet for this question.