The data science team has requested assistance in accelerating queries on free form text from user reviews. The data is currently stored in Parquet with the below schema: item_id INT, user_id INT…

Question

Options

• ADelta Lake statistics are not optimized for free text fields with high cardinality.
• BText data cannot be stored with Delta Lake.
• CZORDER ON review will need to be run to see performance gains.
• DThe Delta log creates a term matrix for free text fields to support selective filtering.
• EDelta Lake statistics are only collected on the first 4 columns in a table.

• A
  88% (46)
• C
  2% (1)
• D
  6% (3)
• E
  4% (2)

Explanation

Converting the data to Delta Lake may not improve query performance on free text fields with high cardinality, such as the review column. This is because Delta Lake collects statistics on the minimum and maximum values of each column, which are not very useful for filtering or skipping data on free text fields. Moreover, Delta Lake collects statistics on the first 32 columns by default, which may not include the review column if the table has more columns. Therefore, the junior data engineer's suggestion is not correct. A better approach would be to use a full-text search engine, such as Elasticsearch, to index and query the review column. Alternatively, you can use natural language processing techniques, such as tokenization, stemming, and lemmatization, to preprocess the review column and create a new column with normalized terms that can be used for filtering or skipping data.

No community discussion yet for this question.