A Delta Lake table representing metadata about content posts from users has the following schema: user_id LONG, post_text STRING, post_id STRING, longitude FLOAT, latitude FLOAT, post_time TIMESTAMP,

Question

Options

• AStatistics in the Delta Log will be used to identify partitions that might Include files in the filtered
• BNo file skipping will occur because the optimizer does not know the relationship between the
• CThe Delta Engine will use row-level statistics in the transaction log to identify the flies that meet
• DStatistics in the Delta Log will be used to identify data files that might include records in the
• EThe Delta Engine will scan the parquet file footers to identify each row that meets the filter criteria.

• C
  3% (1)
• D
  93% (28)
• E
  3% (1)

Explanation

The correct answer is D. The table is partitioned by the date column, not longitude. Partition pruning will not help here because the filter is on a non-partition column. However, Delta Lake's transaction log (Delta Log) stores min/max column statistics for data files (collected for the first 32 columns by default). When the query engine evaluates the filter longitude < 20 AND longitude > -20, it reads the Delta Log to find which data files have a longitude min/max range that overlaps the filter range. Files where the maximum longitude is less than -20 or the minimum longitude is greater than 20 can be skipped entirely - a process called data skipping. This is more precise than partition pruning alone but less granular than row-level filtering, which happens after the relevant files are read.

No community discussion yet for this question.