DEA-C01 · Question #292
A company stores Apache Parquet files in an Amazon S3 data lake. The data lake receives thousands of files from multiple sources every hour. The files range in size from 50 KB to 100 KB. The company…
The correct answer is C. Configure Iceberg table properties to enable automatic compaction based on thresholds for file. Apache Iceberg is designed to manage table performance over time by controlling file layout through compaction. Enabling Iceberg’s automatic compaction via table properties allows the table to rewrite many small files into fewer larger files based on configured thresholds…
Question
A company stores Apache Parquet files in an Amazon S3 data lake. The data lake receives thousands of files from multiple sources every hour. The files range in size from 50 KB to 100 KB. The company is evaluating the implementation of Apache Iceberg tables for the data lake. The company is using AWS Glue Data Catalog as part of the evaluation. The company needs a solution to optimize query performance in Iceberg. The solution must ensure that Iceberg table performance does not degrade when more files are added over time. Which solution will meet these requirements?
Options
- AUse an AWS Glue job to compact the files into a standard size of 512 MB at the end of each day.
- BConfigure the Data Catalog to automatically compact the files every minute.
- CConfigure Iceberg table properties to enable automatic compaction based on thresholds for file
- DImplement a partition strategy in Amazon S3. Run an AWS Glue crawler to update the Data
How the community answered
(15 responses)- B7% (1)
- C87% (13)
- D7% (1)
Explanation
Apache Iceberg is designed to manage table performance over time by controlling file layout through compaction. Enabling Iceberg’s automatic compaction via table properties allows the table to rewrite many small files into fewer larger files based on configured thresholds, preventing the “small files problem” and keeping query planning and scan performance from degrading as new files continue to arrive.
Topics
Community Discussion
No community discussion yet for this question.