DEA-C01 · Question #232
A company wants to build a dimension table in an Amazon S3 bucket. The bucket contains historical data that includes 10 million records. The historical data is 1 TB in size. A data engineer needs a…
The correct answer is D. Develop an Amazon EMR job to read new changes into Apache Spark DataFrames. Use the. By using Apache Hudi on EMR you get native upsert support against your S3 "base table." Hudi's indexing and metadata management lets it rewrite only the small set of changed files (your ~10,000 daily records) rather than scanning and rewriting the full 1 TB. This minimizes…
Question
A company wants to build a dimension table in an Amazon S3 bucket. The bucket contains historical data that includes 10 million records. The historical data is 1 TB in size. A data engineer needs a solution to update changes for up to 10,000 records in the base table every day. Which solution will meet this requirement with the LOWEST runtime?
Options
- ADevelop an Apache Spark job in Amazon EMR to read the historical data and the new changes
- BDevelop an AWS Glue Python job to read the historical data and new changes into two Pandas
- CDevelop an AWS Glue Apache Spark job to read the historical data and new changes into two
- DDevelop an Amazon EMR job to read new changes into Apache Spark DataFrames. Use the
How the community answered
(18 responses)- A6% (1)
- B11% (2)
- C6% (1)
- D78% (14)
Explanation
By using Apache Hudi on EMR you get native upsert support against your S3 "base table." Hudi's indexing and metadata management lets it rewrite only the small set of changed files (your ~10,000 daily records) rather than scanning and rewriting the full 1 TB. This minimizes processing and achieves the lowest runtime for daily updates.
Topics
Community Discussion
No community discussion yet for this question.