An online retail company wants to perform analytics on data in large Amazon S3 objects using Amazon EMR. An Apache Spark job repeatedly queries the same data to populate an analytics dashboard. The…

Question

Options

• ACopy the source data into Amazon Redshift and rewrite the Apache Spark code to create
• BCopy the source data from Amazon S3 into Hadoop Distributed File System (HDFS) using
• CLoad the data into Spark DataFrames.
• DStream the data into Amazon Kinesis and use the Kinesis Connector Library (KCL) in multiple
• EUse Amazon S3 Select to retrieve the data necessary for the dashboards from the S3 objects.

• A
  19% (5)
• B
  4% (1)
• C
  70% (19)
• D
  7% (2)

Explanation

Two complementary approaches minimize load time for repeatedly queried S3 data in Spark: (C) Loading data into Spark DataFrames with caching (df.cache() or df.persist()) stores the data in EMR cluster memory or disk after the first read, so subsequent queries against the same dataset are served from cache rather than re-reading from S3, drastically reducing latency. (E) Amazon S3 Select allows Spark to push down filter predicates to S3 itself, so only the rows and columns needed for the dashboard are transferred over the network rather than entire objects - this reduces I/O and speeds up initial load. Together, S3 Select minimizes data transferred on first load, and DataFrame caching eliminates re-reads on subsequent queries. Option A (copying to Redshift) changes the architecture entirely. Option B (copying to HDFS) helps but requires significant upfront copy time. Option D (Kinesis streaming) is for real-time ingestion, not batch analytics optimization.

No community discussion yet for this question.