DEA-C01 · Question #173
A company has three subsidiaries. Each subsidiary uses a different data warehousing solution. The first subsidiary hosts its data warehouse in Amazon Redshift. The second subsidiary uses Teradata…
The correct answer is B. Use the Amazon Athena federated query connectors for Amazon Redshift, Teradata, and. Amazon Athena federated query allows querying data from multiple data sources, including Amazon Redshift, Teradata, and Google BigQuery, using their federated query connectors. This solution offers a serverless approach, reducing the operational overhead of managing…
Question
A company has three subsidiaries. Each subsidiary uses a different data warehousing solution. The first subsidiary hosts its data warehouse in Amazon Redshift. The second subsidiary uses Teradata Vantage on AWS. The third subsidiary uses Google BigQuery. The company wants to aggregate all the data into a central Amazon S3 data lake. The company wants to use Apache Iceberg as the table format. A data engineer needs to build a new pipeline to connect to all the data sources, run transformations by using each source engine, join the data, and write the data to Iceberg. Which solution will meet these requirements with the LEAST operational effort?
Options
- AUse native Amazon Redshift, Teradata, and BigQuery connectors to build the pipeline in AWS
- BUse the Amazon Athena federated query connectors for Amazon Redshift, Teradata, and
- CUse the native Amazon Redshift connector, the Java Database Connectivity (JDBC) connector for
- DUse the native Amazon Redshift, Teradata, and BigQuery connectors in Amazon Appflow to write
How the community answered
(25 responses)- A4% (1)
- B60% (15)
- C24% (6)
- D12% (3)
Explanation
Amazon Athena federated query allows querying data from multiple data sources, including Amazon Redshift, Teradata, and Google BigQuery, using their federated query connectors. This solution offers a serverless approach, reducing the operational overhead of managing infrastructure while allowing SQL-based transformations across all data sources. Once the data is read and joined, Athena can write the results back to Amazon S3 in the Iceberg table format with a Merge operation. This approach minimizes the operational effort as Athena manages the complexity of connecting to different databases through its connectors, and you can perform the necessary transformations and data joins using familiar SQL. While AWS Glue is a powerful ETL tool, it requires more operational effort to manage complex transformations across multiple systems, and managing native transforms across different engines (Redshift, Teradata, BigQuery) in Glue can introduce additional complexity. Amazon EMR with PySpark can handle the task, but it requires more operational effort to manage and maintain the EMR cluster. Writing and maintaining PySpark code can also be more complex compared to using SQL in Athena. Appflow is primarily designed for simple data movement between SaaS applications and AWS services, but it does not provide the complex transformation and joining capabilities needed for this scenario. Using Athena after Appflow for joins adds unnecessary complexity compared to directly using federated queries in Athena.
Topics
Community Discussion
No community discussion yet for this question.