DAS-C01 · Question #146
A hospital uses an electronic health records (EHR) system to collect two types of data. - Patient information, which includes a patient's name and address - Diagnostic tests conducted and the…
The correct answer is B. Use an AWS Glue crawler to catalog the data in Amazon S3. To automate the ingestion of patient information with updates/deletes and append-only diagnostic test data into Amazon Redshift tables, using an AWS Glue crawler is a crucial foundational step for schema discovery.
Question
A hospital uses an electronic health records (EHR) system to collect two types of data.
- Patient information, which includes a patient's name and address
- Diagnostic tests conducted and the results of these tests Patient information is expected to
change periodically Existing diagnostic test data never changes and only new records are added. The hospital runs an Amazon Redshift cluster with four dc2.large nodes and wants to automate the ingestion of the patient information and diagnostic test data into respective Amazon Redshift tables for analysis. The EHR system exports data as CSV files to an Amazon S3 bucket on a daily basis. Two sets of CSV files are generated. One set of files is for patient information with updates, deletes, and inserts. The other set of files is for new diagnostic test data only. What is the MOST cost-effective solution to meet these requirements?
Options
- AUse Amazon EMR with Apache Hudi.
- BUse an AWS Glue crawler to catalog the data in Amazon S3
- CUse an AWS Lambda function to run a COPY command that appends new diagnostic test data to
- DUse AWS Database Migration Service (AWS DMS) to collect and process change data capture
How the community answered
(46 responses)- A7% (3)
- B74% (34)
- C4% (2)
- D15% (7)
Why each option
To automate the ingestion of patient information with updates/deletes and append-only diagnostic test data into Amazon Redshift tables, using an AWS Glue crawler is a crucial foundational step for schema discovery.
Amazon EMR with Apache Hudi is excellent for managing mutable data in a data lake in S3 but requires additional integration or ETL steps to automate ingestion specifically into Amazon Redshift tables.
An AWS Glue crawler automatically infers the schema and partitions for data stored in Amazon S3, creating metadata tables in the AWS Glue Data Catalog. This catalog is essential for subsequent automated processing by AWS Glue ETL jobs or for querying by services like Redshift Spectrum, enabling efficient data loading and change data capture (CDC) management into Amazon Redshift tables.
An AWS Lambda function with a COPY command is suitable for appending new data to Redshift, like diagnostic tests, but it does not inherently provide an automated solution for handling updates and deletes required for patient information, nor does it automate schema inference.
AWS Database Migration Service (DMS) is designed for migrating databases and continuous replication (CDC) between database systems. While powerful for CDC, using DMS directly for file-based CDC from S3 into Redshift for varying schemas can be more complex than an AWS Glue-centric approach.
Concept tested: AWS Glue Crawler for schema discovery and cataloging
Source: https://docs.aws.amazon.com/glue/latest/dg/components-overview.html#crawler-overview
Topics
Community Discussion
No community discussion yet for this question.