A data engineer is using AWS Glue ETL jobs to process data at frequent intervals. The processed data is then copied into Amazon S3. The ETL jobs run every 15 minutes. The AWS Glue Data Catalog…

Question

Options

• AUse the AWS Glue Data Catalog to manage the data catalog
• BUse the AWS Glue Data Catalog to manage the data catalog
• CUse an Apache Hive metastore to manage the data catalog
• DUse the AWS Glue Data Catalog to manage the data catalog

• A
  7% (1)
• B
  14% (2)
• C
  7% (1)
• D
  71% (10)

Explanation

The most cost-effective solution is to configure the AWS Glue ETL job to automatically update the Data Catalog upon completion. In AWS Glue ETL jobs, you can enable the 'enableUpdateCatalog' and 'partitionKeys' options in the job's DynamicFrame sink configuration. This causes the job itself to register new partitions in the Data Catalog after writing data to S3, with no additional services required. This avoids running a separate AWS Glue crawler (which would add cost and latency) after each of the frequent 15-minute job runs. Option C (Apache Hive metastore) would require managing a separate metadata store, increasing operational and infrastructure costs. Options A and B are similar but D represents the specific Glue-native auto-update capability that is most cost-effective.

No community discussion yet for this question.