A company uses Amazon S3 to store semi-structured data in a transactional data lake. Some of the data files are small, but other data files are tens of terabytes. A data engineer must perform a…

Creating a custom AWS Lambda function to identify changes from daily full snapshots can become complex and expensive to develop and maintain, especially with large datasets, and might not be as optimized as specialized data lake formats for merge operations.

Ingesting a full daily snapshot into Amazon RDS for MySQL is not a cost-effective solution for a data lake strategy, as relational databases are typically not optimized for storing large, raw analytical datasets, and AWS Database Migration Service (DMS) is primarily for database migration and replication, not for performing CDC directly on S3 data from flat files.

Open-source data lake formats such as Apache Hudi, Delta Lake, or Apache Iceberg are specifically designed to provide ACID properties, upsert capabilities, and schema evolution directly on data stored in Amazon S3, enabling efficient and cost-effective change data capture from full snapshots into a transactional data lake.

Ingesting a full daily snapshot into an Amazon Aurora MySQL DB instance that runs Aurora Serverless, followed by using AWS DMS, faces similar issues to option B, as it's an expensive and less suitable approach for managing a transactional data lake on S3 compared to specialized data lake formats.