An investment company needs to manage and extract insights from a volume of semi-structured data that grows continuously. A data engineer needs to deduplicate the semi-structured data, remove records that are duplicates, and remove common misspellings of duplicates. Which solution will meet these requirements with the LEAST operational overhead?

Question

Accepted Answer

A. Use the FindMatches feature of AWS Glue to remove duplicate records. AWS Glue FindMatches is a machine learning transform that intelligently identifies and links records referring to the same real-world entity, even if they have different values or misspellings. This feature is specifically designed for fuzzy matching and deduplication of dirty data with minimal operational overhead as a fully managed service.

Answer

B. Use non-Windows functions in Amazon Athena to remove duplicate records. Using standard SQL functions in Amazon Athena can identify exact duplicate records but is not designed for fuzzy matching or correcting misspellings across semi-structured data without extensive custom logic.

Answer

C. Use Amazon Neptune ML and an Apache Gremlin script to remove duplicate records. Amazon Neptune ML is used for machine learning on graph data, and Apache Gremlin is a graph traversal language; neither is suitable for general-purpose fuzzy deduplication of semi-structured data in an ETL context.

Answer

D. Use the global tables feature of Amazon DynamoDB to prevent duplicate data. Amazon DynamoDB global tables provide multi-region replication for DynamoDB data to enhance availability and low-latency access, but this feature does not address the problem of identifying and removing fuzzy duplicate data within incoming semi-structured data streams.

An investment company needs to manage and extract insights from a volume of semi-structured data that grows continuously. A data engineer needs to deduplicate the semi-structured data, remove…

Question

Options

How the community answered

Why each option

Topics

Community Discussion