nerdexam
Microsoft

DP-700 · Question #111

An e-commerce platform processes a high volume of order events through an event streaming system. Recently, the analytics dashboard began showing incomplete and inconsistent data. Upon…

The correct answer is A. Implement deduplication logic by including a unique event ID in the data schema B. Use watermarking techniques to handle late-arriving events while maintaining processing order D. Add horizontal scaling to the processing nodes to handle higher volumes of events. To resolve issues with duplicated events, incorrect processing of late-arriving events, and poor scaling in an event streaming system, you should implement deduplication, use watermarking, and add horizontal scaling.

Design and implement data ingestion and transformation

Question

An e-commerce platform processes a high volume of order events through an event streaming system. Recently, the analytics dashboard began showing incomplete and inconsistent data. Upon investigation, the following observations were made: - Some events are being duplicated in the stream. - Late-arriving events are not being processed correctly. - The system is not scaling properly under high load. As the data engineer, you need to address three issues. Which of the following actions would resolve the identified problems? (Select THREE)

Options

  • AImplement deduplication logic by including a unique event ID in the data schema
  • BUse watermarking techniques to handle late-arriving events while maintaining processing order
  • CIncrease the event batch size to reduce the number of events processed concurrently
  • DAdd horizontal scaling to the processing nodes to handle higher volumes of events
  • EConfigure the system to drop late-arriving events to avoid inconsistencies

How the community answered

(24 responses)
  • A
    83% (20)
  • C
    4% (1)
  • E
    13% (3)

Why each option

To resolve issues with duplicated events, incorrect processing of late-arriving events, and poor scaling in an event streaming system, you should implement deduplication, use watermarking, and add horizontal scaling.

AImplement deduplication logic by including a unique event ID in the data schemaCorrect

Implementing deduplication logic, often by tracking unique event IDs, directly addresses the problem of duplicate events in the stream, ensuring data accuracy and consistency.

BUse watermarking techniques to handle late-arriving events while maintaining processing orderCorrect

Watermarking techniques provide a robust mechanism in stream processing to manage and correctly process late-arriving events by defining a threshold for how long to wait for out-of-order data, maintaining data completeness and processing order.

CIncrease the event batch size to reduce the number of events processed concurrently

Increasing the event batch size might reduce processing overhead per batch but does not fundamentally address the system's inability to scale under high load as effectively as adding more processing nodes.

DAdd horizontal scaling to the processing nodes to handle higher volumes of eventsCorrect

Adding horizontal scaling to processing nodes directly resolves the issue of the system not scaling properly under high load, increasing throughput and ensuring the system can handle higher volumes of events effectively.

EConfigure the system to drop late-arriving events to avoid inconsistencies

Configuring the system to drop late-arriving events resolves inconsistencies by data loss, leading to incomplete data, which contradicts the goal of addressing "incomplete" data and providing correct processing.

Concept tested: Event streaming challenges - deduplication, late data, scaling

Source: https://learn.microsoft.com/en-us/stream-analytics/stream-analytics-event-ordering

Topics

#Event Streaming#Data Deduplication#Late Event Handling#System Scalability

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full DP-700 Practice