nerdexam
Google

PROFESSIONAL-DATA-ENGINEER · Question #306

A web server sends click events to a Pub/Sub topic as messages. The web server includes an eventTimestamp attribute in the messages, which is the time when the click occurred. You have a Dataflow…

The correct answer is C. Messages in your Dataflow job are processed in less than 30 seconds, but your job cannot keep up with the backlog in the Pub/Sub subscription. Optimize your. Option C is correct because data freshness (40 seconds) measures how old the oldest unprocessed message in the backlog is - meaning messages are waiting ~40 seconds before Dataflow even begins processing them, which already exceeds the 30-second SLA before any processing…

Submitted by femi9· Mar 30, 2026Building and operationalizing data processing systems

Question

A web server sends click events to a Pub/Sub topic as messages. The web server includes an eventTimestamp attribute in the messages, which is the time when the click occurred. You have a Dataflow streaming job that reads from this Pub/Sub topic through a subscription, applies some transformations, and writes the result to another Pub/Sub topic for use by the advertising department. The advertising department needs to receive each message within 30 seconds of the corresponding click occurrence, but they report receiving the messages late. Your Dataflow job's system lag is about 5 seconds, and the data freshness is about 40 seconds. Inspecting a few messages show no more than 1 second lag between their eventTimestamp and publishTime. What is the problem and what should you do?

Options

  • AThe advertising department is causing delays when consuming the messages. Work with the advertising department to fix this.
  • BMessages in your Dataflow job are taking more than 30 seconds to process. Optimize your job or increase the number of workers to fix this.
  • CMessages in your Dataflow job are processed in less than 30 seconds, but your job cannot keep up with the backlog in the Pub/Sub subscription. Optimize your
  • DThe web server is not pushing messages fast enough to Pub/Sub. Work with the web server team to fix this.

How the community answered

(53 responses)
  • A
    19% (10)
  • B
    9% (5)
  • C
    68% (36)
  • D
    4% (2)

Explanation

Option C is correct because data freshness (40 seconds) measures how old the oldest unprocessed message in the backlog is - meaning messages are waiting ~40 seconds before Dataflow even begins processing them, which already exceeds the 30-second SLA before any processing occurs. The system lag of only 5 seconds confirms individual messages process quickly once picked up; the bottleneck is that the job can't drain the Pub/Sub subscription fast enough to keep the backlog from growing. The fix is to increase parallelism or optimize throughput so data freshness drops below 30 seconds.

Why the distractors are wrong:

  • A is wrong because the delay occurs before messages even reach the advertising department - it's a Dataflow throughput problem, not a consumer problem.
  • B is wrong because system lag (per-message processing time) is only 5 seconds, ruling out slow individual processing.
  • D is wrong because the ≤1 second gap between eventTimestamp and publishTime proves the web server is publishing near-instantly.

Memory tip: Think of data freshness as the "waiting room clock" (how long messages wait in line) and system lag as the "service desk speed" (how fast each message is handled). A 30-second SLA is broken by a long wait, not a fast desk - so watch data freshness, not system lag.

Topics

#Dataflow#Streaming Performance#Monitoring Metrics#Latency Optimization

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full PROFESSIONAL-DATA-ENGINEER Practice