nerdexam
Google

PROFESSIONAL-DATA-ENGINEER · Question #261

You need to give new website users a globally unique identifier (GUID) using a service that takes in data points and returns a GUID. This data is sourced from both internal and external systems via…

The correct answer is D. Batch the job into ten-second increments. Explanation/Reference: https://cloud.google.com/blog/products/data-analytics/guide-to-common-cloud-dataflow-use-case-patterns-part-1 If the call takes on average 1 sec, that would cause massive backpressure on the pipeline. In these circumstances you should consider batching…

Submitted by alyssa_d· Mar 30, 2026Building and operationalizing data processing systems

Question

You need to give new website users a globally unique identifier (GUID) using a service that takes in data points and returns a GUID. This data is sourced from both internal and external systems via HTTP calls that you will make via microservices within your pipeline. There will be tens of thousands of messages per second and that can be multi-threaded. and you worry about the backpressure on the system. How should you design your pipeline to minimize that backpressure?

Options

  • ACall out to the service via HTTP
  • BCreate the pipeline statically in the class definition
  • CCreate a new object in the startBundle method of DoFn
  • DBatch the job into ten-second increments

How the community answered

(36 responses)
  • A
    14% (5)
  • B
    6% (2)
  • C
    31% (11)
  • D
    50% (18)

Explanation

Explanation/Reference: https://cloud.google.com/blog/products/data-analytics/guide-to-common-cloud-dataflow-use-case-patterns-part-1 If the call takes on average 1 sec, that would cause massive backpressure on the pipeline. In these circumstances you should consider batching these requests,

Topics

#Dataflow DoFn#backpressure#external HTTP calls#pipeline optimization

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full PROFESSIONAL-DATA-ENGINEER Practice