PROFESSIONAL-DATA-ENGINEER · Question #286
You want to create a machine learning model using BigQuery ML and create an endpoint for hosting the model using Vertex AI. This will enable the processing of continuous streaming data in near-real ti
The correct answer is D. Create a Pub/Sub topic and send all vendor data to it. Use Dataflow to process and sanitize the Pub/Sub data and stream it to BigQuery.. D is correct because it follows the canonical GCP real-time streaming architecture: Pub/Sub acts as a durable, scalable message broker that decouples the multiple vendor sources from downstream processing, and Dataflow provides a fully managed, autoscaling pipeline specifically d
Question
Options
- ACreate a new BigQuery dataset and use streaming inserts to land the data from multiple vendors. Configure your BigQuery ML model to use the "ingestion"
- BUse BigQuery streaming inserts to land the data from multiple vendors where your BigQuery dataset ML model is deployed.
- CCreate a Pub/Sub topic and send all vendor data to it. Connect a Cloud Function to the topic to process the data and store it in BigQuery.
- DCreate a Pub/Sub topic and send all vendor data to it. Use Dataflow to process and sanitize the Pub/Sub data and stream it to BigQuery.
How the community answered
(36 responses)- A8% (3)
- B6% (2)
- C3% (1)
- D83% (30)
Explanation
D is correct because it follows the canonical GCP real-time streaming architecture: Pub/Sub acts as a durable, scalable message broker that decouples the multiple vendor sources from downstream processing, and Dataflow provides a fully managed, autoscaling pipeline specifically designed for exactly-once stream processing and data sanitization - ensuring invalid values are handled before clean data lands in BigQuery for ML training and inference via the Vertex AI endpoint.
A is wrong because "ingestion" is not a real BigQuery ML configuration option - this answer fabricates a feature that doesn't exist, and BigQuery streaming inserts alone provide no mechanism to validate or sanitize invalid values mid-stream.
B is wrong for the same sanitization reason: streaming inserts dump data directly into BigQuery with no transformation layer, so invalid values would corrupt your dataset before the ML model ever sees it.
C is wrong because Cloud Functions are event-driven and stateless - they're ill-suited for high-throughput, continuous stream processing at scale; Dataflow is purpose-built for this workload with windowing, aggregation, and fault tolerance that Cloud Functions can't reliably provide.
Memory tip: Remember "Pub/Sub → Dataflow → BigQuery" as GCP's golden streaming trio - whenever a question involves real-time ingestion + transformation + analysis, this pipeline is almost always the answer.
Topics
Community Discussion
No community discussion yet for this question.