PROFESSIONAL-MACHINE-LEARNING-ENGINEER · Question #221
You have trained a model by using data that was preprocessed in a batch Dataflow pipeline. Your use case requires real-time inference. You want to ensure that the data preprocessing logic is applied…
The correct answer is B. Refactor the transformation code in the batch data pipeline so that it can be used outside of the. To ensure consistent data preprocessing for models trained with batch Dataflow and served for real-time inference, the transformation logic must be reusable across both environments.
Question
Options
- APerform data validation to ensure that the input data to the pipeline is the same format as the
- BRefactor the transformation code in the batch data pipeline so that it can be used outside of the
- CRefactor the transformation code in the batch data pipeline so that it can be used outside of the
- DBatch the real-time requests by using a time window and then use the Dataflow pipeline to
How the community answered
(35 responses)- A9% (3)
- B86% (30)
- C3% (1)
- D3% (1)
Why each option
To ensure consistent data preprocessing for models trained with batch Dataflow and served for real-time inference, the transformation logic must be reusable across both environments.
Data validation ensures input quality but does not guarantee that the same preprocessing transformations are applied consistently between training and serving.
The core transformation code from the batch Dataflow pipeline needs to be extracted and made available to the real-time serving environment. This ensures that the exact same logic is applied to data before training and before real-time prediction, preventing training-serving skew. By refactoring the code, it can be integrated into the serving application without relying on the Dataflow pipeline itself for real-time processing.
This choice is identical to B, implying the same correct technical reason for refactoring the code.
Batching real-time requests and using Dataflow for processing would introduce significant latency, making it unsuitable for real-time inference requirements.
Concept tested: Training-serving skew prevention, data preprocessing consistency
Source: https://cloud.google.com/architecture/overview-ml-training-serving-skew
Topics
Community Discussion
No community discussion yet for this question.