PROFESSIONAL-MACHINE-LEARNING-ENGINEER · Question #244
You are building a custom image classification model and plan to use Vertex AI Pipelines to implement the end-to-end training. Your dataset consists of images that need to be preprocessed before…
The correct answer is B. DataflowPythonJobOp, WaitGcpResourcesOp, and CustomTrainingJobOp. To preprocess images with existing Python functions and then train a custom model in a Vertex AI Pipeline, use DataflowPythonJobOp for scalable preprocessing and CustomTrainingJobOp for model training.
Question
Options
- ADataprocSparkBatchOp and CustomTrainingJobOp
- BDataflowPythonJobOp, WaitGcpResourcesOp, and CustomTrainingJobOp
- Cdsl.ParallelFor, dsl.component, and CustomTrainingJobOp
- DImageDatasetImportDataOp, dsl.component, and AutoMLImageTrainingJobRunOp
How the community answered
(18 responses)- A11% (2)
- B83% (15)
- C6% (1)
Why each option
To preprocess images with existing Python functions and then train a custom model in a Vertex AI Pipeline, use `DataflowPythonJobOp` for scalable preprocessing and `CustomTrainingJobOp` for model training.
`DataprocSparkBatchOp` would be for Spark-based processing, not directly leveraging existing Python functions in a general Dataflow context for image preprocessing.
`DataflowPythonJobOp` is suitable for executing custom Python preprocessing functions at scale using Dataflow, providing a robust and managed service for data transformation. `CustomTrainingJobOp` is then used to orchestrate the custom model training job after preprocessing, and `WaitGcpResourcesOp` ensures the Dataflow job completes before training starts.
`dsl.ParallelFor` and `dsl.component` are general Kubeflow Pipelines constructs for defining pipeline steps and parallelism, but they don't specify the underlying managed service for large-scale data processing like Dataflow.
`ImageDatasetImportDataOp` is for importing data, not for custom preprocessing, and `AutoMLImageTrainingJobRunOp` is for AutoML training, not custom model training as specified in the question.
Concept tested: Vertex AI Pipelines, custom preprocessing with Dataflow
Source: https://cloud.google.com/vertex-ai/docs/pipelines/run-python-code-on-dataflow-python-job-op
Topics
Community Discussion
No community discussion yet for this question.