You work for a food product company. Your company's historical sales data is stored in BigQuery.You need to use Vertex AI's custom training service to train multiple TensorFlow models that read the da

Using Spark on Dataproc for transformations introduces additional operational overhead, cluster management, and potentially higher cost and development effort compared to native BigQuery SQL for common transformations.

Performing min-max scaling and bucketing directly in BigQuery using SQL queries leverages its highly optimized, serverless processing capabilities. This minimizes preprocessing time and cost as data remains in-place, and for common transformations like these, SQL offers a low-effort development path compared to external processing frameworks.

Adding these transformations as preprocessing layers within each TensorFlow model would lead to redundant computation on the large dataset for every training run, significantly increasing training time and cost rather than minimizing preprocessing effort upfront.

Creating a Dataflow pipeline involves writing Beam code and managing a Dataflow job, which adds more development effort and complexity than simply using SQL queries within BigQuery for easily expressible transformations.