You have created a Vertex AI pipeline that includes two steps. The first step preprocesses 10 TB data completes in about 1 hour, and saves the result in a Cloud Storage bucket. The second step uses th

Adding new pipeline parameters and steps increases complexity and does not directly address the efficiency of re-running the preprocessing step or the cost of redundant computations.

Creating another pipeline and hardcoding paths is a manual, non-reproducible, and unmanaged approach that bypasses pipeline lineage and introduces maintenance overhead, rather than leveraging pipeline features for efficiency.

While optimizing machine resources for training might speed up that specific step, it does not address the cost and time incurred by repeatedly re-running the 1-hour preprocessing step, which is the primary target for optimization here.

Enabling caching for the pipeline job allows the expensive 10 TB preprocessing step to reuse its output if its inputs haven't changed, saving significant time and cost. Disabling caching specifically for the model training step ensures that each change to the model code (testing different algorithms) triggers a fresh training run, preventing outdated results from being served from the cache.