You have a custom job that runs on Vertex AI on a weekly basis. The job is implemented using a proprietary ML workflow that produces the datasets, models, and custom artifacts, and sends them to a Clo

The Vertex AI Metadata API is designed for tracking ML metadata, including the lineage of datasets, models, and custom artifacts across different executions and contexts. Using this API directly within the custom job ensures that all necessary information, including the link between models and their creating artifacts, is recorded for compliance and traceability.

Vertex AI Experiments track runs, and autologging helps, but it may not capture the specific custom artifacts or the detailed lineage required for compliance beyond what's predefined.

TensorFlow Extended (TFX) ML Metadata is an open-source solution, but the question implies using managed Google Cloud services where Vertex AI Metadata is the native and managed alternative.

The Vertex AI Model Registry manages model versions, but it primarily focuses on models themselves; it does not inherently provide the detailed lineage tracking to associated datasets and custom artifacts with the same granularity as the Metadata API.