CERTIFIED-DATA-ENGINEER-PROFESSIONAL · Question #104
The data science team has created and logged a production model using MLflow. The model accepts a list of column names and returns a new column of type DOUBLE. The following code correctly imports the
The correct answer is C. df.select("customer_id", model(*columns).alias("predictions")). When a logged MLflow model is loaded and used in a Spark context, it behaves as a Spark UDF. The correct syntax is df.select('customer_id', model(*columns).alias('predictions')), which applies the model UDF to the unpacked feature columns and returns a DataFrame with exactly the
Question
The data science team has created and logged a production model using MLflow. The model accepts a list of column names and returns a new column of type DOUBLE. The following code correctly imports the production model, loads the customers table containing the customer_id key column into a DataFrame, and defines the feature columns needed for the model. Which code block will output a DataFrame with the schema "customer_id LONG, predictions DOUBLE"?
Options
- Amodel.predict(df, columns)
- Bdf.map(lambda x:model(x[columns])).select("customer_id, predictions")
- Cdf.select("customer_id", model(*columns).alias("predictions"))
- Ddf.apply(model, columns).select("customer_id, predictions")
- Edf.select("customer_id", pandas_udf(model, columns).alias("predictions"))
How the community answered
(25 responses)- A8% (2)
- C76% (19)
- D4% (1)
- E12% (3)
Explanation
When a logged MLflow model is loaded and used in a Spark context, it behaves as a Spark UDF. The correct syntax is df.select('customer_id', model(*columns).alias('predictions')), which applies the model UDF to the unpacked feature columns and returns a DataFrame with exactly the two requested columns. Option A is not valid Spark syntax. Option B uses Python's map which breaks the distributed execution model. Option D uses pandas-style apply which doesn't work on Spark DataFrames this way. Option E incorrectly wraps the already-loaded model with pandas_udf(), which is not how you invoke a pre-loaded MLflow model.
Topics
Community Discussion
No community discussion yet for this question.