GENERATIVE-AI-ENGINEER-ASSOCIATE · Question #2
An AI developer team wants to fine tune an open-weight model to have exceptional performance on a code generation use case. They are trying to choose the best model to start with. They want to…
The correct answer is A. Big Code Models Leaderboard B. Number of model parameters. Two attributes matter here: (A) The Big Code Models Leaderboard on HuggingFace specifically benchmarks models on code generation tasks (HumanEval, MBPP, etc.), making it the most relevant performance metric for this use case. (B) The number of model parameters directly…
Question
An AI developer team wants to fine tune an open-weight model to have exceptional performance on a code generation use case. They are trying to choose the best model to start with. They want to minimize model hosting costs, and are using Huggingface model cards and spaces to explore models. Which TWO model attributes and metrics should the team focus on to make their selection? (Choose two.)
Options
- ABig Code Models Leaderboard
- BNumber of model parameters
- CMTEB Leaderboard
- DChatbot Arena Leaderboard
- ENumber of model downloads last month
How the community answered
(30 responses)- A90% (27)
- C3% (1)
- D7% (2)
Explanation
Two attributes matter here: (A) The Big Code Models Leaderboard on HuggingFace specifically benchmarks models on code generation tasks (HumanEval, MBPP, etc.), making it the most relevant performance metric for this use case. (B) The number of model parameters directly correlates with model hosting costs - fewer parameters means smaller GPU/CPU requirements, lower memory usage, and cheaper inference. The MTEB Leaderboard (C) ranks text embedding models, not generative code models, so it is irrelevant. The Chatbot Arena Leaderboard (D) measures general conversational ability, not code generation quality. Number of downloads (E) reflects popularity but not task-specific performance or hosting cost.
Topics
Community Discussion
No community discussion yet for this question.