When working with smaller data sets (N<200), which method is preferred to perform honest assessment?

Question

Options

• ATraining: 40% Validation: 30% Testing: 30%
• BK-fold cross validation
• CCross validation using 4th quartile observations
• DUse the AIC goodness of fit statistic

• A
  14% (9)
• B
  6% (4)
• C
  75% (47)
• D
  5% (3)

Explanation

With small datasets (N<200), splitting off a dedicated holdout set for final testing wastes too many precious observations, so using the 4th quartile observations as the validation set is the preferred honest assessment method - it reserves only ~25% for testing while keeping ~75% available for model training, striking a practical balance between evaluation integrity and learning from limited data.

Why the distractors are wrong:

• A (40/30/30 split): A three-way partition is better suited for large datasets; with N<200, this leaves too few observations in each partition for stable model training or reliable evaluation.
• B (K-fold cross validation): While K-fold is excellent for model selection, the question asks about honest assessment (final performance estimation), which requires a true holdout set the model has never touched - K-fold reuses data across folds.
• D (AIC): AIC measures goodness-of-fit and is used for model comparison, not for generating an honest out-of-sample performance estimate.

Memory tip: Think "small N = save the last chunk." The 4th (final) quartile = the last 25% held out = honest because the model never sees it during training. "4th quartile for honest assessment on small data" - both "4th" and "final/held-out" rhyme conceptually.

No community discussion yet for this question.