MS-DS Master of Data science Model Evaluation and Validation 2 — Questions and Answers
Question 1: Which evaluation metric is most appropriate when the cost of a false negative is much higher than the cost of a false positive?
- Precision
- Recall (Correct answer)
- Specificity
- Accuracy
Correct answer: Recall
Recall (sensitivity) measures the proportion of actual positives correctly identified, making it critical when missing a positive case (false negative) is very costly.
Question 2: In k-fold cross-validation, what happens as k increases toward n (leave-one-out CV)?
- Bias increases and variance decreases
- Bias decreases and variance increases (Correct answer)
- Both bias and variance decrease
- Both bias and variance increase
Correct answer: Bias decreases and variance increases
As k approaches n, each fold uses nearly all data for training (low bias), but the estimates become highly variable across folds (high variance).
Question 3: The Brier score is used to evaluate which type of model output?
- Class labels from a classifier
- Probability estimates from a probabilistic classifier (Correct answer)
- Feature importance rankings
- Regression residuals
Correct answer: Probability estimates from a probabilistic classifier
The Brier score measures the mean squared difference between predicted probabilities and actual binary outcomes, assessing probabilistic calibration.
Question 4: A model achieves 99% accuracy on a dataset where 99% of examples are negative. This is an example of:
- Overfitting
- Underfitting
- The accuracy paradox (Correct answer)
- Data leakage
Correct answer: The accuracy paradox
The accuracy paradox occurs when high accuracy is misleading because a naive model predicting the majority class always scores well on imbalanced datasets.
Question 5: What does the area under the Precision-Recall curve (AUC-PR) measure?
- The tradeoff between sensitivity and specificity
- The overall quality of ranked probability predictions for imbalanced datasets (Correct answer)
- The model's average calibration error
- The proportion of variance explained by the model
Correct answer: The overall quality of ranked probability predictions for imbalanced datasets
AUC-PR summarizes the precision-recall tradeoff across all thresholds and is preferred over AUC-ROC when classes are highly imbalanced.
Question 6: Which technique specifically tests whether a model has learned spurious correlations by evaluating it on data where the target variable is shuffled?
- Ablation study
- Permutation test (Correct answer)
- Bootstrap validation
- Stratified sampling
Correct answer: Permutation test
A permutation test shuffles the labels and measures model performance; if the model scores high on shuffled data, it likely learned noise rather than true signal.
Question 7: Expected Calibration Error (ECE) measures:
- The gap between a model's confidence and its actual accuracy (Correct answer)
- The difference between training loss and validation loss
- The mean absolute error of probability predictions
- The variance of predictions across bootstrap samples
Correct answer: The gap between a model's confidence and its actual accuracy
ECE quantifies how well a model's predicted probabilities align with empirical frequencies, with lower ECE indicating better-calibrated predictions.
