MS-DS Master of Data science Master of Data science Model Evaluation and Validation 1 — Questions and Answers
Question 1: Which of the following best describes the purpose of a calibration curve in model evaluation?
- To measure the trade-off between precision and recall at various thresholds
- To compare predicted probabilities against actual outcome frequencies (Correct answer)
- To visualize the decision boundary of a classifier
- To plot training loss against validation loss across epochs
Correct answer: To compare predicted probabilities against actual outcome frequencies
A calibration curve (reliability diagram) plots the mean predicted probability against the actual fraction of positive outcomes in bins, revealing whether a model's confidence scores are trustworthy — a perfectly calibrated model falls on the diagonal.
Question 2: You train a logistic regression model and observe AUC-ROC = 0.97 on the training set but AUC-ROC = 0.61 on the test set. Which action most directly addresses this problem?
- Increase the learning rate
- Apply stronger regularization (increase λ) (Correct answer)
- Switch the evaluation metric to F1-score
- Use a larger batch size during training
Correct answer: Apply stronger regularization (increase λ)
A large gap between training and test AUC-ROC indicates overfitting (high variance). Increasing the regularization strength penalizes model complexity, reducing the gap between train and test performance.
Question 3: What does the Brier Score measure in probabilistic model evaluation?
- The area under the precision-recall curve
- The mean squared error between predicted probabilities and actual binary outcomes (Correct answer)
- The ratio of true positives to the sum of true positives and false negatives
- The logarithmic loss averaged over all samples
Correct answer: The mean squared error between predicted probabilities and actual binary outcomes
The Brier Score computes the mean squared difference between the predicted probability and the true binary label (0 or 1). Lower scores indicate better-calibrated probabilistic predictions, with 0 being perfect and 1 being worst.
Question 4: In a multi-class classification problem with severe class imbalance, which averaging strategy for F1-score best accounts for class frequency when reporting overall model performance?
- Macro-average
- Weighted-average (Correct answer)
- Micro-average
- Binary-average
Correct answer: Weighted-average
Weighted-average F1-score calculates F1 for each class independently and then takes the average weighted by the number of true instances per class, making it sensitive to class imbalance and giving more influence to larger classes.
Question 5: Which technique is specifically designed to estimate the generalization error of a model when the available dataset is very small (e.g., fewer than 50 samples)?
- Holdout validation with a 70/30 split
- 5-fold cross-validation
- Leave-one-out cross-validation (LOOCV) (Correct answer)
- Bootstrap aggregating (bagging)
Correct answer: Leave-one-out cross-validation (LOOCV)
LOOCV uses every single sample as a test set exactly once, maximizing training data usage (n-1 samples per fold). This makes it ideal for very small datasets where a standard train/test split would leave too few samples for reliable training or evaluation.
Question 6: A data scientist reports that their model achieves 95% accuracy on a binary classification task. The dataset contains 95% negative and 5% positive examples. What is the most critical concern about this result?
- The model likely suffers from underfitting due to the high accuracy
- The accuracy metric is misleading because a trivial classifier predicting all negatives achieves the same score (Correct answer)
- The model should be re-evaluated using mean absolute error instead
- A 95% accuracy on imbalanced data always indicates a well-performing model
Correct answer: The accuracy metric is misleading because a trivial classifier predicting all negatives achieves the same score
When the majority class represents 95% of the data, a naïve classifier that always predicts the negative class will also achieve 95% accuracy without learning anything meaningful. Metrics such as precision, recall, F1-score, or AUC-ROC are needed to reveal true performance.
