Completed
research
MetroPT Predictive Maintenance
Early detection of air leaks in a metro-train compressor 30 minutes before failure — 1.5M sensor readings, 16 models compared, tuned GradientBoosting at 0.993 ROC-AUC
By the numbers
0
ROC-AUC (test)
0
PR-AUC (test)
0M
Sensor readings
0 min
Lead time
The Problem
What I was solving
Only about 2% of the windows are positive, and neighbouring windows are autocorrelated — so a random split leaks the future into training and every accuracy number becomes meaningless. On top of that, at the default 0.5 threshold the strongest models look useless: RandomForest scores F1 = 0.000 and GradientBoosting F1 = 0.231, which would normally get them thrown out.
My Approach
How I built it
Time-based split (train up to 1 June 2020, test from that date), TimeSeriesSplit with three folds inside training, and PR-AUC as the primary metric because ROC-AUC flatters imbalanced data. Sixteen models under an identical protocol, then RandomizedSearch plus explicit threshold tuning on the winner — the threshold is a hyperparameter here, not a default. SHAP on the final model to check the reasoning rather than trust the score.
Tech choices
- Time-based split + TimeSeriesSplit— The windows overlap and neighbouring rows are nearly identical. A shuffled split would put the same failure on both sides of the line and return a beautiful, fake score.
- Threshold tuning as a step, not a default— At 0.5 the best model scored F1 0.231 and looked like a failure. Fitting the threshold on validation turned the same model into F1 0.878 — on imbalanced data the cutoff carries as much weight as the algorithm.
- PR-AUC as the primary metric— With ~2% positives, ROC-AUC stays high even for a model that misses most failures. PR-AUC is the one that moves when recall on the rare class is actually bad.
- SHAP over correlation— The correlation matrix and the model disagreed about which sensor mattered. SHAP explains the model that will actually run, so it wins that argument.
Outcome
What came out of it
The tuned GradientBoosting with a fitted threshold reaches ROC-AUC 0.993, PR-AUC 0.889, F1 0.878 (precision 0.873 / recall 0.882) on the held-out test period — up from F1 0.231 for the same algorithm at the default threshold. The most interesting result is not the score: SHAP put Oil_temperature_min first with a mean |SHAP| of 0.361, while linear correlation had pointed at DV_pressure_mean. Pearson correlation ignores feature interactions and SHAP does not — which is exactly why the feature the correlation matrix liked was not the one the model relied on.