All work
Recall on bad cycles
94.2%
Precision on bad cycles
75.4%
AUC-ROC
0.988
Accuracy
94.38%

The problem

Bad cycles are rare, and a missed one costs far more than a false alarm: the hackathon's problem statement put an undetected bad cycle at $32,560 in losses. So the model is judged on how many bad cycles it catches, not on overall accuracy.

Data

  • 16,504 training cycles, recorded at 5-minute intervals from March to May 2020.
  • Each row holds furnace sensor readings and operating settings (22 columns).
  • About 15% of cycles are bad, so a model that always predicts "good" would already score about 85% accuracy.

Approach

  1. Imputed missing values: the mean for continuous columns, the mode for categorical ones.
  2. Rebalanced the training data by oversampling bad cycles with SMOTE.
  3. Compared Logistic Regression, Random Forest and XGBoost.
  4. Tuned with GridSearchCV and 3-fold cross-validation over 243 hyperparameter combinations, optimizing recall on bad cycles.
  5. Picked a Random Forest with the tuned settings.

Correlation analysis pointed to the strongest single sensors: B_15 (0.655), B_16 (0.631) and B_19 (0.613).

Results

Confusion matrix for the Random Forest on 3,239 held-out cycles: 2,588 good cycles predicted good, 153 good cycles flagged as bad, 29 bad cycles missed and 469 bad cycles caught.
Random Forest on 3,239 held-out cycles.
  • Caught 469 of 498 bad cycles (94.2% recall) and missed 29.
  • Raised 153 false alarms on good cycles, for 75.4% precision and an F1 score of 0.84.
  • 0.988 AUC-ROC and 94.38% overall accuracy, against the roughly 85% that always predicting "good" would score.