Skip to content

Release 0.8.450

Need Help?

The SuperCowPowers team is happy to give any assistance needed when setting up AWS and Workbench. So please contact us at workbench@supercowpowers.com or on chat us up on Discord

Training predictions split into two files and full_cross_fold becomes leak-free. Hyperparameter search now ranks whole trials.

Out-of-fold and validation predictions are separate files

Training produces two kinds of scored rows, and they measure different things:

  • Out-of-fold — each row scored by the one fold model that held it out. Single-model, leak-free.
  • Held-out validation — your validation_ids rows, scored by the full ensemble.

You can't get a leak-free ensemble score on training rows — for any fold's held-out rows, the other four models trained on them. So the two are genuinely different estimators, and they now write to oof_predictions.csv and val_predictions.csv instead of sharing one file with a marker column.

Why it matters: full_cross_fold was computed over both row types at once. A model with 13,170 training rows and 9,442 validation rows drew ~42% of its reported metric from ensemble predictions. It now reads out-of-fold only.

Held-out rows also carry real uncertainty now — scored through the same path production inference uses for a new molecule — so val_predictions.csv is a direct read on whether production UQ is calibrated.

Models using model_class (scikit-learn) train on a single split, so they write an empty oof_predictions.csv and no longer produce a full_cross_fold capture.

Hyperparameter search ranks whole trials

Pruning is gone. Every trial now trains its full ensemble, so every objective value sits on the same basis and the shortlist compares like with like. The objective is the pooled out-of-fold error (cv_mae), or holdout_mae when validation_ids are designated.

Budgets are sized per framework when you don't set n_trials — 40 ChemProp, 100 PyTorch, 250 XGBoost, scaled to what one trial costs.

trial_counts reports failures. completed should equal attempted; any failed means the budget was quietly spent on trials that died (CUDA OOM is the usual cause when trials share a GPU).

Data capture

Endpoints declare CSV and JSON content types explicitly, and captured data is parsed in a single pass. Backup is now a thin aws s3 sync wrapper rather than a bespoke script.

REPL and Bosco

Named sessions with a recency-sorted listing, launched Batch jobs watched and reported on completion, and light/dark REPL themes driven by REPL_THEME.

Also

  • Model exposes training-job utilization; training-job readers moved out of model_utils.
  • Inference image carries the X11 and libexpat libraries ChemProp's cuik-molmaker needs.
  • Security bumps: aiohttp 3.14.3, GitPython 3.1.58.

Upgrade notes

  • Retrain when convenient, not urgently. Models trained before this release are read through their old capture, filtered to the out-of-fold rows — so they report the corrected leak-free metric without retraining. Each read logs a warning naming the model.
  • Model.get_inference_predictions("model_training") now resolves to out-of-fold predictions.
  • Dashboard views read stored inference captures, so existing model pages are unaffected.
  • No client code changes are required.

Questions?

The SuperCowPowers team is happy to answer any questions you may have about AWS and Workbench. Please contact us at workbench@supercowpowers.com or on chat us up on Discord