Release 0.8.398

Need Help?

The SuperCowPowers team is happy to give any assistance needed when setting up AWS and Workbench. So please contact us at workbench@supercowpowers.com or on chat us up on Discord

The headline of this release is first-class validation-set support for model training. Along the way, sample_weight is de-overloaded into three explicit, orthogonal row roles — so designating a held-out set, dropping an outlier, and weighting a row are no longer the same knob.

First-class validation sets

A model can now designate a held-out validation set that the training container sees during training: the rows are kept in the training view, routed out of the train/CV path, and scored as an honest held-out read (holdout_mae). This works across all three model frameworks (XGBoost, PyTorch, ChemProp) and unblocks in-training validation and HPO.

fs.to_model(
    name="my-model",
    model_type=ModelType.REGRESSOR,
    model_framework=ModelFramework.XGBOOST,
    target_column="activity",
    validation_ids=holdout_ids,   # scored, never trained
)

Held-out predictions land in validation_predictions.csv with a validation column (True for held-out rows), and the primary-target held-out MAE is logged as holdout_mae.

Row roles: `sample_weight`, `validation`, `exclude`

Previously, sample_weight == 0 did triple duty — a framework weight, an "exclude this row" flag, and a holdout marker. These are now separate:

role	how to set	in training data?	trains?	scored held-out?
weight	`sample_weights={id: w}`	yes	yes, at weight `w`	via CV
exclude (outlier/anomaly)	`exclude_ids=[...]`	no (dropped)	no	no
validation (holdout)	`validation_ids=[...]`	yes, marked	no	yes

sample_weight is now a pure framework weight, forwarded as-is to the model script. exclude wins if an id is in both exclude_ids and validation_ids.

Behavior change: sample_weight == 0 no longer drops rows

Zero-weight rows used to be silently excluded from the training view. That is no longer true — a weight of 0 is now just a zero framework weight. Use exclude_ids to drop rows and validation_ids to hold them out. See the porting guide below. Existing models are unaffected until they retrain (a model's script and training view are co-created and co-owned, so migration is per-model).

Porting guide: `sample_weights={id: 0.0}` → roles

If you designated rows by zero-weighting them, pick the role that matches your intent:

Excluding outliers / anomalies (rows that should never be seen):

# before
sample_weights = {row_id: 0.0 for row_id in bad_ids}
fs.to_model(..., sample_weights=sample_weights)

# after
fs.to_model(..., exclude_ids=bad_ids)

Holding rows out for a scored validation read (the PXR / temporal pattern):

# before — zero-weight the holdout, score it later via endpoint.inference
sample_weights = {row_id: 0.0 for row_id in holdout_ids}
fs.to_model(..., sample_weights=sample_weights)

# after — held out AND scored in-training (holdout_mae)
fs.to_model(..., validation_ids=holdout_ids)

Actual fractional weighting is unchanged — non-zero weights still work exactly as before:

fs.to_model(..., sample_weights={row_id: 0.5 for row_id in down_weighted})

`temporal_split` now returns a list

FeatureSet.temporal_split() returns a list of holdout ids (was a {id: 0.0} dict), to feed validation_ids directly:

# before
sample_weights = fs.temporal_split("date_col", end_date="2025-10-17")   # {id: 0.0}
fs.to_model(..., sample_weights=sample_weights)

# after
validation_ids = fs.temporal_split("date_col", end_date="2025-10-17")   # [id, id, ...]
fs.to_model(..., validation_ids=validation_ids)

Combined pattern (temporal holdout + anomaly exclusion)

The matched pair — designate the holdout at to_model, confirm on the endpoint with ts_inference — still holds; the roles just become explicit:

exclude_ids = list(compute_sample_weights(...))          # anomalies → dropped
validation_ids = fs.temporal_split("date_col", end_date="2025-10-17")
model = fs.to_model(..., exclude_ids=exclude_ids, validation_ids=validation_ids)
...
end.ts_inference("date_col", after_date="2025-10-17", exclude_ids=exclude_ids)  # unchanged

Anomalies that fall in the post-date window get exclude precedence — dropped from both training and the held-out read.

Ray / awswrangler coexistence

Workbench now pins awswrangler to its single-node Python engine at import. With ray present in an environment (e.g. HPO's ray[tune]), awswrangler would otherwise auto-switch to its distributed engine and break the single-node boto3 calls Workbench relies on for Athena/S3. Ray Tune is unaffected — it's an independent consumer of Ray.

Upgrade notes

Training images must be rebuilt on this workbench version — the templates import the new workbench.training.validation util (base and pytorch_chem training images).
No client code changes are required unless you relied on sample_weight == 0 to drop rows — see the porting guide.

Questions?

The SuperCowPowers team is happy to answer any questions you may have about AWS and Workbench. Please contact us at workbench@supercowpowers.com or on chat us up on Discord