Tomorrow’s Churn Model
A retention model posts extraordinary validation results. Determine which features knew the future before a pilot reaches customers.
- ROLE
- Model Validation Analyst
- TIMEBOX
- 75–105 minutes
- COMPLEXITY
- Intermediate / 3 of 5
- ROTATION
- Brief 6 of 16
The decision has already reached your desk.
A contractor cancellation model reports near-perfect offline discrimination from a numeric wildcard and a random 80/20 snapshot-row split. Product wants an account-level campaign file.
The feature table contains 213 columns and repeated account snapshots. Its final fields contain realized future outcomes. The submitted split operates on rows, while the proposed use scores an account at feature_as_of_at for cancellation within the following 90 days.
You have enough time to reproduce the submitted result, inspect the predictor set and test account overlap—not to complete a production replacement model.
Approve, quarantine or reject the submitted model and decide whether any account-level score may leave the governance review.
Treat feature_as_of_at as the scoring instant and future_90d_cancelled_flag as the submitted target. Reproduction alone cannot authorize customer action.
Run the brief as a controlled assignment.
The workbench mounts only the listed source neighborhood and supplies neutral starter worksheets. It does not grade the conclusion or reveal the mechanisms planted in the larger assignment.
- Open the dedicated brief workspace and confirm the brief ID in the queue.
- Establish table grain, cutoff and control totals before joining or modeling.
- Make at least two distinct evidence moves and test a credible rival explanation.
- Leave the requested polished artifact, then export the workspace or portfolio package.
The source files are shared; the draft workspace is not. Changing to another full assignment drops brief mode deliberately.
Real tables. A deliberately bounded neighborhood.
These Parquet files already belong to Meridian’s public 96-table estate. Use the mounted schema.table names in SQL or Python; download links are provided for learners working in a local DuckDB environment.
GRAIN / One account feature snapshot; multiple rows per account are expected.
CAUTION / All 213 columns are retained, including realized future outcomes in the final fields.
GRAIN / One submitted partition label per feature snapshot.
CAUTION / The contractor supplied a random row-level 80/20 split; repeated accounts may cross the boundary.
ANALYSIS CUTOFF / 20 Jan 2026 / 09:15 ET deployment gate
Questions to pressure-test—not steps to copy.
These prompts define the analytical territory without prescribing an order, technique or conclusion.
- 01
Which fields are consequences or near-consequences of the target event?
- 02
Can repeated account snapshots cross the training and validation boundary?
- 03
Which source-watermark, pipeline and status-event clocks could establish point-in-time availability?
Leave work another analyst can review.
Artifact presence can be recorded; analytical quality remains a human judgment. A complete brief has evidence, reasoning and a decision—not merely executed code.
- 01Submitted-result reproduction
Reproduce the wildcard/split logic and report selected columns, validation performance and account overlap.
- 02Predictor quarantine
Classify material predictors as allow, exclude or investigate using names, future-field position and available clocks.
- 03Governance disposition
Deliver a polished release gate covering model use, account-level export and the minimum grouped-forward rerun.
Review the reasoning after a real attempt.
The debrief does not contain an official answer. It identifies defensible analytical moves, common failure modes and questions a reviewer may use to challenge the handoff.
SPOILER-GATED REVIEW / REVEAL AFTER YOUR FIRST HANDOFF
DEBRIEF REVEALED / THIS MAY CHANGE HOW YOU APPROACH THE BRIEF
Leakage is a mismatch between what the model is allowed to know and what the validation lets it know. Feature names alone cannot settle that question.
Defensible approaches
- Anchor every feature to the scoring event and ask when the value became available in the production path, including backfills and later updates.
- Measure account overlap in the supplied split and outline a grouped forward design that separates entity leakage from temporal validity.
- Define a minimal safe feature allowlist and a validation gate whose cohort, cutoff and 90-day horizon correspond to the proposed use.
Common traps
- Accepting an “available” flag without reconciling materialization timing.
- Removing only the most obviously named leakage field while leaving post-cutoff workflow features.
- Dropping only columns beginning with future_ while ignoring repeated-entity and source-watermark problems.
Reviewer questions
- Could each allowed value have been observed at the scoring instant without a later backfill?
- Are snapshots from one account isolated appropriately across evaluation boundaries?
- Does the release gate define evidence, not merely request “more testing”?
CHALLENGE A COLLEAGUE
Pass the Brief
The link shares this spoiler-free briefing. It never includes your work, identity, or browser progress.SUGGESTED NOTEA 0.93 AUC looked different after asking what the model knew at scoring time. I audited Tomorrow’s Churn Model from The Analyst.
