The Lift in the Fine Print
A navigation experiment reports more pages per session. Decide whether that engagement signal and its commercial guardrails support rollout.
- ROLE
- Experiment Review Analyst
- TIMEBOX
- 90–120 minutes
- COMPLEXITY
- Advanced / 4 of 5
- ROTATION
- Brief 4 of 16
The decision has already reached your desk.
The mobile_navigation experiment ended on 30 April. Product reports higher pages per session and calls that evidence of better navigation.
Randomization occurred at the session-assignment grain. The estate separately records first exposure, web events, a converted-order pointer and order timestamps. More events can indicate useful discovery, confusion, or simply longer paths.
The release branch closes at 15:30. Product needs a ship, hold, scoped-release or retest vote with magnitude, uncertainty and a commercial guardrail.
Make a rollout recommendation that preserves intent-to-treat assignment and distinguishes engagement from customer and commercial benefit.
Freeze the readout at 05 May 2025 09:00 ET. Do not condition the primary result on post-assignment behavior or count event rows as independent experimental units.
Run the brief as a controlled assignment.
The workbench mounts only the listed source neighborhood and supplies neutral starter worksheets. It does not grade the conclusion or reveal the mechanisms planted in the larger assignment.
- Open the dedicated brief workspace and confirm the brief ID in the queue.
- Establish table grain, cutoff and control totals before joining or modeling.
- Make at least two distinct evidence moves and test a credible rival explanation.
- Leave the requested polished artifact, then export the workspace or portfolio package.
The source files are shared; the draft workspace is not. Changing to another full assignment drops brief mode deliberately.
Real tables. A deliberately bounded neighborhood.
These Parquet files already belong to Meridian’s public 96-table estate. Use the mounted schema.table names in SQL or Python; download links are provided for learners working in a local DuckDB environment.
GRAIN / One registered experiment.
CAUTION / This row defines the dates, primary metric and assignment unit; it is design metadata rather than outcome evidence.
GRAIN / One randomized session assignment.
CAUTION / Preserve the full assignment population; first exposure is post-assignment and warehouse timing controls inclusion.
GRAIN / One web session.
CAUTION / The assigned session is the experiment unit; converted_order_id is only a pointer and does not certify attribution.
ANALYSIS CUTOFF / 05 May 2025 / 09:00 ET
Questions to pressure-test—not steps to copy.
These prompts define the analytical territory without prescribing an order, technique or conclusion.
- 01
Does the assignment population reconcile one-to-one with sessions, and how much first-exposure loss exists by variant?
- 02
What does pages per assigned session measure, and what rival user behavior could create the same lift?
- 03
Do conversion and order-value guardrails change the rollout decision once timing and assignment grain are preserved?
Leave work another analyst can review.
Artifact presence can be recorded; analytical quality remains a human judgment. A complete brief has evidence, reasoning and a decision—not merely executed code.
- 01Assignment audit
Reconcile assignment, session and exposure coverage and document any exclusion before estimating effects.
- 02Effect and guardrail view
Estimate the primary pages-per-session effect with uncertainty and pair it with an order-based commercial check.
- 03Release-council slide
Deliver a polished rollout vote, evidence, counterevidence and one follow-up condition on a single screen.
Review the reasoning after a real attempt.
The debrief does not contain an official answer. It identifies defensible analytical moves, common failure modes and questions a reviewer may use to challenge the handoff.
SPOILER-GATED REVIEW / REVEAL AFTER YOUR FIRST HANDOFF
DEBRIEF REVEALED / THIS MAY CHANGE HOW YOU APPROACH THE BRIEF
The brief tests whether the analyst can keep assignment, exposure, estimand, and exploration separate while still making a release decision.
Defensible approaches
- Lead with the assignment-based comparison because randomization occurs at assignment, then use exposure views diagnostically.
- Reduce web events to session-level outcomes before comparing variants and report effect magnitude with an uncertainty interval.
- Build a cutoff-safe order guardrail from converted_order_id and order timing, while keeping engagement and commercial claims separate.
Common traps
- Conditioning only on exposed users and continuing to describe the estimate as randomized.
- Counting event rows as independent observations or using them to define the experiment population.
- Calling more navigation a customer benefit without testing a plausible rival interpretation.
Reviewer questions
- Did you name the effect your calculation actually estimates?
- Would your recommendation change if engagement rose but order conversion did not?
- What release option preserves learning while containing downside?
CHALLENGE A COLLEAGUE
Pass the Brief
The link shares this spoiler-free briefing. It never includes your work, identity, or browser progress.SUGGESTED NOTEA “winning” navigation experiment changed meaning when engagement met its guardrails. I reviewed The Lift in the Fine Print from The Analyst.
