Every Click Is Not a Page
A product KPI reconciles physically but may not mean what its label claims. Audit the event taxonomy before the metric becomes a permanent dashboard contract.
- ROLE
- Product Analytics Engineer
- TIMEBOX
- 75–105 minutes
- COMPLEXITY
- Intermediate / 3 of 5
- ROTATION
- Brief 12 of 16
The decision has already reached your desk.
The navigation experiment readout describes a session summary field as pages per session. Product wants to retain the KPI in the permanent weekly dashboard after the experiment closes.
The session count can be reconciled to the event log, but the event taxonomy mixes page views, product and plan views, searches, support views, cart actions and purchases. Physical completeness and semantic validity are separate questions.
You own the data contract. Decide whether the KPI should be certified, renamed, redefined or retired and specify how historical reporting should transition.
Certify, rename, redefine or retire the published engagement KPI and define the production dashboard migration.
Preserve experiment assignment as the denominator and event sequence as recorded. Row-count reconciliation does not prove semantic validity, and no event type may be reclassified without a declared rule.
Run the brief as a controlled assignment.
The workbench mounts only the listed source neighborhood and supplies neutral starter worksheets. It does not grade the conclusion or reveal the mechanisms planted in the larger assignment.
- Open the dedicated brief workspace and confirm the brief ID in the queue.
- Establish table grain, cutoff and control totals before joining or modeling.
- Make at least two distinct evidence moves and test a credible rival explanation.
- Leave the requested polished artifact, then export the workspace or portfolio package.
The source files are shared; the draft workspace is not. Changing to another full assignment drops brief mode deliberately.
Real tables. A deliberately bounded neighborhood.
These Parquet files already belong to Meridian’s public 96-table estate. Use the mounted schema.table names in SQL or Python; download links are provided for learners working in a local DuckDB environment.
GRAIN / One experiment contract.
CAUTION / The contract sets scope and timing but does not validate a downstream metric label.
GRAIN / One randomized session assignment.
CAUTION / Assignment—not observed event presence—defines the analysis population.
GRAIN / One summarized web session.
CAUTION / event_count is a stored summary whose label and construction require separate validation.
ANALYSIS CUTOFF / 05 May 2025 / 09:00 ET
Questions to pressure-test—not steps to copy.
These prompts define the analytical territory without prescribing an order, technique or conclusion.
- 01
Does the session summary reconcile to event rows, and what does that establish or fail to establish?
- 02
Which event types and sequences create the difference described by the opening readout?
- 03
What definitions could defensibly represent page exposure, navigation effort or meaningful engagement?
- 04
Does the interpretation remain coherent across device types and key journey outcomes?
Leave work another analyst can review.
Artifact presence can be recorded; analytical quality remains a human judgment. A complete brief has evidence, reasoning and a decision—not merely executed code.
- 01Session-event reconciliation
Test completeness and one-session assignment without confusing those controls with meaning.
- 02Event taxonomy contract
Classify pages, actions, outcomes and ambiguous events under explicit rules.
- 03KPI comparison
Compare the published measure with at least two defensible alternatives by variant and a material slice.
- 04Dashboard migration note
Specify the chosen definition, name, backfill boundary and stakeholder communication.
Review the reasoning after a real attempt.
The debrief does not contain an official answer. It identifies defensible analytical moves, common failure modes and questions a reviewer may use to challenge the handoff.
SPOILER-GATED REVIEW / REVEAL AFTER YOUR FIRST HANDOFF
DEBRIEF REVEALED / THIS MAY CHANGE HOW YOU APPROACH THE BRIEF
A metric can reconcile exactly and still be mislabeled. Physical controls, semantic contracts and decision usefulness each require separate evidence.
Defensible approaches
- Prove session-to-event reconciliation first, then separately define what each event type represents.
- Decompose the result by event type, sequence and variant while preserving the assigned-session denominator.
- Choose the production metric from its intended meaning and stability rather than selecting the most favorable experimental result.
Common traps
- Declaring the KPI valid because the stored count matches event rows.
- Calling every event a page or silently narrowing to one event type after inspecting the result.
- Designing a new label that breaks historical comparability without a migration boundary.
Reviewer questions
- What real behavior does one unit of the recommended metric represent?
- Can historical and future values be compared under the proposed contract?
- Would a product leader mistake additional recorded actions for customer benefit?
CHALLENGE A COLLEAGUE
Pass the Brief
The link shares this spoiler-free briefing. It never includes your work, identity, or browser progress.SUGGESTED NOTEThe count reconciled perfectly—and the metric still needed a semantic audit. I rebuilt the contract in Every Click Is Not a Page from The Analyst.
