Why improving cohorts is worth the effort
Cohort analysis groups scanners by when they first scanned, such as the week of first scan, then follows each group over time. It reveals whether newer scanners behave differently from earlier ones, which a single total hides.Cohorts make change visible. If each new cohort retains worse than the last, something in the experience is getting worse.
Diagnose before you change anything
Improvement starts with diagnosis, not with redesign. Before changing a code, split cohorts by first-scan week, placement, and offer and look for the weakest segment. The weakest segment is where a fix buys the most, because it holds the largest pool of lost value. Also check the steps around cohorts: a problem upstream, such as poor placement, can masquerade as a problem with the code itself, and a problem downstream, such as a slow landing page, can waste scans that were earned fairly. Fix the constraint, not the symptom.Write down your diagnosis in one paragraph before you touch anything. State what you observed, which segment is weakest, and what you believe is causing it. This paragraph is your hypothesis, and it keeps the work honest: if the fix does not move the number, the hypothesis was wrong, and you learn something instead of merely changing things.
Levers that move cohorts
- Compare cohorts at the same age, such as 30 days after first scan. - Investigate what changed when a cohort performs unusually well or badly. - Tailor follow-up to the cohort that needs it most.Work through the levers in order of expected impact per unit of effort. A lever that costs nothing and affects every scanner beats a lever that costs a print run and affects one placement. For each lever, state the change, the cost, and the segment it should move, then implement only that change. Bundling several changes together feels productive, but it destroys your ability to learn which one worked.
Test the change properly
Cohort retention = members of a cohort who return within a window divided by the cohort size, multiplied by 100.Illustrative example numbers, not real-world data: a January cohort of 200 scanners has 60 return within 30 days, giving 60 / 200 x 100 = 30% cohort retention. A February cohort of 200 scanners with 40 returning gives 20%, so February scanners retained less well in this example.
Run the change as a comparison wherever you can: the old version against the new version, on comparable placements, over the same kind of period. Keep everything else stable, and decide in advance how long the test runs and what counts as a win. A test that ends the moment one variant pulls ahead is not a test, it is a reaction. If traffic is too small for a formal test, change one thing, record the before and after numbers, and treat the result as suggestive rather than proven.