Appointment reminder experiment

Text reminders are one of the cheapest changes an outpatient service can make, and one of the easiest to get wrong in the analysis. This project runs a four-arm randomised experiment on synthetic outpatient appointments for a fictional acute hospital: no reminder, one text three days before, a second text the day before, and a text that names the cost of a missed appointment. It sizes the trial from pre-trial data before any outcome exists, then analyses it the way an analysis plan should.

All the data is synthetic, generated by the project code with a fixed seed. The effects are planted, so every estimate can be checked against the truth.

In short: two texts cut the DNA (did not attend) rate from 9.3% to 7.2%, a fall of 2.1 percentage points (95% CI 0.9 to 3.3). The cost message did about as well, at 1.8 points. One text probably helps, but this trial can’t show it: its interval runs from a 2.2-point fall to a 0.2-point rise. At 3p a text, reminders cost about £2.50 per DNA avoided, so money isn’t the hard part of the decision.

The question

A service manager wants to know three things: do texts reduce missed appointments, which wording works best, and is a second text worth sending? Whatever they choose will apply to every patient on the list, including those with no mobile number on record. That last point shapes the whole analysis.

Trial design

The cost-message arm borrows from two UK trials by Hallsworth and colleagues (PLOS ONE, 2015), where a reminder stating what a missed appointment costs the NHS had a DNA rate of 8.4%, against 11.1% for the existing message. I planted a smaller effect than that.

The analysis plan was fixed before any trial outcome existed:

• Unit of randomisation: the patient. Someone with three appointments gets the same arm for all three, so nobody receives mixed messages. It also means their appointments aren’t independent, so every interval is clustered by patient.
• Outcome: DNA, per appointment.
• Primary analysis: intention to treat (ITT). Every appointment counts in its assigned arm, whether a text reached the patient or not.
• Three comparisons, each text arm against no reminder, each tested at 0.05 ÷ 3 = 0.0167 (Bonferroni), so the chance of any false positive stays at or below 5%. I still show 95% intervals, because that’s what readers compare.
• Adjustment covariates: age band, sex, deprivation (Index of Multiple Deprivation, IMD, quintile of the patient’s area), mobile number on record, and DNAs in the prior year. All recorded before randomisation.
• No interim analysis. Sixteen weeks, one look at the end.

In a real service, randomising patients to different reminders counts as research under the Health Research Authority’s decision tool, so it would need the matching approvals; A/B testing in operations covers that side.

What’s in the data

Every pattern is a named constant at the top of trial_data.py.

Sizing the trial before looking

The sample size comes from 16 weeks of pre-trial appointments (19,357 of them, from 8,000 patients), with nothing from the trial itself. The baseline DNA rate was 10.0%. I set the smallest effect worth detecting at 2 percentage points: big enough to matter to a clinic, realistic for a reminder. Power 80%, two-sided test at 0.0167.

def appointments_per_arm(
    baseline: float, mde: float, alpha: float, power: float, design_effect: float = 1.0
) -> int:
    """Appointments needed in each arm to detect a fall of `mde` in a DNA rate of `baseline`."""
    effect = proportion_effectsize(baseline, baseline - mde)
    n = NormalIndPower().solve_power(effect_size=effect, alpha=alpha, power=power, ratio=1.0)
    return math.ceil(n * design_effect)

If appointments were independent, that’s 4,285 per arm. They aren’t: patients averaged 2.42 appointments in 16 weeks, and someone who misses one is more likely to miss the next. The pre-trial design effect (the variance of the DNA rate clustered by patient, divided by the binomial variance) was 1.16, so the plan needed 4,956 appointments per arm, about 8,194 patients in all.

Data table

Appointments per arm	Fall of 1.5 points	Fall of 2 points	Fall of 3 points
250	3%	5%	10%
500	5%	9%	21%
750	7%	13%	33%
1000	9%	17%	44%
1250	12%	22%	55%
1500	14%	27%	64%
1750	17%	32%	72%
2000	19%	37%	78%
2250	22%	42%	83%
2500	24%	46%	88%
2750	27%	51%	91%
3000	30%	55%	93%
3250	32%	59%	95%
3500	35%	63%	96%
3750	38%	66%	97%
4000	40%	70%	98%
4250	43%	73%	99%
4500	46%	76%	99%
4750	48%	78%	99%
5000	50%	80%	100%
5250	53%	82%	100%
5500	55%	84%	100%
5750	57%	86%	100%
6000	60%	88%	100%
6250	62%	89%	100%
6500	64%	90%	100%
6750	66%	92%	100%
7000	68%	93%	100%
7250	69%	94%	100%
7500	71%	94%	100%
7750	73%	95%	100%
8000	74%	96%	100%
8250	76%	96%	100%
8500	77%	97%	100%
8750	78%	97%	100%
9000	80%	98%	100%
9250	81%	98%	100%
9500	82%	98%	100%
9750	83%	98%	100%
10000	84%	99%	100%
10250	85%	99%	100%
10500	86%	99%	100%
10750	87%	99%	100%
11000	88%	99%	100%
11250	89%	99%	100%
11500	90%	99%	100%
11750	90%	100%	100%
12000	91%	100%	100%

Synthetic data. Source: projects/dna-reminder-experiment. Two-sided test at 0.0167 (5% split across three comparisons), allowing for repeat appointments (design effect 1.16).

The planted one-text effect, 1.6 points, is below that 2-point target. Re-running the trial 2,000 times on the same patients, with fresh randomisation and fresh outcomes, it passed the 0.0167 threshold 63% of the time, against 89% for two texts and 94% for the cost message. An inconclusive result for one text was always a likely outcome.

Did randomisation work?

8,194 patients, 2,048 or 2,049 per arm, with 19,790 appointments (4,917 to 5,005 per arm).

Every standardised difference from the control arm is below 0.1; the largest is 0.06, for the share aged 65+ in the cost-message arm. I don’t test these differences for significance. Under randomisation, any imbalance is chance by definition; the useful question is whether it’s big enough to matter, and the standardised difference answers that. One gap does matter later: the cost-message arm had more patients with a prior DNA.

Results

Data table

Arm	DNA rate	DNA rate (low)	DNA rate (high)
No reminder	9.3%	8.4%	10.1%
One text	8.3%	7.4%	9.1%
Two texts	7.2%	6.4%	7.9%
Cost message	7.5%	6.7%	8.3%

Synthetic data. Source: projects/dna-reminder-experiment. Intention to treat: every randomised appointment counts.

Changes are in percentage points, intention to treat, clustered by patient. Two texts and the cost message pass the pre-specified threshold. One text doesn’t.

The trial can’t rank the two stronger arms. The cost message minus two texts is +0.3 points (−0.8 to +1.4), and the planted difference is only 0.2 points the other way. Two texts minus one text is −1.1 points (−2.2 to +0.0, p = 0.059), against a planted −0.5.

Why the intervals are clustered

Treating the 19,790 appointments as independent gives intervals 6% to 7% narrower. In the 2,000 re-runs, the estimates’ real standard deviation was about 0.61 points; the naive standard errors averaged 0.56 to 0.57, the clustered ones 0.60 to 0.61. So the naive “95%” intervals contained the planted effect only 92% to 93% of the time, the clustered ones 94% to 95%.

What adjustment did

Adjusting for the baseline covariates narrowed the intervals by only about 2%: with a binary outcome and modest predictors, there’s little variance to remove. It did move the cost-message estimate from −1.8 to −2.0 points, because that arm happened to have more patients with a prior DNA (18.0% against 16.7%). Correcting chance imbalance in factors named before the trial is exactly what adjustment is for.

The logistic regression gives odds ratios, which managers can’t use. So the code predicts every appointment under each arm, averages, and subtracts, giving a risk difference; the interval comes from the delta method with the patient-clustered covariance:

def mean_and_gradient(arm: str) -> tuple[float, np.ndarray]:
    scenario = x.copy()
    scenario[:, columns] = 0.0
    if arm != CONTROL:
        scenario[:, fit.model.exog_names.index(ARM_TERMS[arm])] = 1.0
    p = 1.0 / (1.0 + np.exp(-scenario @ beta))
    return float(p.mean()), (p * (1 - p)) @ scenario / len(p)

Intention to treat, per-protocol, and the patients no text reaches

In the text arms, the three-day text reached 82.4% of appointments. The rest had no mobile number on record (10.6%), were booked too late (4.5%), or the text failed (2.5%).

Data table

One text

Analysis	This trial	This trial (low)	This trial (high)	Planted value
Appointments independent	-1.0	-2.1	0.1	-1.6
Clustered by patient	-1.0	-2.2	0.2	-1.6
Adjusted (logistic)	-1.1	-2.3	0.1	-1.6
Mobile-number holders	-1.2	-2.5	0.0	-1.8
Per-protocol	-1.3	-2.5	0.0	-2.0
Receiving a text (IV)	-1.2	-2.7	0.2	-2.0

Two texts

Analysis	This trial	This trial (low)	This trial (high)	Planted value
Appointments independent	-2.1	-3.2	-1.0	-2.1
Clustered by patient	-2.1	-3.3	-0.9	-2.1
Adjusted (logistic)	-2.2	-3.3	-1.0	-2.1
Mobile-number holders	-2.5	-3.7	-1.3	-2.4
Per-protocol	-2.6	-3.8	-1.5	-2.4
Receiving a text (IV)	-2.3	-3.6	-1.1	-2.4

Cost message

Analysis	This trial	This trial (low)	This trial (high)	Planted value
Appointments independent	-1.8	-2.9	-0.7	-2.3
Clustered by patient	-1.8	-2.9	-0.6	-2.3
Adjusted (logistic)	-2.0	-3.1	-0.8	-2.3
Mobile-number holders	-2.0	-3.2	-0.7	-2.6
Per-protocol	-2.2	-3.4	-1.0	-2.9
Receiving a text (IV)	-2.2	-3.6	-0.7	-2.9

Synthetic data. Source: projects/dna-reminder-experiment. Each analysis answers a slightly different question, so each has its own planted value.

ITT answers the service’s question: what happens to the DNA rate if we switch this on for everyone? Per-protocol drops the treated appointments no text reached, but keeps every control appointment, so it compares a selected group with an unselected one. For one text it gives −1.3 points with p = 0.045: a “significant” result that the randomised comparison doesn’t support.

There are two honest ways to ask what a text does for someone it reaches:

• Restrict every arm to patients with a mobile number. That’s recorded before randomisation, so the comparison stays randomised: −1.2, −2.5 and −2.0 points, against planted values of −1.8, −2.4 and −2.6.
• Divide the ITT effect by the share of appointments a text reached (the Wald instrumental-variable estimator): −1.2, −2.3 and −2.2 points, against planted effects among those reached of −2.0, −2.4 and −2.9.

In this synthetic world, per-protocol’s bias turned out small: across the re-runs it averaged −2.56 points for two texts against a true −2.42, and was within 0.01 points for the other arms. That’s because the patients texts miss have much the same baseline risk as those they reach (pre-trial DNA rate 10.4% without a mobile number, 10.0% with one): the older patients and short-notice bookings among them offset the extra risk. In a service where the unreachable patients are the likeliest to miss, the bias would be bigger, and nothing inside the per-protocol analysis would show it.

The no-mobile patients are also an equity question. They’re older and more often from deprived areas, and a text-only policy gives them nothing: their estimates in the subgroup chart scatter around zero, as planted. Chasing missing numbers, or a letter or call for those without one, belongs in the policy.

Subgroups, with caution

Data table

One text

Subgroup	Estimate	Estimate (low)	Estimate (high)	Planted effect
All patients	-1.0	-2.2	0.2	-1.6
Aged 18-29	-2.9	-6.3	0.6	-3.0
Aged 30-44	-1.9	-4.4	0.6	-2.2
Aged 45-64	-1.2	-3.3	0.9	-1.5
Aged 65+	0.5	-1.4	2.5	-0.5
IMD 1 (most deprived)	-0.6	-3.4	2.3	-2.1
IMD 2	-0.6	-3.1	1.9	-1.8
IMD 3	-3.1	-5.7	-0.5	-1.5
IMD 4	0.0	-2.7	2.6	-1.3
IMD 5 (least deprived)	-1.1	-3.5	1.4	-1.2
No mobile number	0.8	-3.3	5.0	0.0
Has a mobile number	-1.2	-2.5	0.0	-1.8

Two texts

Subgroup	Estimate	Estimate (low)	Estimate (high)	Planted effect
All patients	-2.1	-3.3	-0.9	-2.1
Aged 18-29	-3.4	-6.8	0.0	-4.0
Aged 30-44	-3.0	-5.4	-0.5	-2.9
Aged 45-64	-3.3	-5.2	-1.4	-1.9
Aged 65+	0.2	-1.7	2.2	-0.7
IMD 1 (most deprived)	-2.7	-5.3	0.0	-2.8
IMD 2	-1.3	-3.7	1.1	-2.4
IMD 3	-3.8	-6.3	-1.3	-2.0
IMD 4	-1.3	-4.0	1.3	-1.8
IMD 5 (least deprived)	-1.2	-3.8	1.4	-1.6
No mobile number	1.5	-2.4	5.5	0.0
Has a mobile number	-2.5	-3.7	-1.3	-2.4

Cost message

Subgroup	Estimate	Estimate (low)	Estimate (high)	Planted effect
All patients	-1.8	-2.9	-0.6	-2.3
Aged 18-29	-3.2	-6.5	0.2	-4.4
Aged 30-44	-2.9	-5.4	-0.3	-3.1
Aged 45-64	-2.0	-4.0	0.0	-2.1
Aged 65+	-0.2	-2.1	1.7	-0.8
IMD 1 (most deprived)	-2.7	-5.4	0.0	-3.0
IMD 2	-0.4	-3.0	2.2	-2.6
IMD 3	-3.0	-5.6	-0.5	-2.1
IMD 4	-0.4	-3.1	2.2	-1.9
IMD 5 (least deprived)	-2.1	-4.4	0.2	-1.7
No mobile number	-0.1	-3.9	3.6	0.0
Has a mobile number	-2.0	-3.2	-0.7	-2.6

Synthetic data. Source: projects/dna-reminder-experiment. Below zero means fewer missed appointments. With 12 rows per arm, expect about one interval in 20 to miss its planted value by chance alone.

The planted effects are larger for younger patients and in deprived areas. The age estimates have the right shape, from −2.9, −3.4 and −3.2 points at 18–29 to +0.5, +0.2 and −0.2 at 65+. But the joint test that the effects differ across age bands gives p = 0.36; across IMD quintiles, p = 0.86; with and without a mobile number, p = 0.30. An interaction test compares estimates that each rest on part of the data, so it needs a much bigger trial than the average effect does.

The deprivation pattern shows why subgroups need care. IMD quintile 3 had the largest estimated effects in every arm (−3.1, −3.8 and −3.0 points), though the planted gradient runs from quintile 1 to 5. None of the 36 subgroup intervals missed its planted value: the estimates are noisy, not wrong. Reporting “reminders work best in IMD quintile 3” would be reporting noise. I’d show every pre-specified subgroup, lead with the interaction test, and treat any pattern as a hypothesis for the next trial.

Why the early weeks would have misled

Data table

One text

Trial week	Estimate so far	Estimate so far (low)	Estimate so far (high)
1	1.2	-3.0	5.4
2	1.3	-1.6	4.3
3	0.4	-2.0	2.9
4	-0.4	-2.5	1.8
5	-1.3	-3.3	0.7
6	-1.8	-3.7	0.0
7	-1.6	-3.3	0.0
8	-1.7	-3.4	-0.1
9	-1.2	-2.7	0.4
10	-1.1	-2.6	0.4
11	-0.7	-2.1	0.7
12	-0.9	-2.2	0.5
13	-0.7	-2.1	0.6
14	-0.6	-1.9	0.7
15	-0.9	-2.1	0.4
16	-1.0	-2.2	0.2

Two texts

Trial week	Estimate so far	Estimate so far (low)	Estimate so far (high)
1	-0.8	-4.7	3.0
2	0.5	-2.4	3.3
3	-0.5	-2.9	1.8
4	-0.9	-3.0	1.2
5	-2.4	-4.3	-0.5
6	-2.4	-4.1	-0.6
7	-2.4	-4.0	-0.8
8	-2.9	-4.5	-1.3
9	-2.8	-4.3	-1.3
10	-2.8	-4.2	-1.4
11	-2.2	-3.6	-0.9
12	-2.1	-3.4	-0.8
13	-2.0	-3.2	-0.7
14	-1.8	-3.0	-0.6
15	-2.0	-3.1	-0.8
16	-2.1	-3.3	-0.9

Cost message

Trial week	Estimate so far	Estimate so far (low)	Estimate so far (high)
1	0.5	-3.6	4.5
2	0.7	-2.2	3.6
3	0.0	-2.5	2.4
4	-0.7	-2.8	1.4
5	-2.0	-3.9	0.0
6	-2.3	-4.0	-0.5
7	-1.9	-3.5	-0.2
8	-2.7	-4.3	-1.1
9	-2.5	-4.0	-1.0
10	-2.5	-4.0	-1.1
11	-2.1	-3.4	-0.7
12	-1.9	-3.3	-0.6
13	-2.0	-3.2	-0.7
14	-1.6	-2.8	-0.4
15	-1.7	-2.9	-0.5
16	-1.8	-2.9	-0.6

Synthetic data. Source: projects/dna-reminder-experiment. Intervals clustered by patient, not adjusted for repeated looks.

After weeks 1 and 2, the one-text arm had a higher DNA rate than the control (+1.2 and +1.3 points). By week 8, two texts and the cost message showed falls of 2.9 and 2.7 points, bigger than the final estimates and the planted effects. One text crossed p < 0.05 in weeks 6 and 8, then finished at p = 0.096. A manager checking every week at 5% would have declared it a success in week 6.

The fixed horizon is what lets the final p-values mean what they say. Peeking and sequential testing simulates how fast repeated looks inflate false positives, and covers designs that allow planned interim looks.

Is it worth it?

The assumptions are constants in trial_results.py, not findings: 3p per text segment, two segments for the longer cost message, and £50 of clinic time put to use for each DNA avoided. Set-up and admin costs are left out.

Fewer texts than appointments go out because patients with no mobile number and short-notice bookings don’t get the three-day text. The second text costs £271 more per 10,000 appointments and avoids an estimated 108 more DNAs (−4 to 221).

At the point estimates, each arm pays for itself once an avoided DNA is worth more than £2.50 to £2.86. So the decision turns on what this sketch leaves out: set-up, opt-outs, how patients feel about a message that mentions cost, and whether freed slots are actually rebooked.

What I’d recommend

On this evidence, two texts as the default: it has the clearest effect, and the second text is close to free. The cost message did about as well with one text, and the trial can’t separate the two. The next trial should test the combination directly.

Limits

• Planted, not observed. The effects, their gradients and who lacks a mobile number are my assumptions.
• No rebooking. A DNA usually generates another appointment. I didn’t model that, or cancellations and opt-outs.
• Per-appointment outcome. “Did this patient miss any appointment?” is a different question, with a different answer.
• Sixteen weeks, one population, with no seasonality beyond week-to-week swings.
• A cost-benefit sketch. The assumed £50 carries the whole net-value row.

What I’d do next

• A 2 × 2 factorial trial: one or two texts, standard or cost wording. It answers both remaining questions with the same patients.
• Blocked randomisation within age bands, to guarantee balance where the effect is expected to vary most.
• A group sequential design with O’Brien-Fleming boundaries, if the service wants planned interim looks.
• Model rebooking, and report a patient-level outcome alongside the appointment-level one.

Run it yourself

uv run python projects/dna-reminder-experiment/run.py

It sizes, simulates, analyses and re-runs the trial, rewrites every chart and download on this page, and prints the tables quoted here, in 20 to 50 seconds. A rerun gives byte-identical files. The full code is in the download below, with a README that explains each file.