Nudging no-shows in hospital appointments: Phase 2 trial in health fails to replicate well-known behavioural insight
By Pelle Guldborg Hansen, Caroline Gundersen,
Raoni Demnitz,, Sidsel Bruun
, and Jesper Enøe Elbæk
In a quasi-experimental Phase 2 field replication of integrating descriptive social norms and institutional cost in SMS reminders to reduce missed hospital appointments, we found no effect on no-shows. The result is now published in Behavioural Public Policy, with a pointer to our new framework for how such replications should be understood.
_________
Missed medical appointments are a costly and persistent problem in healthcare [1-2]. When patients do not attend and do not cancel in advance, time slots go unused, staff time is wasted, and other patients may face longer waits [3-5]. In the UK alone, the National Health Service (NHS) estimates the annual cost of missed general practitioner appointments to be over £216 million [6], while in the United States, missed visits are estimated to cost the health system US$150 billion annually (£112 billion) [7]. Similar patterns appear in low- and middle-income countries; for example, specialty outpatient clinics in Santa Maria, Brazil, reported an average DNA rate of 9.4% from 2016 to 2021, resulting in substantial costs for municipal health budgets [8].
Because of the size of these numbers, even small improvements in attendance can obviously matter. To this end, many behavioural scientists and behavioural insights units around the world have experimented with how to reduce missed medical appointments – commonly referred to as ‘no-shows’ or ‘Do Not Attend’ (DNAs). SMS-reminders are already widely used to reduce no-shows. They are cheap, scalable, and easy to implement – and the effect of reminders on DNAs is robust [9]. Thus, although reminders should not matter in principle – people ought to remember their appointments and have no reason to forget – in practice they serve as a nudge to reduce DNAs.
Reminders and messages to reduce no-show
But what about the content of a message? Can small changes in the way messages are formulated in SMS-reminders serve as a nudge to further decrease DNAs? That is the fundamental claim of a whole cluster of experimental behavioural insights papers out there – and we believe that fundamental claim is true.
But what messages work?
A well-known Behavioural Insights experiment by Hallsworth et al. ventured into testing exactly this with a surprising finding [10]. The study compared a standard reminder with messages that either included a descriptive social norm or highlighted the institutional cost of a missed appointment to the health system. The question was: what would work?
The right answer at any university exam would be: “the descriptive social norm… hands down!”
Yet, Hallsworth et al found something else. Surprisingly, writing the institutional cost of a missed appointment reduced no-shows compared with the generic reminder and beat the descriptive norms as well.
So, empiricism seems to eat even well-trained intuition for lunch.
And then we might need to think again… and again
Because Behavioural Insights currently suffers from a series of poorly dealt-with crises, we need to start thinking twice… at least.
First came the ever-ongoing crisis that few behavioural insights units actually seem to know what Behavioural Insights is about [11]. Then came the replication crisis, tainting the psychological roots of the paradigm [12]. And finally came the crisis of the meta-studies – reviews that pooled nudges together by their surface label rather than by the mechanism each was built to exploit, and so ended up evaluating ‘pills by their colour’ [13-15].
We believe that while the first crisis should be cured by reading proper academic books, the second holds the remedy for the third.
That is: we need to replicate.
Take your Organisation
In-person training, COPENHAGEN 2026
Our replication
Thus, in a quasi-experimental, time-blocked field trial across similar, if not identical, instances of the same underlying behavioural problem as addressed by Hallsworth et al, we tried to replicate their findings at a major Danish hospital. The experiment included 20,867 outpatient appointments across three departments (Cardiology, Endocrinology and Pulmonology).
Patients received one of three reminder messages (see Figure 1 below):
- A standard reminder,
- A reminder including the cost of a missed appointment to the healthcare system,
- A reminder including a descriptive social norm, stating that more than nine in ten patients turn up or cancel in good time.
Figur 1. The figure shows the different nudges patients received via SMS (in Danish).
We then looked at two outcomes:
- whether outpatients did not attend without cancelling;
- whether outpatients cancelled in advance if they did not attend.
This allowed us to test not only whether the messages reduced DNAs, but also whether they changed cancellation behaviour if they did not show.
What we found
So, what happened?
Not much. And that is the finding.
Neither message moved the no-show rate. A patient who received the social-norm reminder was no more likely to turn up than one who received the plain reminder, and the cost message fared no better. Across nearly twenty-one thousand appointments, the three groups sat level with one another.
We did not replicate. The words in the reminder did not reduce no-shows. One thing did move, though not the thing we set out to find: the cost message noticeably raised the share of patients who cancelled in advance, rather than simply not appearing. It did not get more people through the door. But it got more of them to say so in time – and a slot handed back in time is a slot another patient can take.
Figure 2. Missed appointments by reminder. The three bars sit level with one another. Neither extra message moved the no-show rate. Error bars show 95% confidence intervals.
Figure 3. Advance cancellations by reminder. The cost message lifted the share of patients who cancelled ahead of time. Error bars show 95% confidence intervals.
Across the three departments the picture varied, with no consistent pattern.
Why a null is a result
So, was this a failure?
It is tempting to say so: file the null under defeat and move on. We think that is exactly the wrong lesson.
Behavioural Insights, taken seriously, is an applied science: Behavioural Insights as Applied Science, or BIAS. And like medicine, or engineering, it does not settle a question in a single, striking study. A finding earns its confidence across phases, each asking its own question. The paper sets out three [16].
The first phase asks whether an effect can happen at all: a proof of concept, under favourable conditions. Hallsworth et al. gave us exactly that in 2015 – a cost-framed reminder, a real fall in no-shows. A good Phase 1 result. It shows the effect is possible. It does not show it is durable.
The second phase asks whether the effect holds – whether it returns when the same behavioural problem is met again, in the ordinary conditions of a working hospital. That is what we did here: not a fresh discovery, a replication, whose task was never to dazzle but to find out whether the first win travels.
It did not.
The third phase asks how far a finding reaches – for whom, and under what conditions – before it is scaled into policy. But that phase is earned, not assumed: you reach it only once the second has held. We are not there.
Now, two readings tempt us here, and each has already hardened into a camp. The persuasionists will say we simply need a better message: the win is real, it only wants sharper or wider deployment. The empiricists will say the opposite, that behaviour is all context, so the task is to understand each setting and tailor a message to it, one case at a time. The second is the more seductive, and by far the more corrosive: taken to its end, it dissolves the discipline altogether. If every effect is only ever local, there is nothing to generalise – and behavioural science becomes a scientific-sounding name for ordinary consultancy.
We refuse both.
Our wager is that beneath a missed appointment in this hospital and a missed appointment in that one lies a shared behavioural structure, and that the work is to find it. So a null does not send us back for a cleverer slogan, nor out to tailor forever. It sends us back to the diagnosis: did we read the behaviour correctly, and did the message truly engage the mechanism we assumed – or only seem to?
That is: you do not scale a Phase 1 win.
A striking first result is where the science begins, not where it ends. Read this way, the null is not the collapse of the story. It is the method, BIAS, working exactly as designed: catching, at the cheapest possible stage, an effect that would never have survived being scaled.
Where this leaves us
So what does a null like this leave a practitioner with?
Not a better slogan, and not a shrug – a discipline.
A behavioural insight that worked somewhere is a hypothesis, not a solution. Before it is rolled out, it should be run again where you actually stand: your patients, your clinic, your queue. On this evidence, the honest instruction is the unglamorous one – hold, don’t scale.
(If your aim happens to be a freed slot rather than a filled one, the cost message may still earn its keep – but that is a different goal, honestly named, not the effect we went looking for.)
And this is not a caveat on the science. It is the science. A striking first result is a beginning; whether it hardens into knowledge depends on what happens when someone runs it again. So the null we report is not the close of the story – it is an opening onto the next replication, ours or yours.
The full study (the framework, the phases, and the trial in full) is open access [16]. And if this is the manner of thinking you would like more of, it is what we send the people who follow the centre’s work: what we judge worth the reading, and little else. Sign up for the Insights Community here.
References
[1] Dantas, L. F., Fleck, J. L., Oliveira, F. L. C. & Hamacher, S. (2018). No shows in appointment scheduling – a systematic literature review. Health Policy, 122(4), 412–421. https://doi.org/10.1016/j.healthpol.2018.02.002
[2] Nørgård, B. M., Iachina, M., Ammentorp, J., Schwalbe, D. M., Waidtløw, K. Y., Richardt, L. & Sodemann, M. (2025). Non-attendance in hospital appointments based on data from the entire region of Southern Denmark. Clinical Epidemiology, 17, 303–314. https://doi.org/10.2147/CLEP.S512971
[3] Groden, P., Capellini, A., Levine, E., Wajnberg, A., Duenas, M., Sow, S., Ortega, B., Medder, N. & Kishore, S. (2021). The success of behavioral economics in improving patient retention within an intensive primary care practice. BMC Family Practice, 22(1), 253. https://doi.org/10.1186/s12875-021-01593-8
[4] Boone, C. E., Celhay, P., Gertler, P., Gracner, T. & Rodriguez, J. (2022). How scheduling systems with automated appointment reminders improve health clinic efficiency. Journal of Health Economics, 82, 102598. https://doi.org/10.1016/j.jhealeco.2022.102598
[5] Toker, K., Ataš, K., Mayadağlı, A., Görmezoğlu, Z., Tuncay, I. & Kazancioglu, R. (2024). A solution to reduce the impact of patients’ no-show behavior on hospital operating costs. Healthcare, 12(21), 2161. https://doi.org/10.3390/healthcare12212161
[6] Werner, K., Alsuhaibani, S. A., Alsukait, R. F., Alshehri, R., Herbst, C. H., Alhajji, M. & Lin, T. K. (2023). Behavioural economic interventions to reduce healthcare appointment non-attendance: a systematic review and meta-analysis. BMC Health Services Research, 23(1), 1136. https://doi.org/10.1186/s12913-023-10059-9
[7] Mooney, J. (2024, August 29). The cost of missed medical appointments: a hidden burden on healthcare. TransLoc Blog. https://transloc.com/blog/the-cost-of-missed-medical-appointments-a-hidden-burden-on-healthcare/
[8] Jacobi, E. R. T., Jacobi, L. F., Souza, A. M., Dorneles, T. D. C. & Coronel, D. A. (2023). Valores financeiros que deixaram de ser repassados ao município de Santa Maria–RS. 10º Congresso Internacional em Saúde.
[9] Gurol-Urganci, I., de Jongh, T., Vodopivec-Jamsek, V., Atun, R. & Car, J. (2013). Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database of Systematic Reviews, 12, CD007458. https://doi.org/10.1002/14651858.CD007458.pub3
[10] Hallsworth, M., Berry, D., Sanders, M., Sallis, A., King, D., Vlaev, I. & Darzi, A. (2015). Stating appointment costs in SMS reminders reduces missed hospital appointments. PLOS ONE, 10(9), e0137306. https://doi.org/10.1371/journal.pone.0137306
[11] Osman, M., McLachlan, S., Fenton, N., Neil, M., Löfstedt, R. & Meder, B. (2020). Learning from behavioural changes that fail. Trends in Cognitive Sciences, 24(12), 969–980. https://doi.org/10.1016/j.tics.2020.09.009
[12] Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
[13] Mertens, S., Herberz, M., Hahnel, U. J. J. & Brosch, T. (2022). The effectiveness of nudging: a meta-analysis of choice architecture interventions across behavioral domains. PNAS, 119(1), e2107346118. https://doi.org/10.1073/pnas.2107346118
[14] Maier, M., Bartoš, F., Stanley, T. D., Shanks, D. R., Harris, A. J. L. & Wagenmakers, E.-J. (2022). No evidence for nudging after adjusting for publication bias. PNAS, 119(31), e2200300119. https://doi.org/10.1073/pnas.2200300119
[15] Szaszi, B., Higney, A., Charlton, A., Gelman, A., Ziano, I., Aczel, B. & Tipton, E. (2022). No reason to expect large and consistent effects of nudge interventions. PNAS, 119(31), e2200732119. https://doi.org/10.1073/pnas.2200732119
[16] Guldborg Hansen, P., Demnitz, R., Elbæk, J. E., Bruun, S. & Gundersen, C. D. (2026). Replicating behavioural insights in health: a quasi-experimental Phase 2 trial of integrating descriptive social norms and institutional cost in SMS reminders to reduce missed hospital appointments. Behavioural Public Policy, 1–25. https://doi.org/10.1017/bpp.2026.10030
WISH TO STAY IN THE LOOP?
Community
