I’ve been looking into the National Key Performance Indicators (KPIs) for the Improving Access to Psychological Therapies (IAPT) programme for 2011–12, comparing IAPT services throughout England on several measures. In this article I focus on the results at the level of primary care trusts (PCTs), and on Gloucestershire’s mental health trust, 2gether.
2gether provides three IAPT services, codenamed “Let’s Talk”, for South Gloucestershire, Herefordshire and Gloucestershire. They are separate in that they are the result of three separate contracts with three separate PCTs. In order to make comparisons I had to work with the data for all the IAPT services in England, and I’ve made all my data available for anyone whose interest may lie in different parts of the country.
The original data is published quarterly by the Health and Social Care Information Centre (HSCIC) under the heading NHS specialist mental health services. I’ve consolidated data from the four quarters and computed some totals, trends and rankings to give what I hope is a clearer picture of what’s going on.
For a general overview of the KPIs and some discussion of what they mean, see my previous article in this series, Performance. I think it is important to read that article first, because some of the figures you’ll see here probably don’t mean what you might at first imagine them to mean.
You can see my data for all PCTs in England in the document IAPT KPI trends (PDF). If you want to work with my data yourself, the same figures are also in the spreadsheet IAPT KPI trends (ODS format, using macros for formatting). My analysis is not authoritative because I have not checked everything very thoroughly. If you plan to draw important conclusions, then I suggest checking my figures independently. Please let me know if you find mistakes.
There are some major limitations in the data and the analysis. One is that the percentages do not really compare like with like, so that services that are in the midst of change can have wildly inaccurate indicators. This is a fault in the original data as collected, which does not track patients quarter by quarter. Another is that there are no correlations, so that there is nothing to show how the indicators are linked. This is a limitation in the analysis, and someone could extend my analysis to overcome it.
Population and risk
On the Population pages in my results I use population estimates from 2005 onwards to extrapolate estimates for 2011. I didn’t feel this was very interesting, so I didn’t make any use of the results.
On the Risk pages I compare the prevalence of depression and anxiety (KPI1) with the 2005 population to reverse-engineer the weightings applied in the original research. I ranked the PCTs by their relative percentage weighting.
For example Manchester (+63%, 1st) has the highest risk of depression and anxiety, with Liverpool (60%, 2nd) only just behind, while Herefordshire (-32%, 150th) has the lowest risk.
Risk of depression and anxiety
On the Prevalence pages in my results I estimate the number of people who are treatable for depression and anxiety as 20% of KPI1 for each PCT, and rank the PCTs. I chose 20% because that’s the percentage given in the original research that KPI1 comes from, but of course the ranking for KPI1 itself is the same. These figures indicate the required relative size of each IAPT service to provide an equal level of service throughout England.
Hertfordshire (1st) would need the largest IAPT service, and Bassetlaw (151st) would need the smallest.
People with treatable anxiety and depression
On the Referrals pages in my results I use the results for the four quarters to estimate the growth of each IAPT service as a percentage per annum, and I rank the growth rates. On this basis, in around one in six IAPTs there is falling demand (but my estimate is not a very reliable measure).
In Hounslow (1st) demand appears to be growing fastest, having started the year with nothing, while in Ashton, Leigh and Wigan (150th) demand appears to be shrinking fastest.
Growth rate in referrals per annum
On the Waiting pages in my results I give the total numbers of people waiting more than 28 days for treatment during the year 2011-12, then express that number as a percentage of people starting treatment (KPI4) and rank the percentages.
A percentage greater than 100% is possible. For example, in Surrey (149th) nearly 18,000 people waited more than 28 days, but only about 4,500 started treatment, while in Tower Hamlets (1st) only about 30 people were kept waiting that long from about 3,500 starting treatment.
Referred and waiting more than 28 days
On the Access pages in my results I give the total numbers of people accessing therapy in the year 2011-12, and I compute the trend over the four quarters as a percentage change per annum. This tells you how fast the service is growing.
I also express the number of people treated as a percentage of my own estimates of the number of treatable people (which I described above), and I rank this percentage. This indicates how well the service is meeting the needs of the population.
For example, in Northumberland (1st) I estimated there were 6,685 treatable people, but they actually treated 7,259, or 108%, while in Enfield (150th), where I estimated there were 7,052 treatable people, they only managed to treat 428, or 6%. If you were to use IAPT’s national targets instead of my estimates of the number of treatable people, you’d get higher percentages but exactly the same ranking.
The fact that some IAPT services are treating more people than my estimates suggests that my estimates are on the low side. That, in turn, suggests that these percentages should be lower — in other words, that IAPT services are not doing as well as these numbers make it appear.
Access to psychological therapy
On the Dropout pages in my results I compute the drop-out rate simply by comparing the numbers referred and treated, and I rank the PCTs by the percentage who dropped out. Some of these people attended one appointment, some were offered an appointment but didn’t attend, and some are still waiting.
For example, in Swindon (1st) less than a quarter of the nearly 6,000 referrals dropped out, while in Luton (150th) nearly 2,500 people were referred and according to the reported figures no one at all completed treatment.
On the Treated pages in my results I give the total numbers of people completing treatment during the year 2011-12, and I compute the trend over the four quarters as a percentage change per annum. This tells you how much the service is growing. On this basis, about a third of IAPT services are shrinking (but my estimate is not a very reliable measure).
I also compare the number treated with my estimate of the number of treatable people (which I described above), and I rank this percentage. This indicates how well the service is delivering treatment.
For example, in Swindon (1st) around 4,500 people were treated out of just over 5,000 treatable, while in Hillingdon (149th) only 14 people were treated out of nearly 6,000 treatable. (Hillingdon reported no people completing treatment at all in the first three quarters.)
On the Well pages in my results I give the total numbers of people who were well (“not at caseness”) when they started treatment, and I compute the trend over the four quarters as a percentage change per annum. This tells you to what extent the service is being used to treat people who don’t need treatment.
I also compare the number of people who were well with the total number who accessed treatment, and I rank the PCTs by this percentage. For example, in West Essex (1st) only one person (sometime in the second quarter) was well out of nearly 2,000 who started treatment, but in South Staffordshire (149th) one in every six of the 5,000 who started treatment were well.
On the Recovery pages of my results I give the numbers “moving to recovery” (adjusted for those who were well to start with). I compare them with my own estimate of the number of treatable people, and I rank the PCTs by that percentage. This indicates how effective the service as a whole is in relation to the clinical need.
For example, in Rotherham (1st) nearly a third of the treatable people were recorded as moving to recovery during 2011-12, but in Hillingdon (149th) only four people out of nearly 6,000 treatable people were recorded as moving to recovery.
Uptake is my nickname for the number who access the service (by starting treatment) as a percentage of the number of referrals. On the Uptake pages of my results I rank the PCTs on this indicator, and compute the trend over the four quarters.
Calderdale (1st) had the highest uptake, with only ten people not starting treatment out of nearly 4,000 referred, while Telford and Wrekin (150th) had the lowest, with less than a third starting treatment of the more than 3,000 people referred.
Reach is my nickname for the number who access the service (by starting treatment) as a percentage of the prevalence of depression and anxiety in the population (KPI1). On the Reach pages of my results I rank the PCTs on this indicator, and compute the trend over the four quarters.
Northumberland, Walsall and Swindon were pretty much joint 1st in this indicator, all reaching nearly a quarter of the estimated prevalence, while Enfield and Barnet were pretty much joint last, reaching only around one in eighty of the estimated prevalence.
On the Recovery rate pages in my results I give the percentage of people treated who were unwell at the start and moving to recovery at the end of treatment, with trends and ranks as before. This is a measure of clinical effectiveness because it excludes anyone who dropped out, who was left waiting, or who was treated despite being well.
For example, in West Essex (1st) almost everyone treated was moving to recovery at the end of treatment. Only a couple of dozen of the thousand or so people treated weren’t. But in Central Lancashire (149th) only about one in fifteen were moving to recovery at the end of treatment.
I subsequently added an indicator of IAPT’s perceived success rate, which I describe in detail in Perceived IAPT performance. In a future article, I’ll discuss some of the wider issues and controversies around IAPT.