Performance

Category: 2getherNational Key Performance Indicators (KPIs) for the Improving Access to Psychological Therapies (IAPT) programme in England have just been published for January to March, the final quarter of the 2011–12 reporting year. The KPIs allow comparison of IAPT services on several measures, though the results are not very easy to interpret. In this series of articles I explain what I think the published KPIs mean, attempt a little interpretation, and discuss some of the issues raised.

My particular interest lies in the three IAPT services, codenamed “Let’s Talk”, provided by the 2gether NHS Foundation Trust. These three services cover Gloucestershire, South Gloucestershire and Herefordshire, and they are separate in that they are the result of three separate contracts with three separate primary care trusts (PCTs).

In order to make comparisons I had to work with the data for all the IAPT services in England, and in my next article on this subject I’ll make all my data available for anyone whose interest may lie in different parts of the country. The data is not published by IAPT itself, but by the Health and Social Care Information Centre (HSCIC).

The KPIs depend on a base measure of the population in each PCT’s area for 2005. I checked more recent population estimates, and in most PCTs any changes in population since 2005 are not very significant.

In this article I explain what the KPIs mean and outline the picture for England as a whole. In another article, I’ll look at the detail by PCT.

KPI1: Prevalence

The KPIs start with a measure of prevalence — the estimated number of people in each PCT’s geographical region who have depression and/or anxiety disorders. This number was computed in 2008 by the North East Public Health Observatory (NEPHO) using data from a survey carried out in 2000, the National Psychiatric Morbidity Survey.

I don’t know why they didn’t use the more up to date 2007 survey. Perhaps the results had not been published by the time they needed them. Overall, the 2007 survey showed that although prevalence had increased since 1993 it had changed little since 2000. (This could not have been due to IAPT, by the way, because it had barely started in 2007.) Within the overall picture, however, there were some changes in detailed prevalence which could have affected KPI1.

Results from the previous survey in 1993 had been considerably different, showing that things can change quite a lot in seven years and the rates of change are not linear. All in all, the differences between 1993 and 2000, and the differences between 2000 and 2007, suggest to me that KPI1 might no longer be a very good estimate for 2011-12.

KPI1 is adjusted for the distribution of ages and sexes in each region in 2005, separately for each of six distinct diagnoses, before being combined in an estimate of the total prevalence. The KPI Guidance claims that the figures are further adjusted for the level of deprivation in the region, but this claim is apparently false. The published KPI1 figures for 2011–12 are identical to NEPHO’s figures for 2005, which contain no adjustment for deprivation. I have asked HSCIC if they can clarify this, but at the time of writing they haven’t yet replied.

KPI1 doesn’t change from quarter to quarter, or even from year to year. The original estimate using 2005 population numbers and 2000 prevalence is always used, every year. For the whole of England it’s a little over 6 million people.

In my view KPI1 is very confusing. It would be possible to use the NEPHO data to estimate current prevalence each quarter based on current population estimates, possibly adjusted for deprivation, and certainly adjusted for the cumulative effects of IAPT. The whole point of IAPT was to reduce the numbers, but the way KPI1 remains static year by year conceals any reduction.

Furthermore, KPI1 disregards any regional differences apart from distributions of age and sex. Its assumption of otherwise uniform prevalence throughout England seems unlikely to be true. An adjustment for deprivation would go some way towards addressing this.

An additional problem with the static model of prevalence is that there is no estimate of natural turnover. Every year some people recover or leave:

  • they get better naturally,
  • have treatment,
  • move away, or
  • die

…and some other people become ill or arrive:

  • a new illness develops
  • they are already ill when they move into the region
  • they are already ill when they reach 18 years of age, entering the target population.

An IAPT service has to treat people faster than new cases appear in order to start driving down the prevalence of illness, but no one seems to have any idea how fast new cases appear.

The original NEPHO research found that in 2000 between 17% and 20% of the prevalence figure represented people who could benefit from IAPT:

These interviews provide figures for the proportions of those with disorders who have not had psychological therapies, are judged likely to benefit, and are prepared to have them.

That was a static estimate of the number of people who could benefit at that point in time, after restricted availability of effective treatments in the NHS in the years preceding IAPT had allowed numbers to build up. It was not based on turnover. For the whole of England the estimated number of treatable sufferers (20% of KPI1) is about 1¼ million people.

KPI2: Diagnosis (missing)

KPI2 is missing because it was withdrawn in 2010, “due to identified issues of quality and utility”. It had estimated how many of the people in KPI1 are diagnosed each quarter.

It would be useful to know this number because not everyone who is diagnosed is referred to IAPT. Some receive other treatment, for example medication or counselling.

At the same time, not everyone who is referred to IAPT has been diagnosed. Some people self-refer, and some people may be referred by a doctor who makes no attempt to characterize what is wrong with them.

The diagnoses included in the KPIs are:

  • Phobia
  • Depressive episode
  • Generalised anxiety disorder (GAD)
  • Mixed anxiety/depression
  • Obsessive compulsive disorder (OCD)
  • Panic disorder

It’s important to remember that, in IAPT-speak, “depression and anxiety” includes several distinct psychiatric conditions that are not necessarily all best treated in the same way.

KPI3a: Referrals

KPI3a is simply the number of people referred to the service. It doesn’t distinguish between self-referrals and referrals from GPs or other sources. It reappears in the NHS Operating Framework as SQU16_03.

For the whole of England the total for 2011-12 was nearly 900,000 people, and over the four quarters the trend indicated around 5% per annum growth (although this estimate of the growth rate is not likely to be very accurate).

KPI3b: Waiting

KPI3b is the number of people who have waited more than 28 days for treatment, and who have not dropped out for some reason.

Like other waiting time targets, the 28-day target risks becoming the norm. The reason is as follows. Suppose you manage a service where several patients are waiting for treatment, and an appointment becomes available. You can give the appointment to:

  • Patient A, who has been waiting for three days
  • Patient B, who has been waiting for 27 days
  • Patient C, who has been waiting for 35 days

If you don’t give the appointment to patient B you’ll miss the target for that patient. In terms of hitting the target, Patient A will not be worth treating until he has been waiting for 27 days. Patient C will never be worth treating. So the system tends to ensure that most patients wait 27 days, while some wait indefinitely. The KPIs don’t contain any safeguard against this happening. IAPT services could in theory have local safeguards.

Across the whole of England in 2011-12, very nearly two-thirds of the people who accessed IAPT treatment had to wait more than 28 days.

KPI4: Access

KPI4 is the number of people who have accessed psychological therapy by starting IAPT treatment. It reappears in the NHS Operating Framework as SQU16_01.

The whole point of the IAPT initiative was to increase access, to increase this number. The baseline for this measure is that the original NEPHO research before IAPT was introduced found 5% of sufferers had received psychological therapies.

Unfortunately, KPI4 doesn’t measure access to psychological therapy. It only measures access to IAPT psychological therapy. Other therapy services, the original 5%, are not included in the measure. This means that a commissioner or provider could run down its conventional non-IAPT services in order to boost KPI4, unless there are local safeguards.

Across the whole of England in 2011-12, more than half a million people accessed IAPT treatment. The number is nearly half of my estimate of the number of treatable sufferers, (which tends to suggest that my estimate is too low). The quarterly trend suggests about 2½% per annum growth in the numbers of people accessing IAPT treatment (although this estimate of the growth rate is not likely to be very accurate).

KPI5: Treatment

KPI5 is the number of people who have completed treatment. Unfortunately, attending just twice counts as completing treatment, so it is possible that some are recorded as having received treatment when their perception is that they walked out after attending twice and getting no help.

On the other hand, a patient might receive such marvellously good advice in the first session that there is no need to return, and this would not be counted as treatment. A patient-reported outcome measure could resolve this by asking “Did you complete your treatment?” but KPI5 is just based on data from the appointments system.

Across the whole of England in 2011-12, only about a third of a million people completed IAPT treatment (out of the more than half a million people who began). This is just over a quarter of my estimate of the number of treatable sufferers. I’ll return to the drop-out rate below.

KPI6a: Recovery

KPI6a is the number of people who completed treatment and whose final self-reported symptoms are below the clinical threshold. The clinical threshold is sometimes known as “caseness”.

KPI6a is often described as “moving to recovery” because these patients might not be completely free of symptoms. In particular, they might still have occasional severely disabling symptoms. The questionnaires that are used measure frequency of symptoms, not severity. For example, someone who still feels suicidally depressed once a month could be recorded as recovered.

On the other hand, a patient whose symptoms are very significantly reduced might not be recorded as recovered. This could happen if the patient still has mild but frequent symptoms.

Across the whole of England in 2011-12, nearly 130,000 people were recorded as moving to recovery after IAPT treatment (but read on to see why that’s an over-estimate).

KPI6b: Non-cases

KPI6b is a correction made to KPI6a, to take account of people whose symptoms were below the clinical threshold when they started treatment. If their symptoms were still below the clinical threshold when they finished treatment, it makes no sense to include them in the recovery rate.

Across the whole of England in 2011-12, more than 40,000 people whose symptoms were below the clinical threshold received IAPT treatment anyway. So of the 130,000 people who were recorded as moving to recovery, only around 88,000 were clinically “cases” who then moved to recovery.

Other KPIs and derived measures

KPI7 is the number of people moving off sick pay or benefits. KPI8 – 12 are measures of the IAPT workforce, and they are not published by the HSCIC. I didn’t look at any of these KPIs.

The HSCIC publishes three additional measures derived from the KPIs. They are the two NHS Outcomes Framework measures, which I have nicknamed uptake and reach, and the recovery rate. I derived yet another measure, the drop-out rate.

Uptake

The proportion of people who enter treatment after being referred is NHS Operating Framework SQU16_05. I think of it as the uptake. It measures the accuracy of referrals and people’s perception of the service, or in other words it measures the extent to which the service matches what people want of it.

For example, if GPs often refer people who don’t want psychological therapy then uptake will be low. Or if people refer themselves but they are then put off by their initial experience of the service, uptake will again be low. Uptake will be high if all the people referred feel that psychological therapy is going to be right for them and if the service looks as if it will deliver what they are hoping for.

Nationally, the average is around 60%, meaning that more than a third of people who are referred to IAPT don’t attend even once.

Reach

The proportion of people with anxiety or depression who start treatment is NHS Operating Framework SQU16_04. I think of it as measuring how well the service reaches the people it is intended for. Reach measures the efficiency of referral as well as the uptake and the capacity of the service.

For example, if GPs are good at spotting anxiety and depression and eager to refer patients to IAPT, then the reach will be high. If IAPT is well advertised in ways that people can easily understand, so that they are encouraged to seek treatment, then the reach will again be high. If the service gets a poor reputation amongst local people and GPs, or if it can’t offer treatment to all the people who are referred, then the reach will be low.

Nationally, reach is nearly 9% per annum. The original NEPHO research suggested that it could be 17% to 20%, depending on the condition, so at this rate IAPT should run out of patients in two or three years. However, the unknown turnover that I described above changes the picture. If turnover is also 9% per annum, or more, then IAPT will never run out of patients.

Another unknown is the effect of repeat treatments. If certain patients receive treatment repeatedly, every year for example, it’s not clear whether they would be counted again and again in the figures.

The eventual target for IAPT’s reach is 15% of prevalence (KPI1) per annum by the end of 2013-14. I don’t know how this figure was arrived at. Interim targets define what is called a “trajectory” towards the desired 15%. I don’t know how the trajectory was arrived at either. The interim target for 2011-12, averaged over the four-quarter trajectory, is 59% of 15%, which is 9%. IAPT therefore met its made-up target. HSCIC doesn’t publish this figure.

Recovery rate

The proportion of people treated who recover (ignoring those whose symptoms were below the clinical threshold when they started) is the recovery rate. It measures the clinical effectiveness of the service, because it only includes people who received psychological therapy.

For example, if the therapists are well trained and have high morale, then the recovery rate will be high. However, if many of the patients referred do not really have psychological disorders, then the recovery rate will be low.

Nationally, the recovery rate is only a little over 40%. For the other nearly 60% — more than half the people treated by IAPT — the treatment doesn’t work. It’s hard to know what to make of this, because the other 60% might all go on to receive non-IAPT treatment and recover. Or they might all drop out of the system and never recover. It’s impossible to tell from the IAPT figures alone.

The quarterly reported values of recovery are very unreliable because the people treated in one quarter and the people who recover in that quarter are not the same people. Services that are not operating in a steady state see wild swings in the recovery rate for this reason. For example, one PCT in the third quarter last year reported a recovery rate of 157.7% — for every five people who were treated, about eight people recovered!

Drop-out

There are no official figures for the drop-out rate, so I computed them by comparing the numbers of referrals to the numbers treated.

Across the whole of England in 2011-12, more than half a million people, nearly two-thirds of the number who were referred to IAPT (or self-referred), dropped out before completing treatment. Some of them dropped out after attending just once, and some dropped out without ever attending.


In my next article in the series I’ll compare the KPIs at PCT level, ranking all 150 IAPT services in England.

Advertisements

About Rod

Chairman of the Gloucestershire charity Suicide Crisis, Vice Chair of Relate Gloucestershire & Swindon, and an enthusiast for public involvement in the NHS.
This entry was posted in 2gether and tagged , , , , , , , , . Bookmark the permalink.

One Response to Performance

  1. Brendan Kennedy says:

    saw your post on CBT cafe. Really like this blog it identifies the difference between data and reality/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s