Missing the target

New performance assessment framework (PAF) results for 2001-2
have just been published along with refreshed star ratings and
measurements of councils’ performance for children and
adults.¹ This will be followed by comprehensive
performance assessments of all councils. Meanwhile, each of the 150
councils with social services responsibilities will be returning
their autumn 2002 position statements to the Social Services
Inspectorate.

Money and freedom to operate are linked to all this activity. In
councils reported as performing poorly, staff morale and
recruitment, and the reputation of workers and managers are all
directly affected.²

The aim of the star-rating system and the PAF’s 50 performance
indicators (PIs) is to determine whether these councils are serving
children and adults well. The Department of Health needs to know
whether the evidence they have tells them this reliably.

What the PIs fail to do is to assess how far services cover the
numbers of people in need in any council’s population. The adult
PIs do contain a number of rates per 1,000 population but these
fail to assess the numbers likely to be in need. For instance, in
England fewer than 3 per cent of adults had a completed assessment
in 2000-1 and only 3 per cent of 0-17s had a service from a social
services department in a sample week. Were those who did have an
assessment or service the most in need? Some councils may limit
access, others may provide extensive services but have less needy
populations. Also, the indicators do not assess how far provision
is meeting the specific needs of those who do receive a
service.

For example, should a council whose over-65s population is fitter,
has higher incomes and greater access to informal care and
voluntary services have as high a rate of assessments or provision
as a council that has the opposite of these factors? If lower PAF
scores and fewer stars are given to the former, it has every right
to feel hard done by.

Similarly, children’s PIs do not address how good the coverage of
social care is for children and families – there is no per 1,000
measure in the PAF set. Rather, they focus on those relatively few
looked-after children or those on the child protection register
(about 65,000 and 54,000 each year respectively out of 11 million
aged 0-17). PIs for these children include measures of outcomes
such as educational attainment and school absence, health checks,
involvement in crime and whether they are moved often or have been
reviewed or seen frequently. But no PI assesses adolescent mental
health services nor support for disabled children. Nor do PIs
measure children who need preventive work as they do not come
within the remit of looked-after or child protection
services.

In order to make a well-founded judgement of “how well these
councils serve their populations”, the SSI needs a set of measures
of populations in need and, critically, a sober evaluation of where
the money provided to each council goes in meeting that need. At
present, judgements about stars depend principally on the
measurement of what is reported as being done for those who make it
through referral and eligibility.

The second function of the star-rating system is to determine how
far councils have prospects for improvement. This is measured by
the extent to which they are moving towards specific Department of
Health targets, and by reported success in implementing a wide
range of development initiatives. Here again the indicators and
other measures used are too flimsy to bear the weight of the
ratings derived from them.

For example, take the targets related to some key indicators.

The level of intensive home care (Best Value PI No. 53; PAF C28)
measures all adults aged 65-plus with this level of service,
thereby skewing the results by ignoring people under-65 who use
intensive home care. Also, those receiving this level of service
but funded by direct payments are excluded, which conflicts with
the target that aims to increase the use of direct payments. Add in
the uncertainty about what is being reported by some councils, and
the basis for judgements of a key theme (sustaining more high-need
users at home rather than in institutions) becomes uncertain.
Moreover, the pressure to show increases in this key area may lead
to some manipulation of the numbers.

To use another example about children’s services – stars reflect
how far councils have improved their rate of adoptions from care
(Best Value PI 163, PAF C23). The numbers adopted in a year are
measured against all children looked after at the end of the year,
excluding those under section 20 agreements and those looked after
for less than six months. The results suggest that councils are
arranging adoption for about 5 per cent of them. However, given
that adoption is rare for children older than 10 and unlikely if
the child under 10 is placed at home, if these are excluded from
the calculations, councils are arranging adoption for nearly 15 per
cent. Stars for effort might be better based on this more
appropriate measure and councils that are berated for low scores
may in fact be doing a competent job.

The Department of Health would argue that assessments of “prospects
for improvement” are made on much wider evidence than simply the
PAF data. SSI twice-yearly monitoring has become increasingly
sophisticated and wide ranging. Also, among the measures now used,
councils are being asked to self-assess their progress on more than
80 plans to deliver the government agenda, choosing one of the
following responses:

• Plans outlined by the council are not on target and are
  unlikely to deliver the objectives.
• Some plans outlined are on target and are expected to deliver
  objectives.
• Most plans outlined are on target and are expected to deliver
  objectives.
• All plans outlined are on target and are expected to deliver
  objectives.

It is in local authorities’ interests to show that they are
doing well – and even perhaps to make plans for some services that
can be easily delivered. The SSI too will not want to report that
many councils are nowhere near where the government wants them to
be – also, it has no systematic means to cross-check what the
councils report.

Even if local authorities are planning and implementing initiatives
in line with the government agenda, there are questions about what
evidence there is on which to judge improved performance. As SSI
chief inspector Denise Platt noted in her annual report (in
relation to grants to promote independence for older people),
inspectors found that, “in many cases, there were no arrangements
to evaluate their impact”.³

Another issue is how authorities with overspends can move to roll
out successful schemes. There are many tensions between targets and
councils’ ability to keep within budget. For example, it is likely
that, rather than driving down unit costs, moves to restrict the
number of entrants to permanent care and to increase home support
will drive up both sets of costs. It is difficult to reduce
permanent supported admissions (without redefining placements as
temporary) if reductions must also be made in numbers with delayed
discharges from hospital.

Keeping within budget is also difficult if the health secretary
Alan Milburn commits councils to starting assessments within 48
hours of the user (or a third party) contacting them. Ensuring
cover for sickness, holidays and training, and possibly having a
larger number of staff having to work weekends will squeeze budgets
– and probably be to the detriment of planned reviews being carried
out.

Similarly, proposals to reduce time for delivery of equipment from
21 days to seven working days will have direct implications for
additional store stock capacity, hire of extra administrative and
delivery staff and an increase in cost per unit delivered – when
efficiency savings are also being required.

Directors of social services are principally concerned to deliver
their budgets on target. How far can this be squared with pressure
to deliver more, reflected in PAF PIs and in completed plans? They
will be balancing these priorities against the realities of
recruitment difficulties and supply side problems in residential,
foster and home care. And of course it is not just directors –
middle managers and front-line staff need targets to be clear and
meaningful to them, not mutually contradictory, unattainable and
possibly contrary to the best interests of individual service
users.

Also, improving councils’ coverage of services to the population
and the delivery of targets is made more difficult by the
requirement to create new organisations incorporating health,
education and social care. The experience of the new unitary
authorities and the teething troubles of primary care trusts and
the National Care Standards Commission show that it takes time to
create new organisations. So it may take a significant time before
measurable progress is made and councils run the risk of taking
their eye off the many performance targets while organisational
change is their principal focus.

Amid all the assessments there seems little opportunity to hear
from those most directly affected by a council’s performance:
service users, carers and council and provider agency staff. Nor
are there opportunities for health, education and voluntary
organisations with real experience of authorities’ performance to
make their voices heard – although the SSI does now invite senior
managers from other agencies to attend annual review
meetings.

There is no doubt that the introduction of more systematic
monitoring and measurement has focused councils on areas where
improved performance was – and is – needed. But now it is time for
a wider debate of the assessment regime. After five years of the
PAF and six of joint reviews we need to move on – for the benefit
of all those living in our local authorities.

Nick Miller is an information manager in a large social
services department.

References

¹ DoH, Social Services
Performance Assessment Framework Indicators
2001-2002,
www.doh.gov.uk/paf

² See Community
Care, 31 January and 30 May 2002

³ Denise Platt, Modern
Social Services A Commitment to Reform The 11th Annual Report
of the Chief Inspector of Social Services 2001/2002,
Department of Health and Social Services Inspectorate, August
2002

Websites

Fuller discussion of PIs on Social Services Research
Group site:
www.ssrg.org.uk