News

New performance assessment framework (PAF) results for 2001-2 have just been published along with refreshed star ratings and measurements of councils' performance for children and adults.¹ This will be followed by comprehensive performance assessments of all councils. Meanwhile, each of the 150 councils with social services responsibilities will be returning their autumn 2002 position statements to the Social Services Inspectorate.

Money and freedom to operate are linked to all this activity. In councils reported as performing poorly, staff morale and recruitment, and the reputation of workers and managers are all directly affected.²

The aim of the star-rating system and the PAF's 50 performance indicators (PIs) is to determine whether these councils are serving children and adults well. The Department of Health needs to know whether the evidence they have tells them this reliably.

What the PIs fail to do is to assess how far services cover the numbers of people in need in any council's population. The adult PIs do contain a number of rates per 1,000 population but these fail to assess the numbers likely to be in need. For instance, in England fewer than 3 per cent of adults had a completed assessment in 2000-1 and only 3 per cent of 0-17s had a service from a social services department in a sample week. Were those who did have an assessment or service the most in need? Some councils may limit access, others may provide extensive services but have less needy populations. Also, the indicators do not assess how far provision is meeting the specific needs of those who do receive a service.

For example, should a council whose over-65s population is fitter, has higher incomes and greater access to informal care and voluntary services have as high a rate of assessments or provision as a council that has the opposite of these factors? If lower PAF scores and fewer stars are given to the former, it has every right to feel hard done by.

Similarly, children's PIs do not address how good the coverage of social care is for children and families - there is no per 1,000 measure in the PAF set. Rather, they focus on those relatively few looked-after children or those on the child protection register (about 65,000 and 54,000 each year respectively out of 11 million aged 0-17). PIs for these children include measures of outcomes such as educational attainment and school absence, health checks, involvement in crime and whether they are moved often or have been reviewed or seen frequently. But no PI assesses adolescent mental health services nor support for disabled children. Nor do PIs measure children who need preventive work as they do not come within the remit of looked-after or child protection services.

In order to make a well-founded judgement of "how well these councils serve their populations", the SSI needs a set of measures of populations in need and, critically, a sober evaluation of where the money provided to each council goes in meeting that need. At present, judgements about stars depend principally on the measurement of what is reported as being done for those who make it through referral and eligibility.

The second function of the star-rating system is to determine how far councils have prospects for improvement. This is measured by the extent to which they are moving towards specific Department of Health targets, and by reported success in implementing a wide range of development initiatives. Here again the indicators and other measures used are too flimsy to bear the weight of the ratings derived from them.

For example, take the targets related to some key indicators.

The level of intensive home care (Best Value PI No. 53; PAF C28) measures all adults aged 65-plus with this level of service, thereby skewing the results by ignoring people under-65 who use intensive home care. Also, those receiving this level of service but funded by direct payments are excluded, which conflicts with the target that aims to increase the use of direct payments. Add in the uncertainty about what is being reported by some councils, and the basis for judgements of a key theme (sustaining more high-need users at home rather than in institutions) becomes uncertain. Moreover, the pressure to show increases in this key area may lead to some manipulation of the numbers.

To use another example about children's services - stars reflect how far councils have improved their rate of adoptions from care (Best Value PI 163, PAF C23). The numbers adopted in a year are measured against all children looked after at the end of the year, excluding those under section 20 agreements and those looked after for less than six months. The results suggest that councils are arranging adoption for about 5 per cent of them. However, given that adoption is rare for children older than 10 and unlikely if the child under 10 is placed at home, if these are excluded from the calculations, councils are arranging adoption for nearly 15 per cent. Stars for effort might be better based on this more appropriate measure and councils that are berated for low scores may in fact be doing a competent job.

The Department of Health would argue that assessments of "prospects for improvement" are made on much wider evidence than simply the PAF data. SSI twice-yearly monitoring has become increasingly sophisticated and wide ranging. Also, among the measures now used, councils are being asked to self-assess their progress on more than 80 plans to deliver the government agenda, choosing one of the following responses:

• Plans outlined by the council are not on target and are unlikely to deliver the objectives.
• Some plans outlined are on target and are expected to deliver objectives.
• Most plans outlined are on target and are expected to deliver objectives.
• All plans outlined are on target and are expected to deliver objectives.

It is in local authorities' interests to show that they are doing well - and even perhaps to make plans for some services that can be easily delivered. The SSI too will not want to report that many councils are nowhere near where the government wants them to be - also, it has no systematic means to cross-check what the councils report.

Even if local authorities are planning and implementing initiatives in line with the government agenda, there are questions about what evidence there is on which to judge improved performance. As SSI chief inspector Denise Platt noted in her annual report (in relation to grants to promote independence for older people), inspectors found that, "in many cases, there were no arrangements to evaluate their impact".³

Another issue is how authorities with overspends can move to roll out successful schemes. There are many tensions between targets and councils' ability to keep within budget. For example, it is likely that, rather than driving down unit costs, moves to restrict the number of entrants to permanent care and to increase home support will drive up both sets of costs. It is difficult to reduce permanent supported admissions (without redefining placements as temporary) if reductions must also be made in numbers with delayed discharges from hospital.

Keeping within budget is also difficult if the health secretary Alan Milburn commits councils to starting assessments within 48 hours of the user (or a third party) contacting them. Ensuring cover for sickness, holidays and training, and possibly having a larger number of staff having to work weekends will squeeze budgets - and probably be to the detriment of planned reviews being carried out.

Similarly, proposals to reduce time for delivery of equipment from 21 days to seven working days will have direct implications for additional store stock capacity, hire of extra administrative and delivery staff and an increase in cost per unit delivered - when efficiency savings are also being required.

Directors of social services are principally concerned to deliver their budgets on target. How far can this be squared with pressure to deliver more, reflected in PAF PIs and in completed plans? They will be balancing these priorities against the realities of recruitment difficulties and supply side problems in residential, foster and home care. And of course it is not just directors - middle managers and front-line staff need targets to be clear and meaningful to them, not mutually contradictory, unattainable and possibly contrary to the best interests of individual service users.

Also, improving councils' coverage of services to the population and the delivery of targets is made more difficult by the requirement to create new organisations incorporating health, education and social care. The experience of the new unitary authorities and the teething troubles of primary care trusts and the National Care Standards Commission show that it takes time to create new organisations. So it may take a significant time before measurable progress is made and councils run the risk of taking their eye off the many performance targets while organisational change is their principal focus.

Amid all the assessments there seems little opportunity to hear from those most directly affected by a council's performance: service users, carers and council and provider agency staff. Nor are there opportunities for health, education and voluntary organisations with real experience of authorities' performance to make their voices heard - although the SSI does now invite senior managers from other agencies to attend annual review meetings.

There is no doubt that the introduction of more systematic monitoring and measurement has focused councils on areas where improved performance was - and is - needed. But now it is time for a wider debate of the assessment regime. After five years of the PAF and six of joint reviews we need to move on - for the benefit of all those living in our local authorities.

Nick Miller is an information manager in a large social services department.

References

¹ DoH, Social Services Performance Assessment Framework Indicators 2001-2002, www.doh.gov.uk/paf

² See Community Care, 31 January and 30 May 2002

³ Denise Platt, Modern Social Services A Commitment to Reform The 11th Annual Report of the Chief Inspector of Social Services 2001/2002, Department of Health and Social Services Inspectorate, August 2002

Websites

Fuller discussion of PIs on Social Services Research Group site: www.ssrg.org.uk