News

Social services departments have been given star rat  ings, but will these really make services more accountable to users? Not necessarily so judging by those applied to NHS acute trusts, writes Tony Cutler of the University of London.

The star ratings of social services departments are virtually upon us, part of an inexorable trend to more performance measurement in public services.

A common rationale for these exercises is that they promote accountability to service users and the general public. But will this be the case?

To answer this it is instructive to look at the last instalment of star ratings, which were those for NHS acute trusts, in order to explore the lessons for public sector performance measurement. This is most usefully done by posing several simple questions that the service user and concerned citizen might ask in their search for “accountability”, and by considering how far the acute trust ratings provide relevant responses.

In the NHS, the ratings, published under the title NHS Performance Ratings Acute Trusts 2000-1,¹ divided trusts into four categories from “No Stars” to “Three stars”. Twenty-one measures were used, nine of which referred to “key targets”, which were said to determine whether trusts attained at least a two-star rating; and the further 12 measures were designed to “refine the judgement” as to whether these trusts attained three or two stars.

For each measure there were three categories: targets could be achieved, there could be “some degree of underachievement” or targets were “significantly underachieved”. The ratings had “teeth”, for the management of “no star” trusts were expected to develop action plans. Unsatisfactory plans to “turn around” the trust would result in new management teams being introduced.

So did this system enhance accountability? To answer this we should first look at why these particular measures were chosen. This is fundamental, since it determines what managers are being held to account for. A visit to the website for the report will show each measure’s, “rationale for inclusion”. However, these are frequently narrow. For instance, one key target is the familiar one of reducing waiting lists - the rationale refers to the 1997 Labour manifesto commitment of cutting waiting lists by 100,000. However, there is no recognition that waiting lists are a problematic measure, for in focusing on numbers alone all those waiting are treated as of equivalent significance, whatever the severity of their clinical need. Therefore this concept of a rationale is often attenuated and fails to deal with the debates over key measures.

Second, why are some measures given greater significance than others? This is also a crucial question since the so called “key” targets are given greater weight in determining the ratings. However, this question is not addressed at all, and the DoH acute ratings report states that the “standard of care” is not being measured, but rather what it terms “the overall patient experience”. As a consequence three clinical indicators are not included in the nine key targets, and this raises the question as to how the “patient experience” can be divorced from “standards of care”. It also deepens the confusion as to why some targets are key and others not: for example, the achievement of a “satisfactory financial position” relative to planned income and expenditure is a key target whereas reduction in emergency readmissions following discharge from hospital is not.

Third is the question of how the measures should be interpreted, or what the measures are supposed to mean. A good example is provided by the three non-key measures referring to the filling of staff vacancies regarding consultants, nursing, midwifery and health visitors, and allied health professions. In each case the rationale is the same, which is, “to highlight recruitment and retention problems so that areas with the most challenging problems can be assisted”. This suggests that such problems are either partially or not at all within the control of management and reflect variable conditions in local pay and property markets. However, they also constitute targets and are measures, where failure to achieve will result in a lower place in the ranking.

Fourth, are the performance measures related to the context in which the organisations are working? In the NHS there are major factors which can affect performance, including the nature of the capital stock deployed or the availability of complementary services such as residential care. While the report seems to show an awareness of such issues - for instance by referring to “no star” trusts it states that while they are poorly performing this does not mean that “staff are not working hard in often very difficult circumstances” - there is little detail about such issues and their impact on performance. Consequently, after the publication of the ratings there was considerable comment regarding the fact that while none of the 12 no-star trusts were located in the north of England, half were located in Surrey, Sussex, Kent and Middlesex. Similar regional differences appeared in the targets for financial performance. It was thought that this might indicate the existence of a “southern” effect, where tighter labour markets and other cost drivers affect service provision, but this issue was not addressed in the ratings.

Another obvious question is whether the data collected is sufficiently reliable to support the star ratings. Here, the DoH report states that the ratings are based on data and information that are “not perfect”, but “the best available”. In one case (South Warwickshire General Hospitals Trust) the report indicates a reclassification from three to two stars because of “significant concerns regarding data quality”. However, there is no systematic discussion of what the principal data limitations were and how they might have impacted on the ratings.

Lastly, is it clear, given the evidence presented, how the ratings were arrived at, and are processes involved used with consistency? Take for example the University Hospitals Coventry and Warwickshire (UHCW) Trust. Referring to the key targets this trust under-performed on one key measure, which is a level comparable with the vast majority of “three star” trusts. It also did not do well on the supplementary measures, under-performing on seven and significantly underperforming on two. Given the purported role of these supplementary measures this might be thought to qualify the trust for a two-star ranking. However, UHCW was classified in the no-star category and a footnote to the report indicates that this result was determined by an assessment by the Commission for Health Improvement (CHI) that the trust “performed poorly across the elements of clinical governance and lacked strategic capacity to remedy these weaknesses”.

This suggests a  superordinate status for CHI reports. However, in the text of the report 10 key targets are identified - the nine key targets referred to above and, “not receiving a critical report from the Commission for Health Improvement”. This suggests that the CHI report has the same status as the other nine measures and raises the spectre of multiple and conflicting criteria.

The conclusion must be that members of the public in search of accountability will not be well served by these star ratings.

Perhaps the central problem is the difficulty of reconciling the complexity of performance measurement with the concern for accountability. Current official performance measurement has not grappled with this issue, arguably because for the government, references to complexities are regarded as excuses advanced by managers and professionals for poor performance. Unfortunately, the same problem is likely to arise now that the social services department ratings are published.

Key targets

Key targets for performance ratings of acute health trusts 2001:

- Shorter inpatient waiting lists

- No patients waiting more than 18 months for inpatient treatment

- Reduction in outpatient waiting

- Fewer patients waiting on trolleys for more than 12 hours

- Less than 1 per cent of operations cancelled on the day

- No patients with suspected breast cancer waiting more than two weeks to be seen in hospital

- Commitment to improving the working lives of staff

- Hospital cleanliness

- A satisfactory financial position

- Not receiving a critical report from the Commission for Health Improvement (CHI)

Dr Tony Cutler is course director of the MSc in health services management, school of management, Royal Holloway, University of London.

¹ Department of Health, NHS Performance Ratings Acute Trusts 2000 - 2001, DoH, see website www.doh.gov.uk/performanceratings/performance.pdf