Evaluating partnerships is challenging and needs to be carefully commissioned, write Helen Dickinson and Jon Glasby in the final part of their series on structures
Although evaluation is often considered a technical term we are all involved in it and make evaluations on a daily basis. At its most basic level evaluation may be considered the process one goes through to decide the merit, worth or value of something.
Usually we are looking to get best value for the money we spend, or find the product or service which is most suited to our needs. But we do not only make judgements over the worth or merit of products and services that we are personally involved with purchasing. Whether it is reports of taxpayers’ money being “wasted” through private-financed hospitals, large-scale procurements of computer systems for public services, or rebranding services such as the Child Support Agency, not a day goes by when there is not some report in the media over the alleged misuse of tax-funded services, organisations or products.
We do not evaluate tax-funded services simply to make sure that they are providing value for money purely on a cost basis though. We also want to make sure that the people using them are receiving high quality services and products. Although choice has recently attained a prominent place in the health care agenda, many of us have had little choice over the public services we receive and would expect them all to offer the same high standards.
Moreover, people with complex or chronic conditions may not be able to either actively judge the quality of services which they receive, or have little to compare them with. Such services need to be evaluated to ensure that individuals have access to the quality services they want and need.
Interagency working has assumed a prominent position within public policy not only in the UK, but also more widely throughout the developed world. Much of the rhetoric around the driving factors at the heart of this shift relate to the provision of better services for those who use them, and an aspiration to “create joined-up solutions to joined-up problems”. This reasoning has been further supported by a series of high-profile cases where inability to work effectively in partnership has been presented as a major source of failure which can have very real, negative consequences for individuals and their families.
Despite international interest in partnership it has not been demonstrated that this way of working necessarily improves outcomes for service users. This might be considered problematic in itself (given that partnerships have assumed a central role in many areas of public policy). However, in terms of UK health and social care this might be considered even more questionable given the importance that the New Labour government has afforded to the concept of evidence-based policy and practice. Therefore, evaluating the outcomes of health and social care partnership working is an imperative, if not overdue, task.
However, partnerships are difficult to evaluate effectively and evaluations involve a series of trade-offs regarding what sort of coverage is gained, whose perspectives to involve and the main focus of the study. Not every evaluation will be able to cover every possibility. For example, the National Evaluation of Sure Start is probably one of the largest partnership evaluations of recent years but early reports found little in terms of impact of the programme in those areas targeted by the initiative – in fact some children were found to be worse off.
But this evaluation encountered a range of difficulties. Many of the targets that Sure Start is set up to achieve are long-term and it could be argued that we would not expect to see the real impacts until the children in these areas reach the latter half of their teenage years. This evaluation also demonstrates the difficulties involved in trying to generalise the impacts of quite different local programmes which all exist within particular contexts.
There were also problems with evaluating services in socio-economically deprived areas. When such programmes help individuals and families overcome several key challenges, often one of their first acts will be to move out of that area. These families are usually then replaced by people who have a fresh range of challenging issues to contend with. In other words, although the population remains constant and might reflect similar trends at a macro-scale, this is actually hiding some quite significant changes at the micro-level.
What this means in practice is that we need to be clear from the beginning of the process, not only what the partnership we are evaluating was set up to achieve, but also what it is that we want to achieve in terms of the evaluation.
Once this has been established and agreed upon by the evaluators (and most likely the commissioners of the evaluation) we can then go about selecting which approach is most suitable to that specific partnership – and acknowledge the limitations which this will involve.
Partnerships are tricky things which are difficult to make work at the best of times. Currently they are somewhat in vogue within national and international public policy and more widely within the commercial sector. In one sense this is a positive achievement and more attention than ever has been focused on attempting to provide seamless and accessible services to individuals, families and communities who are often in times of need or experiencing chronic and complex problems.
However, partnership working offers enormous challenges in terms of the ways in which individuals, organisations and sectors can work together in productive and creative ways. The popularity of this concept has also meant that many different ways of working have been subsumed under one umbrella concept – when in fact, partnership takes many forms and is propelled by a variety of drivers. This, in turn, poses an enormous evaluative challenge. The future of partnership working is ultimately dependent on an ability to evidence it. Without demonstrating the positive – and not so positive – impacts that partnerships have then we risk undermining its value.
Implicit in assumptions about partnership is that it is necessarily a “good thing” – and it probably is to some degree. However, to keep on expecting front-line staff to engage with this agenda is naïve when their everyday experiences are of challenging behaviour and organisational and procedural complexity, and wider research is unable to say little definitive about benefits.
It is imperative that more is invested in partnership research so we are better able to say: what types of benefits partnerships are able to deliver for whom how local organisations can work together more effectively to produce better partnerships and, importantly, when partnerships are not necessary and might even be counter-productive. Such information would re-invigorate the partnership agenda and renew its legitimacy.
Helen Dickinson and Jon Glasby, Health Services Management Centre, University of Birmingham, are the editors of the new Better Partnership Working series. For further information, click here
● The previous articles in this series are available at http://www.communitycare.co.uk/108216 (published in the 15 May edition of Community Care), http://www.communitycare.co.uk/108537 (19 June edition), http://www.communitycare.co.uk/108849 (24 July edition) and http://www.communitycare.co.uk/109112 (14 August)
Factors to consider
● Often, large-scale initiatives are long-term and initial evaluations can be misleading.
● Evaluating local programmes can produce poor results because as soon as some people are better off they move out of the area to be replaced by worse off people. Thus the evaluation does not reflect the success of the scheme.
● The diverse nature of multidisciplinary teams and partnerships – for which there is no single template – makes them hard to evaluate in general terms.
● Effective evaluation of partnerships is crucial if we are to establish what sort of structure works best in what sort of situation. Evaluation may show that partnerships are not needed in a particular area.
Follow the series
● This article is based on material from Helen Dickinson’s Evaluating Outcomes in Health and Social Care, the final book in a new series focused on better partnership working. Published by Policy Press in association with Community Care, the five books seek to provide a series of accessible ‘how to’ guides. These books summarise current policy and research in a detailed but accessible manner, offering practical support to those working with other agencies and professions and provide some helpful frameworks to make sense of the complexity of partnership working.
This article is published in the 11 September issue of Community Care under the heading ‘Getting the measure of partnerships’