Our ESF evaluation for Welsh Government – measuring social outcomes remains the challenge
Evaluation of the ESF-LSB Project for the Welsh Government
Si was part of the team that delivered a three-year evaluation (2012-2015) of the European Social Fund Local Service Board Development and Priority Delivery Project (ESF-LSB Project) commissioned by the Welsh Government. The ESF-LSB Project received funding with the intention of improving the effectiveness and efficiency of public services through more effective collaborative working and building the capacity of public services to deliver higher quality services.
The evaluation was established to assess:
- whether the ESF-LSB Project has led to better public services and outcomes for citizens;
- how the processes by which the Project has been implemented and managed have contributed to these outcomes;
- whether (and how) the Project has disseminated learning from the delivery projects, nationally and locally; and
- how the Project has improved the integration of the third sector with LSBs, whether this has led to better services and outcomes, and if this progress is sustainable.
The evaluation included a formative and summative phase. The methodology for the formative phase of the evaluation was made up of: a literature review and development of the theory-based framework; scoping interviews with all delivery projects; two rounds of interviews with national stakeholders; baseline for a two-wave all-Wales Public Service Survey on collaboration; in-depth case studies of seven delivery projects; and analysis of discussions at three learning events. The summative phase aimed to provide an overall assessment of whether and how the outcomes of the ESF-LSB Project have been achieved.
The conclusions and recommendations made as a result of the evaluation were structured according to the four aims of the Project:
Better public services and outcomes for citizens
The evidence that has emerged since the closure of the delivery projects shows that while processes have been put in place and outputs delivered, overall it is still difficult to assess outcomes for service users. Respondents from ESF-LSB delivery projects were more likely to agree that they had delivered against their goals than their counterparts from other partnerships. Many of these goals, however, were processes rather than outcomes. Moreover, the Welsh Government tried to get projects to develop local evaluation frameworks so they understood how activities lead to outputs and outcomes, but the use of such methodologies was inconsistent.
The overall summative evaluation was that mainly process outcomes (rather than service/citizen outcomes) have been delivered.
Implementation and management processes
All the evidence on the role played by project managers was positive. They deployed both generic skills for relationship building as well as project management skills. There was a concern, however, about whether project managers, on their own, had sufficient influence to get things done particularly if there was an absence of strategic leadership.
While the projects had to produce annual reports for the Welsh Government, it was not always clear how the performance of the projects were being held to account. The role of scrutiny was not embedded within delivery projects’ accountability mechanisms.
Learning from the delivery projects
The evaluation suggested that much of the learning that takes place about collaboration for achieving outcomes will be ad hoc, but is fragile and depends on individuals, unlikely to be more widely available to other projects and professionals and may not be sustained.
Integration of the third sector and LSBs
The evidence suggests that neither the ESF-LSB funding nor the Making the Connections funding has been particularly effective in improving LSBs’ relationships with the third sector through collaboration on specific projects. This raises questions about how influential third sector representatives on LSBs are, particularly as the survey findings showed that less than half of delivery projects agreed that the third sector was a key member of their partnership and/or had the capacity to engage in the partnership.
You can see the full report here: http://gov.wales/statistics-and-research/european-social-fund-local-service-board-development-priority-delivery-project/?lang=en