7.6
There is currently a strong emphasis on instrumental and institutional value in evaluation and much less so on intrinsic (artistic) value, yet the more intangible values are often crucial in successfully securing a commission. The measurement of artistic quality remains challenging for projects and those responsible for commissioning or funding. Whilst artistic value is accorded weight in Creative Scotland funded projects, those involved in analysing public policy are often interested in value for money and cost-benefit analyses. Festivals Edinburgh has made progress by commissioning reporting on social, cultural and economic impact in different years, which should inform future thinking in the development of outline monitoring and evaluation framework.
7.7
There is currently a lack of understanding and expertise of the impact of new media on audiences and the cultural experiences they enjoy. The emphasis remains on assessing established media narratives and not how that message is received and shared via social media. Assessment of co-created impacts needs to extend beyond the superficial ‘likes’ and ‘follows’ to understand reach, communities, influence and networks formed. It is important that proactive analytics and data collection methods are in place ahead of programme and project launch and there is agreement on what official communication references are being used; i.e. what is the hashtag (if one is needed), what are the official project account usernames, whether there is a monitoring system in place to update and inform the project developers of alternative spellings or references to their project and what the key objectives for using social media as a promotional and engagement tool are in the first place.
7.8
Whatever approach is chosen, strategic clarity must be accompanied by operational support, from funders and other stakeholders, in the practice of planning for and implementing monitoring and evaluation. The research symposium participants indicated that they would benefit from having the main funding partners brought together to share their knowledge and expectations for monitoring and evaluation. Evaluation training workshops could precede project commencement, focused on reporting requirements, evaluation techniques and resource requirements. Small teams without expertise in evaluation could get advice from experienced project leads from within and outside of the cultural sector, emphasising a collaborative culture. It is also worth giving consideration to the appointment of an evaluation partner (or mentor) for each project from within the main funder. Projects called for progressive, open relationships between funders and those receiving the funding.
7.9
Multiple reporting requirements created significant challenges for projects in managing their monitoring and evaluation processes as part of the Programme. There was an expressed desire for more standardized, yet flexible evaluation tools, involving a more nuanced framework with ‘opt in’ and ‘opt out’ modules. However, this new framework must not be in addition to the plethora of others in use at the moment but instead must rationalise those already in existence, including eventImpacts. A strategic consensus needs to be built amongst key funders as to their information needs and work to produce a useable set of tools that will avoid duplication and maximise the quality and comparability of data.
7.10
There is also need for clearer information on where monitoring and evaluation reporting goes and what impact it has on policy and practice. This would enhance the credibility of the process, secure buy in from projects and contextualise the rationale for evaluation that Creative Scotland has to work within.
Defining the object of study and area of impact 7.11
In bringing together data to analyse the scope and scale of the Programme, Creative Scotland made some choices about those projects which were felt to constitute the programme, and about the time period which the programme ran for. Establishing as early as possible what constitutes the ‘object of study’ for the evaluation of Glasgow 2014 Cultural Programme (what projects are included, in full or some elements) will be crucial in ensuring comprehensive data collection which neither runs the risk of over or understating the scale of the Programme. In addition, there are important considerations in relation to where and when the impact of an activity is anticipated to take place. For example, if a project tours beyond Scotland, is this activity still considered to be contributing to outcomes within Scotland? 41
