STFC evaluation framework update

Presenter: Liz Poulter and Dan Hillier, STFC  

Theme: Evaluation and impact 

Reporter: Laura Thomas 

One sentence summary: In this session STFC presented their draft evaluation framework. They were looking for comment and feedback on the proposed metrics and outcomes for evidencing impact of public engagement activities. 

Description 

The session was jointly presented by Dan Hillier and Liz Poulter of STFC and they provided background on the previous STFC strategy and approach before going on to discuss the proposed changes to the evaluation framework. Participants had the opportunity to ask questions and feedback on the metrics and outcomes presented.  

Setting the scene 

Dan began with an overview of the scope of the public engagement work undertaken by STFC, with this including grant-funded work (eg. Nucleus and Spark award schemes), activities and projects led by the laboratories, and partnership working. The previous framework had been in place since 2017 and had more of a quantitative focus.  

The new evaluation framework is aligned with their new strategy and theory of change and they noted that they have also worked to ensure alignment

with ASDC’s inclusion framework. In summary, their goals for the framework are to provide a concise set of outcomes and metrics which encourage reflection with a shift to more qualitative reporting.  

Introducing the metrics and outcomes 

Liz then moved on to sharing the metrics and outcomes. She noted that with the previous set of outcomes it wasn’t always clear what people should be evaluating. They have identified a set of essential and optional outcomes and who they are evaluating. An example of each is provided in the table below. 

Outcome Who we are evaluating 
Essential: Feel inspired by STFC science, technology and people. Participants 
Optional: Community groups, teachers and partners feel their expertise and priorities shape the funded activities delivered. Grant holder partners 

National Lab PE partners 

 

An example of the strategic aims and proposed linked metrics are in the following table: 

Strategy aim Linked metrics 
Aim1: Celebrating STFC science, technology, people and careers 
  • Number of STFC funded researcher and staff involved in programming 
  • Number of events 
  • Duration of the event 
  • Number of participants/number of students 

 

These metrics have been streamlined compared with the previous framework.  

Discussion 

Participants were then asked the following: “What are your views on the proposed outcomes and metrics? Does anything excite you? Does it pose any challenges?” 

Dan and Liz circulated and answered questions and discussed comments from the groups on the proposed metrics and outcomes. A number of different points were raised when everyone was brought back together. 

  • One person asked about people and inspiration. Did this put pressure on STFC people to be inspirational and do the metrics capture the diversity of people contributing to activities? Liz noted that there has been a long discussion about using inspire in the framework as it can be a challenging concept to measure, but it has been selected by STFC Council as an outcome of interest. The intent was not to put pressure on staff, and Dan commented that staff members that are relatable can be inspirational. They indicated they will consider whether to include something on capturing the diversity of those contributing to projects. 
  • There was a query about the use of engagement hours. Liz commented that they did have dwell time as a metric but had removed it. They see value in small numbers and longer-term engagement and feel quality is measured in other ways rather than just numbers of participants. 
  • Another commented on relevance and how this might be interpreted by audiences. Liz notes that again this was subject to significant discussion before inclusion in the framework. STFC will offer examples of questions which will fall under the umbrella of relevance and projects can tweak the individual questions. 

Conclusion 

STFC want to be able to celebrate and share what is being done and intend to look at producing an annual review of public engagement activities. In terms of next steps, the framework will be finalised by December 2024 and published online early 2025 with data collection via Researchfish going live shortly afterwards. 

Key messages 

  • STFC are updating their public engagement evaluation framework, which will be made available in early 2025. This will be applicable to all of the work being funded and undertaken by STFC. 
  • The evaluation framework links to their new PE strategy and theory of change. 
  • STFC are keen to manage expectations of STFC-funded PE projects so that they have a clear understanding of what they need to report whilst also encouraging more reflection and review of the work being done with a general shift towards more qualitative measures. 

Resources 

New STFC Public Engagement strategy postcards

Symposium Partners