Effective and ethical evaluation

Presenters: Laura Thomas laura@ondata.org.uk; Charlotte Thorley
charlottethorley@gmail.com 

Theme: Evaluation and impact 

Reporter: Dan Cottle 

One sentence summary: Evaluating public engagement and outreach activities can be complex but it can be navigated successfully by planning, working ethically and drawing upon the expertise of others when needed. 

Description 

The workshop started with small group discussions about scenarios that had been set out on each table. Discussing these helped everyone on our table introduce themselves, feel welcome and engaged right from the start.  

The role of ‘hybrid evaluators’ of outreach projects was introduced: physical scientists, working in public engagement and outreach gradually who became interested in social science research methods for evaluating the effectiveness of their activities on people., Charlotte explored how the positionality of those who evaluate can influence the types of impact they are looking for and the questions they ask.  

The session went on challenge us to think about the ethical implication of some of the scientific knowledge we promote, for example, inspiring children with photographs of astronomical objects taken by telescopes that are situated on land taken without consent from indigenous communities.  

The main point of the session was that evaluation should be an essential and integrated part of outreach activities but can be intimidating and complicated to think about. We were reassured that despite the challenges of handling personal data, GDPR, ethics committees and feelings of being out of our comfort zones, we could handle the complexity and produce evaluation that was useful to funders and added to our knowledge of how to improve our practice. During the session, big, provocative questions were asked as stimuli to our thinking including: 

  • Is being ethical in evaluation being risk averse? 
  • Do we really need to do use surveys to evaluate the impact of outreach projects?  

After the big ideas were presented, we were directed back to the scenarios on our tables. We were asked to individually think critically about the ethics of the situation presented and then discuss with others on our table. As an example, one of the scenarios presented was: 

Overview Session Description Aims Objectives 
Working with a primary school science club Support half a term’s worth (6 to 8 weeks) of science club sessions with a sustainability focus. You are involved in developing and delivering a series of pre-planned hands-on activities To support the development of a positive culture of science within the school Increased knowledge of climate-related topics for pupils and teachers 

Increased self-confidence when taking part in science-related activities for pupils and teachers. 

 

The questions for discussion were: 

  • What evidence would be available to address the aims and objectives? 
  • What data collection methods would we use to gather evidence? 

Some of the collective challenges identified were problems with obtaining ethical permission to do evaluation from university ethics committees, avoiding the use of surveys with children if possible, and problems communicating with all the stakeholders in outreach activities (teachers, children, parents, school leaders…).  

In concluding the session, the messiness and complexity of outreach work was again acknowledged. The reality is, we don’t all work in ideal situations, and we need to be ok with not obtaining ideal evaluation data! My key takeaway will be to work harder on framing clearly the aims and objectives of our outreach work from the start, so we can integrate collecting the data into the activity itself.  

 

Key messages 

  • As people with backgrounds in science we have a unique perspective on evaluating the impact of public engagement and outreach. 
  • We should consider the ethical implications of the science we promote and the way in which we evaluate engagement and outreach. 
  • Evaluation is often messy as it involves dealing with people! 
  • Make sure there are clear and realistic aims and objectives of public engagement and outreach supports effective outreach. 
  • You don’t have to do all of the evaluation yourself – bring in other specialists and the OPE community for advice. 

Resources 

 

 

  • Session Materials available at: https://ondata.org.uk/interact2024/This link takes you to a page that hosts the presentation slides and activity scenarios used by Laura and Charlotte in the session, brief guides to evaluation methods and setting out your ethical position, an example evaluation plan template, and some relevant references. 

Symposium Partners