Owen Carter and Mike Slavinsky on knowing your impact. Some context: The Brilliant Club. PhD researchers are recruited to go to work with small groups of high potential students through the Scholars Programme; its twin, Researchers in Schools, places PhD grads as trainee teachers in schools.
This session outlines ways to reliably and realistically evaluate interventions, intervention being anything that happens in addition to classroom teaching: mentoring, external provision etc.
When you’re looking at effect sizes, it’s important to look beyond the average. Feedback surveys, for example, show that 38% of studies showed a negative impact – so you have a 1 in 3 chance of doing more harm than good. (“Mark less!” I can hear Matt Pinkett instructing).
As well as looking at long term outcomes, it’s also important to consider intermediate outcomes, so it can be measured part way through, as opposed to waiting until the intervention is over to find out that it has failed. The Brilliant Club starts working with pupils in year 5, with a long term goal of entry into a highly selective university – so the intermediate outcome is really important.
(So, if my long term goal is to have more A-level students studying history at university, my intermediate goal is…)
Confidence, test scores, use of meta-cognition strategies might all be good intermediate outcomes. We discuss what we are using as intermediate outcomes, but we all seem a bit stumped tbh. Luckily, we move on to discuss valid and reliable tools to help with this.
We do a short questionnaire to show us how to measure confidence. We look at the Marvel app, which helps with the building of these questionnaires and analyse the results, and discuss how this might be used. We then discuss what sort of intervention we do and how we assess its impact.
Historical control groups can counter the ethical issue of not intervening with a group to give you a control group. Using outcomes of similar groups of pupils at similar schools can give you a good idea of what happens without the intervention you’re planning.
Case study. In an intervention – low stakes quizzing – aimed at improved GCSE performance, the intermediate outcomes identified were better achievement at exam questions and better meta-cognition, motivation and extraversion. In these, students did increase their ability to deploy cognitive strategies; lower attaining students became better at judging their own progress. Look out for this in Teaching History in the future.
Good point: if an intervention doesn’t work, it could potentially save the school a lot of money, so negative outcomes are not to be ashamed of. They’re also useful for refining approaches.