Unlocking Success: The Power of a Research-Based Theory of Change for EdTech Companies
Differences in how different countries define standards of evidence in EdTech can often create confusion among companies. This confusion revolves around understanding the required research studies necessary for establishing a robust evidence base. In our review of various evidence frameworks and their connections, we noted a shared expectation: EdTech companies are expected to have a research foundation that documents the scientific principles behind their products and services. This research foundation essentially constitutes a Theory of Change.
The first step in developing a research-based theory of change, is to establish a logic model. In some frameworks, like the ESSA Tier IV used in the USA, there is a clear expectation for EdTech companies to have a logic model. A typical Logic Model lists what the inputs and activities of an EdTech solution are and what outcomes and impacts a EdTech organisation expects. A Logic Model is a bit like a roadmap that shows how different published studies and research fit with the EdTech organisation's ideas. For example, if an EdTech organisation thinks that online tutoring will boost students' scores, they should be able to find research that supports this. But if their idea is so new that there's no research yet, they should at least have a solid theory that explains the rationale of their idea.
To ensure that a theory of change is well-supported by relevant research, it is crucial to select studies that directly align with the features of a particular EdTech solution. This highlights the importance of conducting a comprehensive product review to confirm a match between the product, its unique value, and the available research literature. Generic or weak logic models may be applicable to products with similar targets, but they lack specificity. In WiKIT, we always try to pair subject matter experts from our network of affiliated scientists with the EdTech organisation's specific area of focus. This ensures that when recommending studies, a targeted and thorough literature review can be conducted, tailored to the EdTech organisation's unique value proposition.
Strong theories of change are tailored to the unique characteristics of the product and directly correlate with specific studies and their findings.
Establishing this specific match requires the expertise of a qualified PhD-level researcher capable of conducting a critical literature review. A critical literature review goes beyond surface-level examinations on platforms like Elicit.org or with ChatGPT, but instead delves into the nuanced findings of studies. A thorough review considers whether cited studies are foundational works that other researchers have built upon, assessing the rigour of the studies before recommending it to the EdTech organisation. It also examines potential counter-evidence that may contradict the EdTech organisation's approach. Such cases stimulate discussions between the researchers and the founders and may result in revisions of original impact statements.
Research-based theories of change serve as a compass or decision-making tool for the EdTech organisation in the long run, supporting their product development. They enable the formulation of impactful mission and vision statements that are distinct to the EdTech organisation, grounded not in lofty aspirations but in precise and measurable impacts that can be monitored over time.
For instance, if the theory posits that over time, students will develop increased confidence, leading to heightened motivation to complete high school, these become the key impact metrics to prioritize and gather data on. This integration with the data infrastructure is crucial, as the theory of change guides the selection of data points necessary to continuously track progress and achieve the impacts outlined in the theory.
Weak logic models and theories of change often fall short by merely suggesting pathways without establishing connections to the relevant impact metrics, or list only assumed linear relationships from inputs to outputs. Sometimes, the relationships between the activities supported by an EdTech and the expected outcomes may indeed be linear. However, most of the times, the assumed connections between inputs, outputs, and short- and long-term outcomes are not linear but rather circular or complex, involving various mutually interactive factors. For instance, consider the theory of change developed for Learning Ladders (a student progress tracking software designed to support primary and early years schools to keep on top of assessment data, prepare for inspections, and keep parents informed).
As part of WiKIT’s process of developing research-based theories of change, we establish a research library for each EdTech organisation, compiling a selection of open-access studies that best support their model. Our researchers gather these studies into a folder, which we share with the EdTech organisation. We believe itis essential for every EdTech organisation to curate such a repository of studies. A research library can be used for in-house training and should be regularly updated with new research findings that inspire and inform the organisation’s approach. Being a frequent reader and visitor of the research library is part of being an evidence-informed EdTech!