Effectiveness and Efficacy: Clearing Up the Confusion
At WiKIT, we collaborate with a diverse range of companies within the K12 EdTech sector, each with unique evidence requirements. During our initial consultations with these companies, we conduct a comprehensive diagnostic assessment to ascertain their existing evidence landscape. For some, evidence may be synonymous with a randomized controlled trial with a mature product, while for others, it may mean a few teachers’ reviews of a new prototype. To ensure fair ranking among the various types of evidence available, we conducted a comprehensive review of existing frameworks, relevant scientific literature on evidence and educational sciences, and engaged directly with multiple EdTech organizations. Based on this research, we categorized different types of evidence and their supporting studies in a manner that is relevant to both EdTech providers and policymakers.
EdTech, isn't just any technology; it's specifically designed for education. That means that to understand EdTech’s impact, we must consider what both researchers and educators consider as evidence that an EdTech works.
This concept is encapsulated by two distinct types of impact: efficacy and effectiveness. Efficacy seeks to answer the question, "Does it work?" while effectiveness considers "Could it work?" and "How could it work?" Essentially, since humans are at the center of EdTech, impact is not just about whether an EdTech tool works in controlled settings, but also understanding how it could work when educators and users directly influence its design and implementation. Holistic EdTech evaluations must encompass both efficacy and effectiveness for a thorough evaluation of the tools’ impact.
Types of Efficacy Evidence and Studies at three levels, WiKIT, 2023
At WiKIT, we've been grappling with the question of how efficacy and effectiveness differ and relate to each other for some time. In a review of existing national EdTech standards of evidence, we identified three key levels of how the standards evaluate whether an intervention is evidence-based: emerging, established, or proven.
Level 1: Evidence-ready
Emerging evidence pertains to theoretical evidence based on a theory of change or some form of literature review. Because EdTech serves as pedagogical tools, evaluating effectiveness goes beyond literature reviews to encompass user experience, teacher feedback, and numerous design testing to ensure the tool works as it should. This aspect represents the practical application of emerging evidence. All EdTech organizations should have some foundational evidence that demonstrates how and why they believe their technology works. At WiKIT, we prioritize establishing this foundation as the first step when organizations engage with our research services, and we have developed the product called Get Evidence-Ready to streamline the process.
Level 2: Evidence-based: emerging
Transitioning from conceptual evidence to empirical evidence involves moving from reviews to actual testing of the technology with users in classrooms. In effectiveness evaluations, studies conducted with teachers, drawing on existing resources in the classrooms, aim to understand how EdTech tools could be implemented. In efficacy evaluations, studies run with a control group determine whether the solution works. The presence of a control group allows for testing under ideal parameters to precisely gauge the added value of the tool in the classroom. The more controlled the study, the closer it aligns with experimental parameters, including randomisation to a group of participants that gets the tool and a group that gets a different tool or no tool.
Level 3: Evidence-based: strong
At the third level (represented by the smallest squares in the two figures depicting efficacy and effectiveness evidence), the efficacy strand of studies employs the gold standard: randomized controlled trials. In effectiveness studies, the highest standard is deemed replication studies, demonstrating that an intervention works across various contexts, including factors such as cost and resource effectiveness. Once EdTech organisations achieve this level of evidence, they can be considered to be strongly, or significantly evidence-based.
Types of Effectiveness Evidence and Studies at three levels, WiKIT, 2023
How to achieve a balanced efficacy-effectiveness evidence portfolio?
Conducting rigorous efficacy and effectiveness studies requires not only time and financial resources, but also collaboration among multiple stakeholders. At WiKIT, we partner with organizations like LXD Research for efficacy studies, specializing in ESSA-aligned studies in the USA, and engage with various testbeds worldwide through networks like the GETN for effectiveness studies. We prioritise working with local researchers and recommend local university teams to lead the studies whenever possible.
The larger squares indicate a higher demand for resources in conducting the studies. The relatively smaller size of the blue squares illustrates why randomized controlled trials and replication studies are much more resource-intensive than conducting a review, making them rarer in EdTech. On the flip side, the size of the squares also emphasizes why most EdTech interventions have, or should have, at least some usability tests with teachers and a robust literature review supporting them.
As academia-industry partnerships in EdTech continue to flourish, the importance of both efficacy and effectiveness studies is increasingly recognised and incentivised. If you share our belief in the significance of these studies for the advancement of EdTech and its impact - whether you are an EdTech provider eager to conduct a study, a testbed seeking collaboration to evaluate solutions in classrooms, or a researcher interested in contributing to efficacy and effectiveness studies - we would love to hear from you.