What Variables Determine the Efficacy of EdTech in the Classroom?

“We need to move beyond the first level of the digital divide and focus on the second level: how technology is actually used to facilitate learning. It is not enough to purchase potentially innovative technologies; we need to effectively and equitably integrate them into learning experiences. This is going to be mission-critical if we hope to fulfill the promise of technology as a tool to help all students succeed.”

Christine Tomasik, Chief of Staff and Director of Partnerships, EdTech Evidence Exchange

Edtech tools and products serve as an excellent avenue for students to access learning opportunities and for teachers to better support learning goals in and out of the classroom. They have become especially useful as tools to support education in the mass pivot to distance learning throughout the evolving novel coronavirus pandemic.

And even before the pandemic, the U.S. was spending between $25 and $41 billion per year on education technology—but schools and districts have largely made these massive investments with little tangible data about which tools work where or why. And as a result, this often leads to poor tool selection, management, implementation, and use of edtech products.

To tackle these issues, researchers at the EdTech Evidence Exchange who have studied the development of edtech tools for nearly a decade recently published an “EdTech Genome” study in collaboration with more than 140 teachers, policymakers, and other stakeholders in education identifying and quantifying variables that determine the efficacy of edtech in the classroom.

We spoke with an expert who managed the research project to learn more about the study’s findings, useability, and implications for students, teachers, schools, and companies engaged in edtech.

Christine Tomasik, Chief of Staff and Director of Partnerships, EdTech Evidence Exchange

Christine Tomasik is the Chief of Staff and Director of Partnerships at the EdTech Evidence Exchange, an organization aimed at tackling the problem of low-information edtech purchasing and implementation through cross-sector collaboration, research, and collective action. For the organization’s EdTech Genome Project, Tomasik provided organizational leadership for the report from its inception to its publication.

Tomasik is a leader and instructional specialist with a growth mindset transforming the way teachers and students interact with content through pedagogy, technology, and research. She has over 25 years of experience impacting students’ learning through teaching, leading, creating, connecting, and facilitating professional growth experiences for educators.

Tomasik is also currently a doctoral candidate at George Washington University for an EdD in human and organizational learning. She holds a master’s of education from Holy Family University, a certificate of advanced graduate studies in administration and supervision from Johns Hopkins University, and bachelor’s degrees in English, French, and sociology.

Published in July 2021, the “EdTech Genome” study is a collaboration spearheaded by the EdTech Evidence Exchange. The study found that decision-makers currently select and implement technologies with almost no information about what is likely to work in their schools. They spend tens of billions of dollars each year on edtech that is underused, inequitably used, or ineffectively used.

According to the report, “This cycle persists because educators lack the shared language, incentives, and mechanisms to document their experiences with edtech and to share lessons learned with other educators working in similar schools and districts.”

Ultimately, the EdTech Genome Project built a sector-wide consensus outlining the 10 consequential context variables, definitions, and instruments to measure how school contexts vary in ways that influence the success or failure of edtech in the classroom.

These variables include:

• Vision for teaching and learning
• Selection processes
• Teacher agency
• Infrastructure and operations
• Implementation systems and processes
• Staff culture
• Teacher beliefs and knowledge
• Strategic leadership support
• Professional learning
• Competing priorities

Moving forward, the EdTech Evidence Exchange reports it is launching a massive effort to incentivize and collect implementation data from hundreds of thousands of educators nationwide trained to use the new tools to describe, measure, and document their experiences with various, specific technologies.

Upon data reaching critical mass, it will be leveraged to identify, scale, and implement best practices for educators based on the experiences of those working in similar schools and districts.

Tomasik spoke with OnlineEducation further on the findings of the report and what this means for students, teachers, and stakeholders in education.

OnlineEducation.com: Which findings outlined by the report did you find the most striking? Why?

Tomasik: In many ways, the most striking findings are the ones that led to the creation of the EdTech Genome Context Framework and Inventory in the first place.

An analysis we conducted last year found that the U.S. spends between $26 billion and $41 billion annually on education technology—with almost no information about what tools actually work, where, or why. Unfortunately, the vast majority of that edtech is underused, inequitably used, or ineffectively used.

This should be a wake-up call for policymakers, education leaders, and advocates throughout the country: unless somebody takes action to help answer these questions about what technology works in which contexts, we’ll continue spending money on tools that don’t help students as much as they should.

OnlineEducation.com: In what ways are the educational community and government stakeholders planning to use the study’s findings to improve existing programs, create new opportunities, or address issues in their operations?

Tomasik: That’s the part of the work that we are just beginning now, and we’re very excited about it! In the coming months, we’ll be launching our first data collection efforts with educators in a number of states (to be announced this fall).

Using the Context Inventory from the Genome Project and other tools we’ve developed, teachers and leaders will document their experiences with edtech in specific contexts. We are building a national repository of evidence about which tools work in which conditions.

Over the coming months and years, we’ll continue to expand that work, gathering data from teachers and school leaders around the country to help other educators working in similar school and district contexts make more informed edtech decisions.

We are also working with a number of organizations to plan for ways in which the Context Inventory can support their efforts. For example, organizations that support teachers’ use of technology through professional development can tailor learning opportunities to a specific context, customizing support based on variables such as teachers’ knowledge about technology, existing leadership support, and competing priorities.

OnlineEducation.com: How does U.S. federal or state policy fall short at supporting and evaluating edtech’s efficacy in classrooms? What are some recommendations you would make to remedy the issue?

Tomasik: Great question. Research funding is probably the biggest area where we fall short as a country. Despite our schools spending between $26 billion and $41 billion annually on these tools, the U.S. Department of Education’s Institute of Education Sciences (IES) has a budget of only $0.6 billion per year. And as a result, according to its director, IES has properly studied approximately 0 percent of the 9,000 edtech tools on the market.

Our nation’s schools urgently need independent research and analysis to help them make better-informed decisions about which edtech tools to purchase (or not) and how to implement them.

Educators also want to learn from the insights and experiences of other educators working in contexts like theirs, so they can select and implement tools that are likely to be a good fit for their unique environment. This evidence is critical as a supplement to more general efficacy evidence, which we use to determine if a technology can positively impact student learning under ideal conditions.

Only the federal government can research the efficacy and implementation of edtech at the scale and scope that our educators need. That’s why it’s time to dramatically expand the focus and budget of IES, the Institute of Education Sciences, which oversees education research.

As Congress and the Biden Administration chart a path to recovery from the pandemic, more funding to research education technology must be a critical part of the equation. This research will provide the evidence edtech decision-makers desperately need to see a return on their investment in terms of student learning.

OnlineEducation.com: While there are many innovations coming out, what do you find to be the most exciting opportunity or innovation on the horizon for edtech? And what will be the main challenges to stakeholders in the space moving forward?

Tomasik: I think the most exciting opportunity is actually also the main challenge facing edtech stakeholders today.

After a massive rush to procure devices and provide Internet access to facilitate remote learning during the last 18 months, students have more access to technology than ever before. This influx in access is an incredible opportunity to close the digital divide and equitably support transformative student learning with technology.

But this influx in access is also an incredible challenge. We need to move beyond the first level (i.e., access) of the digital divide and focus on the second level: how technology is actually used to facilitate learning. It is not enough to purchase potentially innovative technologies; we need to effectively and equitably integrate them into learning experiences. This is going to be mission-critical if we hope to fulfill the promise of technology as a tool to help all students succeed.