Online Tutoring: What Works and What Doesn’t

Pandemic learning loss among K-12 students turned into a hot political issue before the 2022 election, driven by shocking evidence of suffering by students. Three studies, in particular, contributed to that evidence.

As early as July 2021, a McKinsey & Company report found that the average United States K-12 student had fallen four months behind in reading and five months behind in math. But those were the average declines; students of color and those from low-income areas performed even worse.

The following December, a study led by the Center for Education Policy Research at Harvard University found that pandemic learning loss could cost American students $2 trillion in lifetime earnings. That’s almost a $45,000 loss in average lifetime earnings for each student.

Then, only nine weeks before the election, the National Assessment of Educational Progress announced that nine-year-olds showed their greatest plummet in reading scores in 32 years, along with their first math score drop since the test’s inaugural assessment 53 years ago in 1969. Although all income levels and races showed declines, the math scores of the students in the bottom 10th percentile fell by 12 points compared to 2020 results, wiping out two decades of progress through score gains.

As a result, policymakers committed unprecedented and staggering amounts of resources to address voters’ concerns about the problem. At the federal level, the Biden Administration’s American Rescue Plan will spend $122 billion on school districts, and the legislation mandates schools must spend 20 percent to address learning loss before the funding permanently expires in September 2024. Phyllis Jordan, the associate director of an education policy think tank that calls itself FutureEd, told the New York Times Magazine, “this is the biggest one-time infusion of federal dollars ever to come to schools. It’s just an astounding amount of money.”

To address learning loss, school districts have typically focused on three approaches. They include summer sessions, after-school programs that extend the school day, and intensive tutoring.

After analyzing the plans of some 5,400 districts, an education finance analysis firm known as Burbio concluded that about a quarter of the districts were planning on tutoring, budgeting roughly $1.4 million each on average. That is truly an astonishing aggregate budget, amounting to $7.56 billion for tutoring nationwide between now and 2024.

But the most effective tutoring programs come with a broad range of challenges. The research shows the strongest evidence for academic gains results from a specific method that’s usually called “high-impact” or “high-dosage” tutoring. These programs aren’t merely informal homework help. Such proven-effective tutoring sessions meet frequently—at minimum three times every week—and are highly structured, led by well-trained tutors following precise lesson plans that complement classroom assignments. Moreover, these programs usually require attendance, schedule sessions during school hours instead of after school or at night, and award academic credit toward graduation.

Unfortunately, these programs have a reputation as expensive, resource-intensive, and difficult options. Individual instruction is pedagogically very different from classroom teaching, so schools typically need to invest in hiring tutors with special training. The programs need extra classroom space during the school day—not idle classrooms at night—and they also need budgets for overhead like lighting, heating, and air conditioning. Plus, the schools have to rearrange the schedules of students and teachers so that time becomes available.

Other things equal, the leadership of most school districts would much rather pay for an online tutoring service that students can use away from their schools.

Meanwhile, university researchers learned in October 2022 about some of the practices that will result in poor student engagement with online tutoring. That’s when a team from Brown University and the University of California at Irvine investigated the effectiveness of the tutoring program at California’s Aspire Public Schools.

Oakland-based Aspire manages a system of 36 college-prep charter schools across the Los Angeles, San Francisco Bay Area, and Metropolitan Sacramento regions. With a 2022 enrollment of about 15,200, the taxpayer-supported but privately-operated system predominantly serves junior high and high school students of color from low-income communities. About 80 percent of Aspire’s graduates go on to college.

In February 2021, Aspire announced that it would provide “unlimited virtual tutoring” to all its secondary students through a partnership with Paper Education, Inc. Paper is a Montreal-based software startup backed by $100 million in venture funding. The lead investor in Paper’s latest funding round is Institutional Venture Partners, one of Silicon Valley’s oldest Sand Hill Road venture capital firms.

IVP is famous for launching Twitter, Netflix, Business Insider, Snap, Slack, Robinhood, and Coinbase. Most recently, IVP backed FTX Trading Ltd., the embattled Bahamas-based crypto derivatives exchange founded by high-profile entrepreneur Sam Bankman-Fried that filed for bankruptcy protection in November 2022. Venture firms like IVP funneled billions of dollars into FTX, pumping the exchange’s valuation up to $32 billion before its spectacular implosion; IVP’s rival Sequoia Capital was even forced to write down to zero its $200 million stake in FTX.

IVP is not the only venture capital firm investing in online tutoring startups. But the venture community has specific requirements for the kinds of online tutoring it wants to fund.

TechCrunch industry analyst Natasha Mascarenhas has written extensively on this trend. For example, she wrote in January 2021 that “the future of tutoring might not look like a 30-minute video on Zoom or Google Hangouts” and “some of the biggest decision-makers in edtech are taking a scalpel to the way that tutoring used to work and hope to scale faster by doing so.”

Six months later, she then wrote that Paper “embodies a lot of what companies like Quizlet, Course Hero, and Brainly have only recently invested in: the rise of smart, not slow, tutoring sessions. They’re all betting that modern-day extra help may not look like a Zoom, but instead a live chat with a whiteboard feature tacked on.”

However, the venture capital community has a problem. Its preconception of the kind of “scalable” online tutoring that deserves funding differs dramatically from the kinds of online tutoring that the research shows will actually engage students. And engagement is the necessary first step that all tutoring programs must achieve before the students in those programs earn better grades and higher standardized test scores.

In the Aspire trial, Paper Education’s software platform enabled students to log in free of charge at any time around the clock, seven days a week, and a live tutor—typically a recent college graduate—would help them with their schoolwork in any of 200 subjects.

But keep in mind a key detail that Paper and competing startups downplay: these are never live conversations over video. In fact, the student can’t even talk live with their tutor over an audio call. Paper’s system is exclusively limited to only three communication options: typed text chat, work together between the tutor and the student on a virtual whiteboard, and document sharing. There’s no spoken conversation between the tutor and the student through calls over video or audio.

Engagement with Paper’s tutoring system among Aspire students was terrible. More than 70 percent of the students didn’t even try it. Of the 7,000 students provided with access to the system, only 26 actually used it at least three times a week, as the High Impact research recommends. Few of the students who actually tried out the system attempted more than four sessions during the entire term studied by the researchers.

Even when they sent email and text “sales” messages to both students and their parents, the best that the researchers could do was persuade only 27 percent of the students to try Paper’s online tutoring. That did amount to a 42 percent increase over the mere 19 percent who tried the service without such message prompting. But even among the students who received the emails and texts and knew they had access to the platform, two-fifths still failed one or more courses.

However, what’s especially discouraging about the researchers’ findings is that, from an educational equity standpoint, Paper’s online tutoring program yielded results that were the opposite of an equitable outcome. Only 12 percent of the failing “DFW” students—those who had earned either a D or an F as letter grades or withdrew from a course—ever tried the system, even though these were the students that Aspire had bought Paper’s service to help. But the students with good grades who didn’t need Paper’s tutoring were twice as likely to take advantage of it.

Does the Aspire trial prove that online tutoring cannot engage students? No, it doesn’t, because two other online tutoring studies reported much more effective results. But the programs evaluated by the favorable studies include tutoring features that are unavailable from an opt-in, text chat platform like that provided by Paper.

The first of these studies was conducted by researchers from the London School of Economics and the Esade Centre for Economic Policy—a research institute within the Esade Business & Law School in Madrid. These researchers evaluated the benefits for secondary students from extremely disadvantaged neighborhoods in Spain of three 50-minute tutoring sessions per week. These 100 percent online sessions primarily focused on math and related social and emotional support topics like motivation and work routines.

This intensive eight-week program took place after school, and students earned credit from it toward graduation. In these sessions, a qualified math teacher covered planned topics over a video connection with groups of two students who shared a personal computer or a tablet.

The research team published their results in a paper entitled “Online Tutoring Works: Experimental Evidence From a Program With Vulnerable Children.” The researchers concluded that the tutoring sessions led to end-of-year grades that improved by nearly half a standard deviation—roughly equivalent on average to the best in-person tutoring—and standardized test scores that were a quarter of a standard deviation higher. The team also found that teachers were 78 percent less likely to require students who had completed the tutoring program to repeat a grade level.

The second study was led by researchers from Harvard University and Bocconi University in Milan. They studied a five-week program called the Tutoring Online Program, or TOP. It provided free individual 100 percent online tutoring to 530 economically disadvantaged Italian middle school students, some from immigrant families. The subject matter was predominantly math, although tutors also taught selected Italian and English language arts topics.

The tutors were university student volunteers who were provided special pedagogical training by a team of experts. Each tutor worked with their randomly-assigned student over online video calls for three to six hours each week. Each student participated via a personal computer, tablet, or smartphone in 20 percent of the sample.

This program also substantially boosted students’ academic performance. As with the Spanish study, standardized test performance among these teenagers at the end of the program improved by almost a quarter of a standard deviation. However, in this case, the improvements were strongest for the students from the lowest socioeconomic status groups, along with those suffering from learning disabilities such as dyslexia.

The TOP researchers also evaluated measures of the students’ psychological well-being, and found that at the program’s end, the students were happier and less depressed. The sample’s score on a psychological wellness index increased by almost one-fifth of a standard deviation.

If all these phenomenal results weren’t impressive enough, the TOP researchers also leaked a surprising detail. Even though many argue that effective live video tutoring programs can’t be offered economically at scale, the investigators estimated that their program only cost about 50 euros per student.

At the education civil rights group Education Trust, Vice President Allison Socol has criticized Paper. In November 2022, she told Chalkbeat:

“It’s not necessarily the virtual part of it. Online homework help puts the responsibility on the student to say: ‘I don’t understand this individual question on my homework, let me reach out to a potentially random adult who I don’t have a relationship with.’”

In both the Spanish and Italian studies, each student was assigned their own tutor who served a greater purpose than only teaching math skills. Through online video, those students also developed a personal, supportive, one-on-one relationship with that tutor. That relationship developed through regularly scheduled meetings comprising 24 hours of work together in the Spanish study, and in the Italian study, either 15 or as many as 30 hours.

But developing a one-on-one relationship is impossible through a platform like Paper. Every impromptu opt-in session a student launches can pair that student with another new tutor, another new stranger with whom they don’t have a relationship.

And they’re not interacting over video through speech, but merely through typed text chatting. Since the tutor can’t see the student, it is even more difficult for the tutor to develop rapport with the student.

What’s more, the tutor can’t even tell if the student seems puzzled by the student’s appearance, nonverbal gestures, or vocal tone. That renders the tutor dependent on the student to explain why they’re confused—which many students don’t understand or can’t adequately express through typing.

Like Socol, Dr. Susanna Loeb, one of the Aspire study’s principal investigators and an education professor at Brown University, similarly had problems with Paper’s approach. She also told Chalkbeat:

“If you expect them to bring their questions to the tutoring, that’s very difficult, too, because many students don’t quite know what they understand or don’t. As a strategy for supporting students in need, it’s not a good strategy.”

The Spanish and Italian studies conformed to the best practices recommended by the high-impact tutoring models. On its website, Paper argues that its particular tutoring brand somehow also conforms to those principles “at scale.” But that’s just not possible, since no regularly scheduled sessions or face-to-face interactions are possible with Paper’s opt-in, text chatting model.

Maybe some students can do well with Paper’s tutoring, but they’re not the at-risk DFW students districts buy the platform to serve. Moreover, at the time of this writing, the company has not yet released any research showing improved grades or test scores due to its involvement with DFW students.

In short, the Spanish and Italian studies show that tutoring with regularly scheduled sessions, qualified tutors, structured lesson plans, and live video connections embodies the kind of tutoring that works online. And when buying their tutoring services under the American Rescue Plan, school districts that care about their students shouldn’t settle for anything less.