Cheating Or Mastering?

An (un)intended consequence of on-line learning?

src

Sebastian Thrun is a co-founder of Udacity, one of the several new companies that are involved in on-line learning. Sebastian started teaching on-line while at Stanford, and now is doing this through Udacity. His bio on the Udacity website says:

At Stanford, he taught 200 students per class. He really wants to teach 200,000 students per class.

Today Ken and I want to comment on the current excitement about on-line learning, and perhaps lift the covers and see what is really happening.

When I (Dick) was visiting Berkeley a few days ago, the conversation at dinner turned to the recent growth of on-line courses. We discussed many issues but the main one was simple: several at the table claimed that while huge numbers were signing up for the course, few were actually finishing them.

As you might imagine this was used as a way to argue that on-line courses were not effective. Yes we have plenty of people drop “normal” classroom classes, but nothing like the numbers that were claimed for online courses. In theory terms, if N signed up for a class, the apparent rate of completion was εN, where ε is about 1/20. Pretty low rate. No?

I recall comments by Sebastian that he made to me many months ago, when I visited his start-up. I believe the completion rate ε is much larger, which is good, but also that N is much smaller, which is bad. Let me explain.

Some Data

Mark Guzdial is a colleague at Georgia Tech who is an expert on many things including theory of education. Last April he wrote a a great post on his “Computing Education Blog” that discussed the drop rates:

Sebastian Thrun and Dave Evans of Udacity came to Georgia Tech this week, and talked about the completion of their CS101 course. 100,000 people signed up for the course, but that was just providing an email address—no cost, no commitment. 50,000 visited the site before the first assignment, and 30,000 completed the first assignment—one of those is probably a better measure of who was serious about taking the course. 10,000 completed the course. There are blog posts around from both completers and non-completers. 3,000 got a perfect score, which is great for Udacity and their business model. (Thanks to Dave who vetted these results for me.)

Note, for us that gives ε = 1/10. We need to use a symbol somewhere—after all this is a theory discussion. Alas there will be no deep proofs nor clever theorems today, but perhaps some insight.

The battle: N vs ε

The companies involved in on-line learning all want N, the number of signed up students, to be large. Since the courses currently are free, it is not surprising that the N‘s for all the offerings are huge. The companies all claim their total N is in the range of about one million, give or take a few hundred thousand students. These are large numbers. They impress. They impress the people that matter: decision makers, funders, and the rest of us. One million students—wow.

However, the actual number of students who complete the courses and get a “certificate of completion” is quite a bit smaller. That is, ε is very small, and is around 1/20 or so.

Of course 1/20 of a million is still a huge number of students: 50,000 is a lot of people to take your courses. Still very impressive. Right? Or is this a sign of a fundamental problem with these on-line courses?

I think it may be a sign of something else.

The Real Issue?

Let’s examine the course model from the perspective of students. Say Alice decides to sign up for one of the courses and of course Bob does too. Alice takes a quick look and decides she already knows the material and moves on to something else with no further action. But Bob is committed: he really wants to learn the material and really wants to get a good grade.

Here is what happens next. Bob signs up for the course multiple times: let’s call them Bob1, Bob2, Bob3, Bob4. Recall there is no cost to Bob for signing up multiple times—none. So why not sign up several times…

Bob’s insight is simple: he now can take the course multiple times and keep only the best grade. Say there is a graded exam. Bob1 takes the exam and gets a 70% on it. Not bad, but not great either. So Bob sees what he got wrong, sees what questions they threw at him. He studies some more, then takes the exam again as Bob2. Of course the exam is different, since all these on-line systems do some randomization. However, the exam covers the same material, so now Bob2 gets an 85% say.

Perhaps Bob is satisfied. But if he is really motivated he studies some more, retakes the exam, and now Bob3 gets 90%. You guessed right. He goes on and takes it one more time as Bob4 who—surprise—gets a perfect 100%.

You see the pattern. As the course goes on the extra Bobs are used to get Bob4 a very high score. Eventually at the very end all Bobs but Bob4 drop and bingo: ‘Bob’ gets a great score. Also many of the “students” do not finish the course, not just three Bobs but also Alice.

Fair?

The companies, I believe, know this is happening. Well Thrun told me about it in person when I visited his company this winter. They also can track IP addresses and they can see what is going on with their students. Note, there is almost no cost to the companies. The exams are all auto-graded, so the extra “students” are ostensibly no problem. They need no extra chairs, nor TA’s. So who cares? It also inflates N so that looks impressive.

Note that Udacity has an honor code against the multiple-take practice:

The Class Sites are available to any User. However, access to the Online Courses is restricted to Attendees or Students that have a registered User Account. By registering, you agree that: you are registered for the course only once and will not set up multiple User Accounts …

And Coursera—see here—also does.

What Ken and I do not know is the rate of violating these clauses. The honor codes also provide against “hurting the results of others.” Should multiple takes of exams and assignments be construed as hurting others? We have found relatively little discussion of this online, even concerning recent articles on cheating and skepticism of online learning.

Mastering

The flip side is the long-standing idea of mastery learning at all levels of education, whereby students do what is needed until they can demonstrate mastery of the content. Indeed the first discussion Ken found of the phantom-student problem in online courses was in slides titled “Using Mastering® in Hybrid or Online Courses” by Lourdes P. Norman-MacKay of Florida State College in Jacksonville. The slides also discuss other students signing up as “sacrificial lambs” and dropping at the end, which need not involve multiple accounts.

My colleague Rich DeMillo places the issue in this context (lightly edited):

We don’t know much for sure about learning, but one of the things that we do know is that “mastery teaching” of this kind is vastly superior to the normal classroom. Here is what Benjamin Bloom and his students established in his famous 1984 meta-study “The Two Sigma Problem”:

In a traditional classroom, you present material and then test to see whether it has been learned in a process that sifts for failure. As Roger Schank points out there are real problems with this approach, and if you wanted to determine how much had been learned with this kind of classroom you should test a year or so later, for example. On the other hand, any time you can permit a student to stay focused on material, testing and re-testing until mastery has been demonstrated, you can move everyone two sigmas on standardized assessments. It doesn’t seem to matter what the subject matter is or what limitations the students might have. It makes everyone a better learner.

That seems to me to be the great value of Bob’s strategy, and it is a wonderful thing that MOOCs have made it economically feasible to do it.

The “problem” referred by Bloom’s paper is schools not having the resources to avail themselves of mastery techniques. These resources include time—in classrooms. Some standard courses have “mastery units” where this kind of teaching is targeted, but this also recognizes that they cannot be followed all the time. Computational complexity of education may be a subject for a future post.

What Ken and I find interesting is that many students are apparently motivated to do so well. Is this a better way to learn, or is it cheating? Should we allow students in “normal” classes to take the exam multiple times or not? We never thought about this previously because we could not afford to have students take more exams, since we had one grade to give by a human grader. But if all exams are auto-graded, then there is no cost.

Open Problems

Is this a better way to learn? Should we encourage it? I suggested that we make Bob’s strategy explicit: students do not have to have multiple “phony” sign ups; they can have official multiple tries at the exams. Is this a good idea? Is 30% perfection among course-completers a goal? What do you think?

Update: Sebastian Thrun wrote to say that in the intervening months since the comments reported in the post, Udacity has started creating in-person testing centers for official creditation, while the online exams are regarded as “practice” insofar as not counting toward a certificate of value.