Skip to content

Cheating Or Mastering?

August 21, 2012

An (un)intended consequence of on-line learning?


Sebastian Thrun is a co-founder of Udacity, one of the several new companies that are involved in on-line learning. Sebastian started teaching on-line while at Stanford, and now is doing this through Udacity. His bio on the Udacity website says:

At Stanford, he taught 200 students per class. He really wants to teach 200,000 students per class.

Today Ken and I want to comment on the current excitement about on-line learning, and perhaps lift the covers and see what is really happening.

When I (Dick) was visiting Berkeley a few days ago, the conversation at dinner turned to the recent growth of on-line courses. We discussed many issues but the main one was simple: several at the table claimed that while huge numbers were signing up for the course, few were actually finishing them.

As you might imagine this was used as a way to argue that on-line courses were not effective. Yes we have plenty of people drop “normal” classroom classes, but nothing like the numbers that were claimed for online courses. In theory terms, if N signed up for a class, the apparent rate of completion was εN, where ε is about 1/20. Pretty low rate. No?

I recall comments by Sebastian that he made to me many months ago, when I visited his start-up. I believe the completion rate ε is much larger, which is good, but also that N is much smaller, which is bad. Let me explain.

Some Data

Mark Guzdial is a colleague at Georgia Tech who is an expert on many things including theory of education. Last April he wrote a a great post on his “Computing Education Blog” that discussed the drop rates:

Sebastian Thrun and Dave Evans of Udacity came to Georgia Tech this week, and talked about the completion of their CS101 course. 100,000 people signed up for the course, but that was just providing an email address—no cost, no commitment. 50,000 visited the site before the first assignment, and 30,000 completed the first assignment—one of those is probably a better measure of who was serious about taking the course. 10,000 completed the course. There are blog posts around from both completers and non-completers. 3,000 got a perfect score, which is great for Udacity and their business model. (Thanks to Dave who vetted these results for me.)

Note, for us that gives ε = 1/10. We need to use a symbol somewhere—after all this is a theory discussion. Alas there will be no deep proofs nor clever theorems today, but perhaps some insight.

The battle: N vs ε

The companies involved in on-line learning all want N, the number of signed up students, to be large. Since the courses currently are free, it is not surprising that the N‘s for all the offerings are huge. The companies all claim their total N is in the range of about one million, give or take a few hundred thousand students. These are large numbers. They impress. They impress the people that matter: decision makers, funders, and the rest of us. One million students—wow.

However, the actual number of students who complete the courses and get a “certificate of completion” is quite a bit smaller. That is, ε is very small, and is around 1/20 or so.

Of course 1/20 of a million is still a huge number of students: 50,000 is a lot of people to take your courses. Still very impressive. Right? Or is this a sign of a fundamental problem with these on-line courses?

I think it may be a sign of something else.

The Real Issue?

Let’s examine the course model from the perspective of students. Say Alice decides to sign up for one of the courses and of course Bob does too. Alice takes a quick look and decides she already knows the material and moves on to something else with no further action. But Bob is committed: he really wants to learn the material and really wants to get a good grade.

Here is what happens next. Bob signs up for the course multiple times: let’s call them Bob1, Bob2, Bob3, Bob4. Recall there is no cost to Bob for signing up multiple times—none. So why not sign up several times…

Bob’s insight is simple: he now can take the course multiple times and keep only the best grade. Say there is a graded exam. Bob1 takes the exam and gets a 70% on it. Not bad, but not great either. So Bob sees what he got wrong, sees what questions they threw at him. He studies some more, then takes the exam again as Bob2. Of course the exam is different, since all these on-line systems do some randomization. However, the exam covers the same material, so now Bob2 gets an 85% say.

Perhaps Bob is satisfied. But if he is really motivated he studies some more, retakes the exam, and now Bob3 gets 90%. You guessed right. He goes on and takes it one more time as Bob4 who—surprise—gets a perfect 100%.

You see the pattern. As the course goes on the extra Bobs are used to get Bob4 a very high score. Eventually at the very end all Bobs but Bob4 drop and bingo: ‘Bob’ gets a great score. Also many of the “students” do not finish the course, not just three Bobs but also Alice.


The companies, I believe, know this is happening. Well Thrun told me about it in person when I visited his company this winter. They also can track IP addresses and they can see what is going on with their students. Note, there is almost no cost to the companies. The exams are all auto-graded, so the extra “students” are ostensibly no problem. They need no extra chairs, nor TA’s. So who cares? It also inflates N so that looks impressive.

Note that Udacity has an honor code against the multiple-take practice:

The Class Sites are available to any User. However, access to the Online Courses is restricted to Attendees or Students that have a registered User Account. By registering, you agree that: you are registered for the course only once and will not set up multiple User Accounts …

And Coursera—see here—also does.

What Ken and I do not know is the rate of violating these clauses. The honor codes also provide against “hurting the results of others.” Should multiple takes of exams and assignments be construed as hurting others? We have found relatively little discussion of this online, even concerning recent articles on cheating and skepticism of online learning.


The flip side is the long-standing idea of mastery learning at all levels of education, whereby students do what is needed until they can demonstrate mastery of the content. Indeed the first discussion Ken found of the phantom-student problem in online courses was in slides titled “Using Mastering® in Hybrid or Online Courses” by Lourdes P. Norman-MacKay of Florida State College in Jacksonville. The slides also discuss other students signing up as “sacrificial lambs” and dropping at the end, which need not involve multiple accounts.

My colleague Rich DeMillo places the issue in this context (lightly edited):

We don’t know much for sure about learning, but one of the things that we do know is that “mastery teaching” of this kind is vastly superior to the normal classroom. Here is what Benjamin Bloom and his students established in his famous 1984 meta-study “The Two Sigma Problem”:

In a traditional classroom, you present material and then test to see whether it has been learned in a process that sifts for failure. As Roger Schank points out there are real problems with this approach, and if you wanted to determine how much had been learned with this kind of classroom you should test a year or so later, for example. On the other hand, any time you can permit a student to stay focused on material, testing and re-testing until mastery has been demonstrated, you can move everyone two sigmas on standardized assessments. It doesn’t seem to matter what the subject matter is or what limitations the students might have. It makes everyone a better learner.

That seems to me to be the great value of Bob’s strategy, and it is a wonderful thing that MOOCs have made it economically feasible to do it.

The “problem” referred by Bloom’s paper is schools not having the resources to avail themselves of mastery techniques. These resources include time—in classrooms. Some standard courses have “mastery units” where this kind of teaching is targeted, but this also recognizes that they cannot be followed all the time. Computational complexity of education may be a subject for a future post.

What Ken and I find interesting is that many students are apparently motivated to do so well. Is this a better way to learn, or is it cheating? Should we allow students in “normal” classes to take the exam multiple times or not? We never thought about this previously because we could not afford to have students take more exams, since we had one grade to give by a human grader. But if all exams are auto-graded, then there is no cost.

Open Problems

Is this a better way to learn? Should we encourage it? I suggested that we make Bob’s strategy explicit: students do not have to have multiple “phony” sign ups; they can have official multiple tries at the exams. Is this a good idea? Is 30% perfection among course-completers a goal? What do you think?

Update: Sebastian Thrun wrote to say that in the intervening months since the comments reported in the post, Udacity has started creating in-person testing centers for official creditation, while the online exams are regarded as “practice” insofar as not counting toward a certificate of value.

31 Comments leave one →
  1. August 21, 2012 11:56 am

    Fascinating! I hadn’t heard about the multiple accounts form of cheating. Dave Patterson at Berkeley also taught a MOOC, and he reported on widespread cheating at an ACM Education Council meeting in June 2012 (quoting from my post at He was the first MOOC teacher I’ve heard admit to “unbounded, worldwide cheating.” They were going to use plagiarism detection software, just to see how much cheating was going on, but they didn’t need to. Large numbers of answers were “bit identical.”

  2. August 21, 2012 11:56 am

    If you were to turn all of this on its head, then you might start with the premise that it is very important is to disseminate knowledge to as many people as possible. To me that always seemed the goal of the original Greek ‘schools’. Somewhere along the way it got turned upside down and university degrees became all about proof of individual merit. That is, you’re not fit for a job, unless you have degree X.

    If what we’re interested in is a smarter more educated world, so that things function better, than does it really matter how people go about learning? So long as they really do learn. What we’d like to avoid is people who claim knowledge that they don’t have, that causes a lot of trouble these days. I’d rather see people take the same course multiple times, then people rush through a course and forget what they learned a week later.


  3. John Sidles permalink
    August 21, 2012 12:00 pm

    This particular Gödel’s Lost Letter essay is outstanding IMHO!

    Q  After all, how many times does it happen, that a researcher publishes article-after-article upon a given subject, until finally (after years or even decades) publish a review article and/or book that exhibits mastery?

    A  Practice-until-mastery is so common in research, as to be nearly universal.

    Moreover, medical school residents learn to be master physicians by similar iterative practice (yikes!) … no other method is known to work.

    Conclusion  The lower on the educational “stack” that practice-of-mastery is embraced, the better the outcome for everyone!   🙂

    Congratulations again upon a *terrific* Gödel’s Lost Letter essay.

  4. henkari permalink
    August 21, 2012 1:00 pm

    Do they already have one “fake” exam which can be taken multiple times but gives you a good approximation of your real grade, and a “real” exam with completely different set of problems which can only be taken once?

  5. August 21, 2012 1:30 pm

    So long as they can’t stop them, why don’t they let them do it officially.

  6. August 21, 2012 2:02 pm

    random 8042,

    One of the reasons is that these courses rely a lot on multiple choice questions, so it is possible to get a good score by systematically trying all the options rather than mastering the material. Although there is some randomization of questions, it is not nearly enough to eliminate this problem.

    A obvious solution would be to have a much wider pool of questions for each assignment/exam, as is the case with Kahn Academy math tests for example. I don’t know how practical this is because it already takes long time to prepare the material for one of these courses in their existing form. A better solution would be to include more quantitative questions, more essay-based questions, and to find a way of including participation in online discussions in the grade. Of course, these solutions require a method for automatically grading these things, or they require a much larger teaching staff for the course. Ultimately though, you are not going to achieve parity with university courses unless you find a way of doing these things.

  7. Marco permalink
    August 21, 2012 9:23 pm

    Indeed, in my university in Italy students can try the same exam up to 5 times a year, for as many year they wants.
    We encourage this kind of “mastering” even saying explicitly something like
    “Even if you do not feel prepared, go to the exam and take it as an exercise”

    Some students even do not accept anything less then max grade for the most important first year courses, because they know they can try the exam 5 times a year, three years in a row.
    This is in conjunction with a pass rate of the exams that is sometimes as low as 15%.

  8. August 22, 2012 5:26 am

    It depends on the goal IMHO. In the (limited resources/high responsibility) world one needs to select the scarce resources (like Scott Aaronson used to talk about himself) by the one-time exam (for saving life in favor of society, or to keep the power in favor of influence groups). For the rest, the mastering seem to be reasonable – the world needs it agents to reach their goal. Again, in the knowledge world there are two needs – the common academic knowledge – breath first kind of approach, and the industrial one where one needs specific in depth knowledge to archive specific goal. Both of them seem to require mastering.

    The problem with multiple exams can be in the mechanical memorizing answers on the same questions, or steeling the answers from others. This problems can be solved by the the large corpus of questions with a selection of individual questions per learner. BTW, for the basic academic knowledge it can be filled in the open source fashion, with many authors for the same topic. The limit of this process is the network (acyclic directed graph) of perceptual units of knowledge, surrounded by the problem sets, with everyone learning in their own pace with the stackoverflow like sites for the clarification of and discussion on the topic. This also provides the feedback for the course material and the problem sets for the authors. With real money work assigned by the employers for the specific set of knowledge with HR commission if one wants the open source business model.

    Finally, IMHO it should be open source with the both the ability for the public to contribute and the minimal cost for the learners. The technical revolution on the boundary of XIX-XX was partially due to the mass education. BTW the scientific publications have similar goal of providing knowledge, and can be embedded into the line of courses for the self-education, with the amount of surrounded clarification, discussion and problem sets depending of the public interest, e.g. being the objective measure of the importance of the solution to the problem.

  9. August 22, 2012 9:18 am

    In thinking about it a bit more I know it would be really great for me to revisit some of the courses I took twenty years ago and have a metric for comparing how I did then vs. how I am doing now. Did my understanding improve? Or did I just forget a whole lot of stuff.

    If the educators could also look at that data, they’d be able to see what information was valuable to me in the industry and what was ignored. With that info, that could tailor the courses to be more effective (but I believe they should still lean heavily on teaching more theory than practice, since particularly in IT, understanding the theory makes it far easier to adopt to new practices).

    If people want to take the course multiple times, I think they should encourage it and keep their data around so that people can revisit as often as needed. It helps both parties.


  10. Bill Gasarch permalink
    August 22, 2012 9:29 am

    Mastery seems like a good goal and I see nothing wrong with a student taking a course multiple times, officially or unofficially. I think its good that students want to acheive mastery.
    But there is a downside that one might call unintentional cheating:

    If students see the same type of questions they may know the answer in a pattern matching way rather than in a real way. Like the chinese room paradox- does the person really know chinese. Or, more real for us- we teach students to proof that
    sqrt(2) is irrational and they memorize the template for the proof and can in a rote
    way produce proofs that sqrt(3), sqrt(4), sqrt(5), etc are all irrational, without a real understanding of these proofs. (Note- I know that sqrt(4) is rational, but that doesn’t stop
    students from proving its irrational.)

    So Mastery might end up being memorization without understanding. The student has no ill intent here and may not realize his or her lack of understanding.

    • August 22, 2012 9:38 am

      I agree some people will just remember stuff long enough to pass, but when I was in university I saw a lot of people like that as well. One of my friends had photographic memory, in the early years he had amazing marks. Towards the end he was really struggling (his memory was short-term, so it faded fast). I knew a lot of students that thought what they were teaching us was a waste of time, and all they wanted was a paper and to get out there and earn money. I think that no matter what you do there will always be people that feel that way (and often seem to rise into management 🙂


    • August 22, 2012 9:49 am

      Bill, that is why you need a lot of questions in increasing complexity, which is hard to archive in physical class model, but much easier to get online. Giving the wrong prove for sqrt(4) is the key for mastering – an expert is a novice that made all kind of mistakes. And that is the point of having feedback for the course and the problem sets. And by the way some people do not need proof machinery, but still may need the classification results. To prevent the memorization a different set of problems have to probe it in a different way – a unique personal way. And if it is archived, than the knowledge is not sitting in Prof. heads, but is abstracted away as the service for indefinite population, even for those who have no contact with Prof’s head. (Or this is too revolutionary?)

  11. Serge permalink
    August 22, 2012 10:04 am

    If everyone knows that everyone cheats at on-line exams, then what does getting an on-line diploma really mean? It’s probably a good learning tool but also a poor way to assess your capacities. A bit like the Tour de France in which all the participants are more or less suspected of taking illegal products… In any case, I bet most recruiters prefer to hire the students from conventional universities.

    Regarding multiple user accounts, they’re not only a problem for on-line universities. In fact it’s a generalized issue all over the Internet, especially in social networks. A guy such as Mark Zuckerberg still doesn’t know how do handle this properly…

  12. August 22, 2012 10:32 am

    My mom has used this system for homework assignments for years in her teaching architecture at community college. If an assignment isn’t good enough, she tells the student what’s wrong, and has them redo it until it’s fine. One implication is that you can set the “pass” bar close to where we currently set the “A” bar. It seems to work really well for assignments where students are not just answering questions, but are creating something, like a drawing or an essay.

  13. August 23, 2012 8:04 am

    Great article. I didn’t know about the multiple accounts problem and how it was affecting online course enrollment.

    I wonder if one of the reasons why (some) students drop online courses is because they dislike the independence it gives them. In a traditional classroom, the professor will often spend a long time rehashing the expectations of the course, the goals, the marking rubrics, etc. In an online course, this material is often posted online and students are expected to familiarize themselves with the course requirements independently. For this reason, I know at least a handful of fellow students in an online course I recently took who complained about the “lack of direction” and the inability of the instructor to “take charge” of the material.

    Online courses aren’t for everyone, but if you’re the type of person who likes to self-direct your own learning, then I think they can be a really enjoyable experience.

  14. JLPC permalink
    August 23, 2012 12:34 pm

    A computer scientist wrote in a business brochure in the 80’s that his programs understood everyday English just like a person does (vd. Winograd & Flores, “Understanding Computing and Cognition”, p. 128). Guess who? Roger Schank. Since then I feel skeptical about every claim he makes :-).

    More seriously, this post is very interesting and points out a real problem with on-line learning: that of automated, somehow intelligent, mass testing. It is technologically achievable (see for example the ASSISTments system), but a lot of time and skill is needed in order to make it educationally sound, gaming-resistant and cheating-resistant.

  15. August 23, 2012 12:45 pm

    Interesting — it seems to codify one of the most common ways of achieving good scores on most standardized tests, to simply take many, many old tests. People do that with the SATs and GREs, and it was standard strategy for passing one’s PhD qualifying examinations when I was in grad school. I did it as well. Get the past ten years of quals, and spend all summer taking them. This was in physics — not a field known for the usefulness of rote learning.

    If you want to do something well, you practice DOING IT. That goes for test-taking as much as for anything else.

  16. August 23, 2012 1:06 pm

    While I agree completely with process oriented learning, in the case of multiple choice exams this seems like an extension of No Child Left Behind test taking skills (i.e. learn how to pass the test, not the information on it). If we’re talking about writing and revising drafts of research papers or essay questions, I think process oriented learning is great. But of course, as someone mentioned above, these kinds of things can’t be graded by a computer.

  17. August 23, 2012 5:34 pm

    “Bob’s insight is simple: he now can take the course multiple times and keep only the best grade.”

    Having recently surveyed Thrun’s Statistics 101 course, I can say that this is irrelevant for his UDacity offerings. All assessments, including the final exam, allow each individual question to be re-submitted as many times as you want, and re-graded, automatically. This includes the several questions on the final that were multiple-choice, for example.

    And aside from that, I’d say that the final was otherwise jarringly easy. Which feeds into a concern that without any accreditation or outside quality-control, projects like this are now incentivized to make assessments as easy as possible to show high completion rates (since that’s now on the mind-sphere radar).

    • August 23, 2012 10:32 pm

      Your comment about the design of the multiple-choice questions gives me a hook for a little self-advertisement. Does anyone design them with the idea that in place of clearly-wrong answers, one gives plausible but inferior answers as the alternative. Each answer i has a value e-d_i where e is the value of the best answer, and d_i is an instructor-designed measure of inferiority for answer i (d_j = 0 for the best answer j).

      If this is a good design methodology, then it’s isomorphic to my “Intrinsic Chess Ratings” model (paper1, paper2), and the math there kicks in to provide a rigorous foundation for assigning grades.

      • JLPC permalink
        August 24, 2012 1:46 am

        Dear professor Regan, there is a whole respectable field of research devoted to these issues, namely psychometrics; and in some well-known psychometric models (IRT) every answer i to every question j is defined by its ICC_ij(\theta),

        ICC_ij(\theta) = p(answer to j is i | student’s knowledge level = \theta)

        ICC’s are not necessarily increasing with \theta (can be always decreasing; decreasing from a certain level \theta_1; …)

        The real challenge is to find each ICC: you need a lot of data to make a more or less reliable estimate.

  18. e weaver permalink
    August 23, 2012 9:15 pm

    Online learning is a wonderful option for people in remote areas or places where what they want to learn is unavailable or where they are unable to attend an institution for a variety of reasons. The thirst for knowledge remains the most important characteristic to support, not how long a person takes to integrate information. I used to teach college courses and find the concept of “cheating” inaccurate when measuring the speed with which one not only understands but can apply concepts. Whether working in collaboration or applying the tenacity to study until one “masters” a lesson isn’t cheating. Cheating is passing off another’s work as one’s own with little if any understanding of the information or communication of that information. Multiple attempts is commendable. I wouldn’t want an online trained surgeon, but wouldn’t care if my web designer was online trained. Kinetic intelligence might not thrive with technology but conceptual, linguistic, numeric among other intelligences can. (Please read Howard Gardner’s FRAMES OF MIND if various “intelligences” makes no sense…here’s a wiki entry about it:

  19. chessweb permalink
    September 10, 2012 4:01 pm

    Well, the goal of the course (and all online courses) is to educate Bob on some topic or the other. Bob couldn’t accomplish the final exam the first time, so he studied and eventually “Bob to the nth” finally got it right. So what exactly is the problem?

  20. October 22, 2012 6:55 am

    I will right away take hold of your rss feed as I can’t find your email subscription link or newsletter service. Do you have any? Please permit me realize in order that I could subscribe. Thanks.


  1. Are MOOC Students Cheating Or Mastering the Material? « Gödel’s Lost Letter and P=NP « Computing Education Blog
  2. A Ler « A Educação do meu Umbigo
  3. Cheating Or Mastering? | The Title Borrower
  4. » Online education evolves: testing centers vs. multiple exam retakes. Gordon's shares
  5. High School Curriculum a Waste of Time? « Pressing About It
  6. Cantor’s Theorem: The Movie « Gödel’s Lost Letter and P=NP
  7. D Plus O Equals Do | Gödel's Lost Letter and P=NP

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s