Certainty Based Marking in Moodle

CBM increases the learning outcome of closed quiz questions in formative assessment

How can you get more out of a simple quiz? Studie Centrum Financiele Branche offers Permanent Education Courses for financial professionals. Their course concept consists of preparation in an online environment and a face-to-face meeting to discuss the learning material. Students have the opportunity to test their knowledge of the course syllabi by taking three online quizzes per subject. After that they send in their solutions to a case.

My client was keen on ways to increase the learning outcome of these quizzes. Serendipity: his father-in-law was asking him : ‘ Why don’t you require your students to state  how sure they are of the answer they are giving, so you you know that they are not simply guessing the answer ?’ I had just been to the UK MoodleMoot where Tim Hunt (Open University) presented the new Quiz module in Moodle 2.1. An interesting new feature of the quiz module is called Certainty Based Marking.“That’s what we need”, my client and I thought, so I started to experiment with it. But first, what is it?

What is CBM?

Certainty Based Marking (CBM) is an assessment system in which a student answers multiple choice, true false or short answer questions. After the student selects an answer he has to decide how sure he is of the correctness of his answer. He has to assert the probability of being correct: 1 = Not very (close to guessing), 2 = Reasonably sure or 3 = Absolutely sure*. In this example you see what it looks like in a Moodle quiz in the SCFB-course.

Klik om te vergroten

The grade of certainty that has been chosen has consequences for the score on the question, as you see in this table.

Cold shower

Trying this out, I found that the consequences for your total score at the end of the test can be pretty devastating. Say you make a test of 10 questions and each question has a value of 3, you can score 30 marks if you have everything right. If you choose certainty 3 two times and your answer is incorrect, you lose 12 marks. If you also score right two times while you haven chosen certainty = 1 you lose another 4. It gets pretty hard to pass the test, if you are not sure of what you do and what you don’t know.

By testing it we learned that the teachers of SCFB had some difficulty accepting their low total scores for situations in which they answered more than half of the questions correct. This is one of the reasons that students need practice to get used to CBM: a student has to get the opportunity to really learn to work with the method and it is not something you apply in an isolated test.

The guru and the programmer

While researching the CBM phenomenon I found the guru Tony Gardner-Medwin, Emeritus Professor, Department of Physiology (NPP), UCL, London. He has made the system or protocol (LAPT) available at UCL,  including availability for anyone interested in using it, and has published about it (see bottom article). He has also written code that can be imported into Moodle, for versions 1.9-2.04. We first used Tony’s LAPT-code, but as soon as Tim Hunt, the Quiz module programmer, incorporated CBM in the standard Moodle quiz features for 2.1 and higher, we switched to using this standard CBM-option. Hunt based his code partly on Tony’s work but not completely.

This might look as a detail for testing nerds, but the difference is pretty important. Tony’s code produces general feedback for the test with the percentage of questions you answered right AND your CBM score. Tims code only keeps the CBM score and pushes this to Moodles gradebook. For the maximum impact on learning outcome a student should be able to see his end score and compare this to his CBM-score, as in LAPT.

A talk with Tony

Links Silvester, rechts Tony

CBM gave lots of food for thought on testing and scoring. Luckily Tony Gardner-Medwin visited the Netherlands in September. During my research I came in contact with Silvester Draaijer (Onderwijscentrum VU). Silvester also wanted to meet Tony, so we had an interview with him.

We talked about the philosophy behind and the advantages of CBM compared to plain old mc or T/F-questions.

Philosophy behind and advantages of CBM

You could say CBM repairs a flaw of standard testing with closed questions. The results on a standard test do not reveal (in Tony’s view) what the student really knows. The student may give a correct answer, but was it by guessing or was it on the basis of solid knowledge? A student who is a very lucky guesser can pass a test while a student who omits uncertain answers, can fail it. CBM rewards students who know which answers they do know and who acknowledge when they are uncertain. It ‘punishes’ students who are never confident, or who claim confidence for incorrect answers.

For students a test with CBM is much more informative and promotes deeper learning. The ‘How sure are you’ question prompts him to think more deeply about the answer, or, if the test is not time limited, to look up the correct answer. After he submits the answers, he can see if his certainty estimation was accurate. At the end of the test he knows which parts of the material he has to study more in depth and which parts are completely between his ears. Hopefully he also sees feedback with an explanation of the correct answer. In this way the learning yield is much higher.

In short, the students focus on:

  • getting as many answers right as possible, and
  • accurately judging certainty to achieve a high CBM-score

 You need to know when you’re right!

Tony explained how CBM plays an important role in the development of a professional attitude. He taught physiology in medical school. It’s pretty crucial that medical staff can make a difference between ‘I am sure’ and ‘I have doubts’. CBM can teach them to act on their doubts (ask a colleague, grab a textbook) in a very simple way: if I am not sure and also right I loose points! This is also what appealed to my client. Financial advisors give maybe 10 advices a day. Nine times they are right but the tenth advice may lead a client to bankruptcy. In the real world advisors have to be absolutely sure 10 times out of 10. The CBM test is a simulation of their professional environment.

I believe in CBM

Just like Tony, I think that only ‘being sure’ is true knowledge. I am also in favour of it because it stimulates students to think just a little bit deeper. Only this extra cognitive activity should already heighten the learning effect of taking a test.

But, is correction of guessing necessary?

Now it is time for some expert criticism. Silvester has some more knowledge of educational measurement. For him, using CBM is not so obvious as it is for me. He says:

‘Of course,in a test with closed questions, a blind guessing score is inevitable. The result of a test as a whole however, is based on a series of questions. The more questions in a test, the more reliable the outcome of a test. The chance of a student passing a test (with a reasonable number of test items) by only guessing, is negligible. And by setting cut-scores that take this guessing score into account, a reasonable estimate concerning the achievement of a student can be made.’

So, Silvester counters the argument that CBM is necessary to correct for guessing. Regular tests do a reasonable good job already.

How does CBM affect your confidence?

Silvester does see benefits in CBM: ‘What should be emphasized is the value of CBM in formative assessments. CBM gives students who over- or underestimate themselves a better insight in their own perception of their knowledge and skill. And this does work on the level of just one question or subject, or possibly at the level of an entire test if this test addresses only one subject.’ (So the CBM scores will improve in the course of taking multiple tests with CBM.)

In general people think that under- or overestimating ones knowledge is an unchangeable personality trait. It would be interesting if Tony had done research on the impact of CBM on other domains. Does it affect their level of confidence also on other material outside this particular test? I don’t think so.’

Well, Tony told us he had not researched this… Maybe someone else will pick this up?

Why doesn’t everybody use it?

We also asked Tony how widespread CBM is. His own software (LAPT) is mainly used in London, at UCL and Imperial College. Even there it is currently only used for formative assessment. Tony is disappointed in the slow uptake despite much effort at dissemination. It might just be fear of the non-standard testing procedures. It might just be fear of the non-standard testing procedures.

In Holland my client SCFB is, to my knowledge, a pioneer in applying CBM. Over 25 years ago there has been some research in the Netherlands on something similar to CBM (multiple evaluation) but the psychometric experts and instructional designers who published about it are either deceased or retired. A whole new generation must be infected with the CBM-virus!

Now it is included in the standard quiz module in Moodle 2.1, albeit as yet in a limited way, other Moodlers will probably try it. And if other big testing/assessment software companies like Question Mark Perception adopt it, it might get a new life. I’ll do my best to advocate CBM at the next Ned Moove seminar at the end of November.

Want to know more?

All Tony’s work on CBM is to be found on the LAPT website of UCL.We thank him for his cooperation on this article.

*SCFB has chosen to translate ‘very sure’ for certainty = 3 into ‘absoluut zeker’. This is not as it should be. Certainty = 3 should stand for: “I am 80 to 100 % sure I have given the correct answer. In the CBM-philosophy you don’t have to be absolutely sure. In the professional world a financial advisor must be totally sure. This is my clients motive for using this terminology.

 

9 thoughts on “Certainty Based Marking in Moodle

  1. It was interesting to meet up with you and Silvester, and to read your account of starting out with CBM. Thanks!

    I always like to encourage your attitude that the main benefit is to make students think more deeply and carefully about Qs they are asked. You’re right of course that initially many people will claim certainty about answers when they should know better – and will be horrified at the end result. They soon learn to be more honest with themselves, which is what I mean by saying CBM rewards students for acknowledging uncertainty. Given practice, in our experience, CBM results in well-calibrated certainty judgements, and in in exams it leads to increased reliability and validity (while retaining all the conventional data examiners are used to). Many studies with broadly similar systems over decades have shown similar results – though our CBM scheme is much simpler than most. From the sound of it, your client very much wants their trainees to acknowledge uncertainty when it is due, to avoid unthinking and dangerous risks. That’s the name of the game! They should persevere.

    It’s a pity that your client has taken up Moodle 2.1 so quickly, because though CBM is starting to be built into 2.1, there is much better functionality currently available with the well established (though not core) code for Moodle 1.9 and 2.0 – see http://www.ucl.ac.uk/lapt – or by using the purpose built LAPT system itself. Much more work is needed on 2.1, which is substantially altered in the architecture of its code. In e-learning environments I think systems should ideally jump outside when specialist functions are required, to wherever these are best developed. That’s how we use Moodle (at UCL) and Blackboard (at Imperial) – jumping outside to LAPT for students to do the CBM quizzes and for them (or their teachers) to look at the CBM grades. There should be no problem doing this from your client’s 2.1 .

    Silvester’s points about guessing are interesting. Having students give answers that are near to being guesses is like adding random numbers to their score (‘noise’). This reduces the ‘reliability’ of an exam test. You can indeed, as Silvester says, get round the problem with enough questions. And of course he is right that no sensible exam board would set a passmark so that students had much chance of passing with just guesses. There are three points to make really:
    1. You still always want to reduce the statistical noise, which CBM achieves by motivating the student to identify uncertain answers and then weighting these less heavily in the overall score.
    2. The main objective is to enhance learning rather than assessment. The sting of getting a confident answer wrong and the insight from being forced to acknowledge uncertainty even though the answer may be right are clear pedagogic benefits.
    3. Even acknowledging (2) and not bothering with (1), the fact that “assessment drives learning” is a good reason, I think, for using CBM in exams – to encourage a better approach to constructive study.

    I never encourage people, however, to think first about using CBM in summative assessment – because exam committees are usually very conservative bodies and won’t (indeed shouldn’t) introduce such changes until the teachers have a lot of experience of what they are talking about. I like to think in any case that improving how students learn is much more important than how they are examined.

    Thanks for a nice stimulating blog!
    Tony

  2. Interesting Blog. I met Tony a few years ago in Colorado, USA and had to opportunity to talk with him about his version of confidence assessment. There are others who have come to similar conclusions surrounding assessing for confidence. Dr. Bruno from UCLA for example examined de Fineetti’s work on partial knowledge to develop a system that assesses for confidence. There is also a system developed by Dr. Leclerc at the University of Liege that attempts to evaluate a degree of certitude for someone’s answer. I have used Tony’s work for a few years with students I work with and will continue to do so because of the ease of use now that it is incorporated with Moodle. I too think it is the best way to formatively assess students, or any learner for that matter.

    • Great to hear from you Tim. I’m going tot do a short talk on CBM uit the next NedMoove-seminar in february. Hope to spread the word amongst Dutch Moodle-users.
      You say you will continue to use it because it is incorporated in Moodle. Are you happy with the way it is done?

      • I forgot to put up here about work I did last month to present the CBM results in a more easily understood way to students, using the concept of a ‘bonus’ to their ordinary accuracy score when their CBM judgement about more and less reliable answers is good. The front end interaction and the mark scheme are not changed. The ‘bonus’ can be negative (unlike bankers’ ones!) but is normally positive (typically around 5%), and the presentation of results should make clear what is wrong if it is small or negative. You can look at the demo of these changes and get the code for Moodle 1.9 and 2.0 at http://www.ucl.ac.uk/lapt/moodle19/moodle. I would dearly like to adapt this code for 2.1/2.2 but have yet to understand the way in which the relevant code zones of 2.1/2.2 work. Is there anyone interested in helping?

        Good to hear from you both, Tim & Isabelle!

        • Hello Tony and company. Our school has finally moved to 2.2.2 and I am able to try the CBM quizzes. I am having issues with the scoring in the current Moodle instance. It does not seem to follow the traditional format that Tony has developed. I would be interested in an explanation of the marking scheme so I can try to evaluate the effectiveness of this CBM system.

  3. A very interesting and informative blog, Isabelle. Thanks, I look forward to your presentation on the next NedMoove-seminar. I learned about CBM reading posts from Tim Hunt about his work for the quiz-module. It intrigued me and still keeps me thinking. As a teacher at a VMBO-school I was thinking to use it to motivate my students. The effects written about in your post will still stand, but my goal is to break the horrible attitude of avoiding to be considered a “schoolnerd” by classmates. Instead they could present themselves as efficient. I did not study it in detail yet, but the idea is to have a set of tests in which the students with a high score (only achievable with certain answers) will have to do lesser tests. Determination is also a hot item, maybe it also useful there. What do you think, could it work?

    • That’s a great idea Johannes. Funny though, that it’s seen as negative if you score well at tests. Maybe ‘I am efficient’ is something to be more proud of? And yes, you could very well combine CBM with conditional activities in Moodle. A certain score on a starttest can lead the student straight to the end-test. Studying well for the first test can save the student a lot of time. And it is abolutely harder to attain a high score if you work with CBM.

      So we’ll talk about it on february 8th!

  4. Just an update. Anyone interested in using CBM in Moodle should look at the new site I’ve set up for demonstration and download of code enhancements for Moodle 2.2, 2.0 and 1.9. These should solve many of the problems along the lines discussed here, still present in core Moodle (2.2.3). See http://www.tmedwin.net/cbm . The code mods are easy to copy into your installation in a single operation (copying just the files that have changed) or to undo if you want, by copying back in the corresponding original files – also included in the downloads.
    Tony GM

    • Interesting Blog. I met Tony a few years ago in Colorado, USA and had to opportunity to talk with him about his viosern of confidence assessment. There are others who have come to similar conclusions surrounding assessing for confidence. Dr. Bruno from UCLA for example examined de Fineetti’s work on partial knowledge to develop a system that assesses for confidence. There is also a system developed by Dr. Leclerc at the University of Liege that attempts to evaluate a degree of certitude for someone’s answer. I have used Tony’s work for a few years with students I work with and will continue to do so because of the ease of use now that it is incorporated with Moodle. I too think it is the best way to formatively assess students, or any learner for that matter.

Comments are closed.