CBM increases the learning outcome of closed quiz questions in formative assessment
How can you get more out of a simple quiz? Studie Centrum Financiele Branche offers Permanent Education Courses for financial professionals. Their course concept consists of preparation in an online environment and a face-to-face meeting to discuss the learning material. Students have the opportunity to test their knowledge of the course syllabi by taking three online quizzes per subject. After that they send in their solutions to a case.
My client was keen on ways to increase the learning outcome of these quizzes. Serendipity: his father-in-law was asking him : ‘ Why don’t you require your students to state how sure they are of the answer they are giving, so you you know that they are not simply guessing the answer ?’ I had just been to the UK MoodleMoot where Tim Hunt (Open University) presented the new Quiz module in Moodle 2.1. An interesting new feature of the quiz module is called Certainty Based Marking.“That’s what we need”, my client and I thought, so I started to experiment with it. But first, what is it?
What is CBM?
Certainty Based Marking (CBM) is an assessment system in which a student answers multiple choice, true false or short answer questions. After the student selects an answer he has to decide how sure he is of the correctness of his answer. He has to assert the probability of being correct: 1 = Not very (close to guessing), 2 = Reasonably sure or 3 = Absolutely sure*. In this example you see what it looks like in a Moodle quiz in the SCFB-course.
The grade of certainty that has been chosen has consequences for the score on the question, as you see in this table.
Trying this out, I found that the consequences for your total score at the end of the test can be pretty devastating. Say you make a test of 10 questions and each question has a value of 3, you can score 30 marks if you have everything right. If you choose certainty 3 two times and your answer is incorrect, you lose 12 marks. If you also score right two times while you haven chosen certainty = 1 you lose another 4. It gets pretty hard to pass the test, if you are not sure of what you do and what you don’t know.
By testing it we learned that the teachers of SCFB had some difficulty accepting their low total scores for situations in which they answered more than half of the questions correct. This is one of the reasons that students need practice to get used to CBM: a student has to get the opportunity to really learn to work with the method and it is not something you apply in an isolated test.
The guru and the programmer
While researching the CBM phenomenon I found the guru Tony Gardner-Medwin, Emeritus Professor, Department of Physiology (NPP), UCL, London. He has made the system or protocol (LAPT) available at UCL, including availability for anyone interested in using it, and has published about it (see bottom article). He has also written code that can be imported into Moodle, for versions 1.9-2.04. We first used Tony’s LAPT-code, but as soon as Tim Hunt, the Quiz module programmer, incorporated CBM in the standard Moodle quiz features for 2.1 and higher, we switched to using this standard CBM-option. Hunt based his code partly on Tony’s work but not completely.
This might look as a detail for testing nerds, but the difference is pretty important. Tony’s code produces general feedback for the test with the percentage of questions you answered right AND your CBM score. Tims code only keeps the CBM score and pushes this to Moodles gradebook. For the maximum impact on learning outcome a student should be able to see his end score and compare this to his CBM-score, as in LAPT.
A talk with Tony
CBM gave lots of food for thought on testing and scoring. Luckily Tony Gardner-Medwin visited the Netherlands in September. During my research I came in contact with Silvester Draaijer (Onderwijscentrum VU). Silvester also wanted to meet Tony, so we had an interview with him.
We talked about the philosophy behind and the advantages of CBM compared to plain old mc or T/F-questions.
Philosophy behind and advantages of CBM
You could say CBM repairs a flaw of standard testing with closed questions. The results on a standard test do not reveal (in Tony’s view) what the student really knows. The student may give a correct answer, but was it by guessing or was it on the basis of solid knowledge? A student who is a very lucky guesser can pass a test while a student who omits uncertain answers, can fail it. CBM rewards students who know which answers they do know and who acknowledge when they are uncertain. It ‘punishes’ students who are never confident, or who claim confidence for incorrect answers.
For students a test with CBM is much more informative and promotes deeper learning. The ‘How sure are you’ question prompts him to think more deeply about the answer, or, if the test is not time limited, to look up the correct answer. After he submits the answers, he can see if his certainty estimation was accurate. At the end of the test he knows which parts of the material he has to study more in depth and which parts are completely between his ears. Hopefully he also sees feedback with an explanation of the correct answer. In this way the learning yield is much higher.
In short, the students focus on:
- getting as many answers right as possible, and
- accurately judging certainty to achieve a high CBM-score
You need to know when you’re right!
Tony explained how CBM plays an important role in the development of a professional attitude. He taught physiology in medical school. It’s pretty crucial that medical staff can make a difference between ‘I am sure’ and ‘I have doubts’. CBM can teach them to act on their doubts (ask a colleague, grab a textbook) in a very simple way: if I am not sure and also right I loose points! This is also what appealed to my client. Financial advisors give maybe 10 advices a day. Nine times they are right but the tenth advice may lead a client to bankruptcy. In the real world advisors have to be absolutely sure 10 times out of 10. The CBM test is a simulation of their professional environment.
I believe in CBM
Just like Tony, I think that only ‘being sure’ is true knowledge. I am also in favour of it because it stimulates students to think just a little bit deeper. Only this extra cognitive activity should already heighten the learning effect of taking a test.
But, is correction of guessing necessary?
Now it is time for some expert criticism. Silvester has some more knowledge of educational measurement. For him, using CBM is not so obvious as it is for me. He says:
‘Of course,in a test with closed questions, a blind guessing score is inevitable. The result of a test as a whole however, is based on a series of questions. The more questions in a test, the more reliable the outcome of a test. The chance of a student passing a test (with a reasonable number of test items) by only guessing, is negligible. And by setting cut-scores that take this guessing score into account, a reasonable estimate concerning the achievement of a student can be made.’
So, Silvester counters the argument that CBM is necessary to correct for guessing. Regular tests do a reasonable good job already.
How does CBM affect your confidence?
Silvester does see benefits in CBM: ‘What should be emphasized is the value of CBM in formative assessments. CBM gives students who over- or underestimate themselves a better insight in their own perception of their knowledge and skill. And this does work on the level of just one question or subject, or possibly at the level of an entire test if this test addresses only one subject.’ (So the CBM scores will improve in the course of taking multiple tests with CBM.)
In general people think that under- or overestimating ones knowledge is an unchangeable personality trait. It would be interesting if Tony had done research on the impact of CBM on other domains. Does it affect their level of confidence also on other material outside this particular test? I don’t think so.’
Well, Tony told us he had not researched this… Maybe someone else will pick this up?
Why doesn’t everybody use it?
We also asked Tony how widespread CBM is. His own software (LAPT) is mainly used in London, at UCL and Imperial College. Even there it is currently only used for formative assessment. Tony is disappointed in the slow uptake despite much effort at dissemination. It might just be fear of the non-standard testing procedures. It might just be fear of the non-standard testing procedures.
In Holland my client SCFB is, to my knowledge, a pioneer in applying CBM. Over 25 years ago there has been some research in the Netherlands on something similar to CBM (multiple evaluation) but the psychometric experts and instructional designers who published about it are either deceased or retired. A whole new generation must be infected with the CBM-virus!
Now it is included in the standard quiz module in Moodle 2.1, albeit as yet in a limited way, other Moodlers will probably try it. And if other big testing/assessment software companies like Question Mark Perception adopt it, it might get a new life. I’ll do my best to advocate CBM at the next Ned Moove seminar at the end of November.
Want to know more?
All Tony’s work on CBM is to be found on the LAPT website of UCL.We thank him for his cooperation on this article.
*SCFB has chosen to translate ‘very sure’ for certainty = 3 into ‘absoluut zeker’. This is not as it should be. Certainty = 3 should stand for: “I am 80 to 100 % sure I have given the correct answer. In the CBM-philosophy you don’t have to be absolutely sure. In the professional world a financial advisor must be totally sure. This is my clients motive for using this terminology.