Are You Asking Questions the Right Way?
I just took a couple of BlockChain courses at edX and was dismayed to see that the test questions were being asked in a way that skewed people’s scores the wrong way. It also reminded me of one of the companies I used to work for and how they also set the questions up in such a way that people didn’t want to do the knowledge checks after internal training. Many folks don’t realize that this methodology is flawed, and hopefully you aren’t making the same mistake in your research or when testing employee knowledge.
Photo courtesy of Unsplash.
What is that flaw? Multi-choice questions that are “select all that apply” and are graded as a whole rather than a percentage. Take for example the following question:
1. Which of the following blockchains are Turing complete and support smart contracts (Chain code)? Select all answers that apply.
This really is a series of 4 True/False questions. If you get 3 out of 4 correct – you still don’t get the questions correct. Instead, think of it as:
- True/False – Hyperledger is a blockchain that is Turing complete and supports smart contracts.
- True/False – Bitcoin is a blockchain that is Turing complete and supports smart contracts.
- True/False – Ethereum is a blockchain that is Turing complete and supports smart contracts.
- True/False – Quorum is a blockchain that is Turing complete and supports smart contracts.
Let me be clear – I see the issue with “multiple-guess” test questions. But with proper questions the testee has consistent odds. If posed “select all that apply” then the question has 16 possible answer combinations. And only one will get you credit. You might know 3/4ths of them – but that won’t be adequate. One or two questions like this out of 20 is not too bad of an influence. But if every one is, your data will show lower scores and will not be an accurate gauge of knowledge.
Don’t believe me? Picture the following: Three questions that are “select all that apply” and have 4 possible choices. The person taking the test could get 10 out 12 correct – with the two mistakes spread across two questions. So instead of 83% the person would get 33%. From a low B to a deep F. If you do this once to over-achievers that work for you they’ll rebel and won’t do the knowledge checks. And if you use it to test knowledge you’ll be lowering your scores tremendously.