The biggest design issue with multiple choice tests is that writing a good, coherent multiple choice question is difficult because of the thin line between right and kinda right. Being adept at it means specializing both in the craft itself and having an encyclopedic knowledge of the field being tested. Most exams that I’ve worked with were written by very informed people who wrote frustratingly ambiguous questions. An example of a bad question would be one that distinguished the right and kinda right answer because one used the word “presumed” and the other used “inferred”. While the words certainly have two different meanings, spotting the distinction had nothing to do with the subject matter of the question. Professional exam writers are aware of this problem and now many exams will test a question out before actually counting it. Out of an exam of 100 questions, 10 will be experimental ones that see how many people get it right. Once they’ve got the rough percentage of how many people get it wrong on average, they factor it in with easier and more difficult ones. It goes back to the overall purpose of the exam being to exclude people but not too many people. A multiple choice exam is looking for a sweetspot of a certain percentage passing, not too high and not too low.
via popmatters.com