ASSESSMENT OF KNOWLEDGE Multiple Choice Question Dr Eqbal

ASSESSMENT OF KNOWLEDGE Multiple Choice Question Dr. Eqbal Anwer Assistant Professor, Deptt. of Physiology, ELMCH

LEARNING OBJECTIVES n n n What is MCQs? Parts of MCQs? Importance, Advantage & Limitations of MCQs? Where to use these MCQs? How to construct MCQs?

What is Assessment? The process of finding out how much each student knows or can do. 2 TYPES 1. Formative assessment (Diagnostic) v Done during the course. v Provide feedback to the student on his/her progress. v Provide feedback to the teacher with data for modification of his/her teaching. 2. Summative assessment (Certifying) v Done at the end of period. v For placing the student in order of merit. v During giving any certificate.

Why Assessment ? ? ? v Ranking of student. v Measurement of improvement in a student. v To diagnose student difficulties. v Evaluate method of teaching v Motivate student to study. CATEGORISE

GOOD ASSESSMENT 1. Validity – 2. Reliability – 3. Reproducibility Acceptability – 4. Measuring what it tends to measure Acceptability by students/assessor Consequences of Assessment – Educational Impact

What to Assess? & How Often? Domain Of Learning

1. Cognitive Domain n 2. Psychomotor Domain n 3. Knowledge Practical Skills Affective Domain n Attitudes Bloom’s Taxonomy

LEARNING COMPONENTS 1. KNOWLEDGE Dose the student or doctor know what he or she should do? Communication skill 2. SKILLS Psychomotor skill. Is he or she able to do it? 3. ATTITUDE/ RESPONSE In practice, does the student or doctor choose to do it when confronted with a situation? Clinical competence is demonstrated when a task is performed using learned skills and knowledge.

CLINICAL COMPETENCE Does Shows How Knows MILLER’S PYRAMID

METHODS USED FOR THE ASSESSMENT OF MEDICAL STUDENT 1. Long Essay Questions LEQ. 2. Oral Examination. 3. Short Essay Questions SEQ. 4. Multiple Choice Question MCQ. 5. Modified Essay Question MEQ. 6. Patient Management Problem PMP. 7. Observational Assessment, OSCE, OSPE.

About Multiple-Choice Tests

MULTIPLE CHOICE QUESTIONS (MCQ) n n In these Question, you have to choose “ one-best answer” from multiple choices. Questions = Items

Anatomy of an ITEM • • • Given a problem – Stem Followed by single question - Lead – in Option / Choices – – – Correct answer – Key Other incorrect Answers- Distracters

Example Q. A 1 -year-old infant is known to have heart disease and is noted to be cyanosed. Which of the following is the most likely diagnosis? a. Atrial septal defect b. Patent Ductus Arteriosus c. Ventricular septal defect d. Tricuspid atresia

Example Q. A 1 -year-old infant is known to have heart disease and is noted to be cyanosed. Which of the following is the most likely diagnosis? a. b. c. d. Atrial septal defect Patent Ductus Arteriosus Ventricular septal defect Tricuspid atresia STEM LEAD - IN DISTRACTOR KEY

Importance of MCQ Ø Ø An ability to sample a wide range of knowledge. Easy to administer Ø Ø to large numbers of students in a short period of time and minimal human intervention Highly Acceptable – preferred tool for written assessment across most of the academic disciplines.

Advantages MULTIPLE-CHOICE ITEMS can provide: n Versatility in measuring all levels of cognitive ability, n Highly Reliable test scores, n Scoring Efficiency and Accuracy, n Objective measurement of achievement or ability, n Wide sampling of content or objectives, n Reduced guessing factor compared with true-false items, and n Different response alternatives which can provide diagnostic feedback. (Ory & Ryan, 1993)

Limitations n Are difficult and time-consuming to construct, n Place a high degree of dependence on the student’s reading ability and instructor’s writing ability, and n Are particularly subject to clueing. (Students can often deduce the correct response by elimination. ) – need plausible distractor n Attention to security of Question paper (Ory & Ryan, 1993)

When to Use n To test a variety of levels of learning n When you have a large number of individuals taking the test n When you have time to construct the test items n When time is limited for scoring n When it is not important to determine, how well individuals can formulate their own answer n When you want to prepare individuals for future assessments, that use a similar format (Clegg & Cashin, 1986)

Planning to construct MCQ

Multiple-choice Test Construction “… the greater your experience in their construction, the longer it takes per [multiple-choice] item to construct a reasonably fair, accurate, and inclusive question. ” - Wilbert J. Mc. Keachie

Initial focus n Standard / Scope of the examination conducted n Number of question – time available for examination n Source of Question

General guidelines n Pre – plan well n n Peer review/ peer validation n n Best time when you teach More critical review – more you gain Trial run n You‘ll determine facility index & discriminative index

General Rules n Each item should be based on an important learning objective n Test for significant/important information n Focus on single idea in single test item n Use items to assess higher order of thinking n Keep language simple

Constructing Test Items

Components Stem: presents the problem Correct or keyed options: correct option Distractor options: incorrect options (Clegg & Cashin, 1986)

Developing an Item 1. Choose an important concept 2. Write the stem 3. Write the correct answer (key) 4. Develop distractors n n n common misconceptions errors that could be made plausible, yet less important information similar in style, length to the key every distractor should be reasonable (Clegg & Cashin, 1986)

Issues Related to Test-wiseness • Grammatical Cues • Logical Cues • Absolute Terms • Long Correct answer • Word Repeats (Clegg & Cashin, 1986)

Issues Related to Irrelevant Difficulty • Options long • Numeric data not stated consistently • Vague terms • Language not parallel • Options in no logical order • “none of the above” is used • Stems tricky or unnecessarily complicated • Answer to an item is “hinged” to the answer of a related item (Clegg & Cashin, 1986)

STEM

Writing Items • Write items on significant concepts, not trivial facts. • Write items that have a definite answer. • Communicate clearly. • • • Don’t give away the answer by including irrelevant cues in the item. Don’t write items that require skills or knowledge irrelevant to what you are trying to measure. Have items reviewed by knowledgeable persons other than the composer of the question if possible. (Clegg & Cashin, 1986)

Writing Stems • Ensure that the directions in the stem are very clear. (examinee should know exactly what is being asked) n Include the central idea in the stem instead of the choices. n Avoid window dressing (excessive verbiage). n Word the stem positively, avoid negatives such as NOT or EXCEPT. If negative words are used, use the word cautiously and always ensure that the word appears capitalized and boldface. (Haladyna, Downing & Rodriguez, 2002)

Writing Stem n n n Avoid very short stems Write stem in either question form or clinical scenario but only relevant text Clinical scenario is better than statement as this stimulate higher level of thinking

Writing Stems Avoid stems that ask for a series of multiple true-false responses. Q. Which of the following is true about the middle adult years? It encompasses ages 19 to 30. B. It is the most conflict-free period of life. C. It is characterized by dramatic changes in our sense of values. D. It is marked by a conflict between intimacy and isolation. A. Better: Q. According to Erickson, the middle adult years are characterized by the conflict between ____ and ___. A. B. C. D. intimacy; isolation generativity; stagnation integrity; despair industry; despondency (Ory & Ryan, 1993)

Writing Stems Eliminate excessive wording and irrelevant information. Q. Sheldon developed a highly controversial theory of personality based on body type and temperament of the individual. Which of the following is a criticism of Sheldon’s theory? A. He was influenced too much by Freudian psychoanalysis. B. His ratings of physique and temperament were not independent. C. He failed to use an empirical approach. D. His research sample was improperly selected. Better: Q. Which of the following is a criticism of Sheldon's theory of personality? (Ory & Ryan, 1993)

Writing Stems Include in the stem any word(s) that might otherwise be repeated in each alternative. Q. The receptors for the vestibular sense are located A. in the fovea. B. in the brain. C. in the middle ear. D. in the inner ear. Better: Q. The receptors for the vestibular sense are located in the A. fovea. B. brain. C. middle ear. D. the inner ear. (Ory & Ryan, 1993)

Writing Stems Use negatively stated stems sparingly. When used, underline and/or capitalize the negative word. Q. Which is not a major technique for studying brain function? A. B. C. D. accident and injury cutting and removing electrical stimulation direct phrenology Better: Q. Which is NOT a major technique for studying brain function? A. B. C. D. accident and injury cutting and removing electrical stimulation direct phrenology (Ory & Ryan, 1993)

Writing Stems When using incomplete statements avoid beginning with the blank space. Better: Q. ___ is the least severe form of behavior disorder. A. B. C. D. Psychosis Panic disorder Neurasthenia Neurosis Q. The least severe form of behavior disorder is ___. A. B. C. D. Psychosis Panic disorder Neurasthenia Neurosis (Ory & Ryan, 1993)

Writing Stems Use familiar language. Better: Q. According to Freud the raison d’être for hysteria was A. B. C. D. sexual conflicts. unresolved feelings of guilt. latent tendencies. repressed fear. Q. According to Freud hysteria was caused by A. B. C. D. sexual conflicts. unresolved feelings of guilt. latent tendencies. repressed fear. (Ory & Ryan, 1993)

Writing Stems Provide sufficient information in the stem to allow students to respond to the question. Q. How many interrelated stages to creative problem solving are there? A. Three B. Four C. Seven D. Ten Better: Q. The textbook indicates that there are ___ interrelated stages to creative problem solving. A. Three B. Four C. Seven D. Ten (Ory & Ryan, 1993)

WRITING OPTIONS

Writing Options n n n Develop as many effective choices as you can, but research suggests three is adequate. Make sure that only one of these choices is the right answer. Vary the location of the right answer according to the number of choices Place choices in logical or numerical order. Keep choices independent; choices should not be overlapping. (Haladyna, Downing & Rodriguez, 2002)

Writing Options n Keep choices homogeneous in content and grammatical structure. n Keep the length of the choices about equal. n None-of-the-above should be used carefully. n Avoid All-of-the-above. n Make all distractors plausible. n Use typical errors of students to write your distractors. n Use humor if it is compatible with the teacher and the learning environment. (Haladyna, Downing & Rodriguez, 2002)

Writing Options n n Phrase choices positively; avoid negatives such as NOT. Avoid giving clues to the right answer, such as: o o o specific determiners including always, never, completely, and absolutely. clang associations, choices identical to or resembling words in the stem. Very noticeable correct choice. pairs or triplets of options that clue the test-taker to the correct choice. blatantly absurd, ridiculous options. (Haladyna, Downing & Rodriguez, 2002)

Writing Options Make sure there is one correct or best response. Q. Which of the following does not belong with the others? A. B. C. D. Wundt Structuralism James Titchener (Ory & Ryan, 1993)

Writing Options Make all alternatives plausible and equally attractive to both less-knowledgeable and skillful students. Q. The number of photoreceptors in the retina of each human eye is about A. 1000, 000. B. 2 million. C. 115 million. D. 2. 37 billion. Better: A. B. C. D. 5 million. 35 million. 65 million. 115 million. (Ory & Ryan, 1993)

Writing Options Minimize the use of the all-of –the-above and none-of-the-above alternatives. Q. Problem representation involves A. B. C. D. determining which factor matters and which do not. the initial state of problem solving. both a and b. neither a nor b. Better: A. determining which factors matter and which do not. B. the initial state of problem solving. C. reducing the problem to manageable segments. D. all of the above. (Ory & Ryan, 1993)

Writing Options All alternatives should be approximately equal in length. Better: Latane and Darley smoke-filled room experiment suggested that people are less likely to help in groups than alone, because people A. B. C. D. in groups talk to one another. who are alone are more attentive. in groups do not display pluralistic ignorance. in groups allow others to define the situation as a non-emergency. Latane and Darley smoke-filled room experiment suggested that people are less likely to help in groups than alone, because people A. B. C. D. talk to one another. are less attentive than people who are alone. do not display pluralistic ignorance. allow others to define the situation as a non-emergency (Ory & Ryan, 1993)

Writing Item Options Make alternatives parallel in construction and consistent with the stem. Which of the following is NOT a defense mechanism? A. B. C. D. Conflict. Repression. Reaction formation. Rationalization. Better: A. B. C. D. Rationalization. Repression. Reaction formation. Regression. (Ory & Ryan, 1993)

Writing Item Options When possible, present alternatives in some logical order (e. g. most to least and chronological) Q. In the course of a dark adaptation , the eye’s best sensitivity to wavelength shifts to A. B. C. D. 580 millimicrons. 477 millimicrons. 505 millimicrons. 600 millimicrons. Better: A. 600 millimicrons. B. 580 millimicrons. C. 505 millimicrons. D. 477 millimicrons. (Ory & Ryan, 1993)

Writing Item Options Make the alternatives mutually exclusive. Q. Rods are found in the A. B. C. D. blind spot. fovea. periphery of the retina. back of the eye. Better: A. blind spot. B. periphery of the fovea. C. periphery of the retina. D. cornea. (Ory & Ryan, 1993)

Writing Item Options Avoid overly wordy alternatives that become confusing and difficult to read. Flooding differs from systematic desensitization in that A. B. C. D. the former is based on classical conditioning and the latter on operant conditioning. systematic desensitization requires insight and the flooding does not. flooding has you start at the top of your fear hierarchy and systematic desensitization has you start at the bottom and work up gradually. flooding emphasizes the use of cognitions to a much greater extent than does systematic desensitization. Better: Flooding differs from systematic desensitization in that flooding A. B. C. D. is based on classical conditioning rather than operant conditioning. doesn’t require insight. starts at the top of the fear hierarchy. places greater emphasis on the use of cognitions. (Ory & Ryan, 1993)

Writing Item Options Avoid irrelevant cues such as grammatical structure, well-known work associations, or connections between the stem and the correct answer. Q. School psychologists who examine and place children in special education settings often apply the research done by A. B. C. D. Better: School psychologists often apply the research done by biopsychologists. educational psychologists. clinical psychologists. counseling psychologists. (Ory & Ryan, 1993)

Writing Options Avoid language that may offend or exclude a particular group of individuals. Q. Which of the following is a characteristic of persons with Down’s syndrome? A. B. C. D. Larger than normal head Obesity Oriental-like skin folds over the eyes Above average height. Better: A. Larger than normal head B. Obesity C. Downward sloping skin fold over the eyes D. Above average height. (Ory & Ryan, 1993)

SUMMARIZE

For developing options n n n Place option in logical/ numerical sequence Use letters in front of options Keep options independent – mutually exclusive Keep content of all options homogeneous Keep length of option nearly equal Phrase the options positively

Rules for Key n n One and only one Balanced – in a set of items, position of correct option should same no. of times for each position

Rules for Distracter n n Plausible – more similar to correct answer No too closer to correct answer – lead to test for precision

AVOID n n n n Cueing one items with another Tricky items that mislead examinees Items based on assumptions Negative phrasing Providing clues – grammatical / making correct answer longer Use of never/ always None of the above/ all of the above

Tips for Writing Tests • • • Compose test items over time. Test what you really want individuals to learn. Check borrowed items carefully. Create a test bank. Start easy to build confidence. Get feedback on items. (Nilson, 2010)

Planning a Test n n n Use a test matrix or blueprint. Identify major ideas and skills rather than specific details. Use Bloom’s cognitive taxonomy or something appropriate for your context. (Nilson, 2010)

Bloom’s Cognitive Domain Evaluation Synthesis Analysis Application Comprehension Knowledge

Test Matrix

Objectives at Different Levels Level: Knowledge Objective: State the average effective radiation dose from chest CT. Q. What is the average effective radiation dose from chest CT? A. B. C. D. 1 m. Sv 8 m. Sv 16 m. Sv 24 m. Sv

Objectives at Different Levels Level: Comprehension Objective: Knows the disease or not Q. A patient is 54 yr male who looks weak & confused. We suspect it to be diabetes mellitus Which of the sign & symptom serve best to confirm diagnosis A. B. C. D. Poor skin turgor with tenting Recent decrease in appetite Increase thirst and urination Unexplained bruising of the abdomen

Objectives at Different Levels Level: Application Objective: Apply knowledge of clinical feature for diagnosis Q. A 72 -yr male who had several attacks of coronary disease is admitted to hospital for gangrene of one leg and found on admission to have sign & symptom of intestinal obstruction. Which is the most probable cause of obstruction? A. B. C. D. Cancer of colon Intussusceptions Mesenteric Thrombosis Volvulus

Objectives at Different Levels Level: Analysis and Synthesis Objective: Analyze data and synthesize the diagnosis Q. A 68 -yr female stuporous with respiratory distress. O/E: Pulse – 136/min, weak Resp – 32/min with crackles BP – 60 mm Hg Palpatory Skin appears cool and diaphoretic. Capillary Refill Time is significantly delayed Which is the most probable cause? A. B. C. D. Cardiogenic Shock Neurogenic Shock Hypovolumic Shock Septic shock

Item Analysis

ITEM ANALYSIS Important step in the development of any assessment strategy It is a statistical technique which is used for selecting and rejecting the items of the test on the basis of their difficulty value and discriminated power

n n Item statistics are used to assess the performance of individual test items Statistical method used – 1. 2. 3. Facility value – (item difficulty) Discriminative index – (item discrimination) Distractor efficiency

STEPS 1. 2. 3. Evaluate answer card - MARKING Arrange marks list in descending order Divide whole list into 2 half n n High achiever Low achiever If list is bigger – divide upper 30%or 25% and lower 30%or 25%

EXAMPLE G HA G in A ct in L rre ct co rre nt co de t tu en f s tud. o fs no. o o H= n L= Q. A 1 -year-old infant is known to have heart disease and is noted to be cyanosed. Which of the following is the most likely diagnosis? a. Atrial septal defect b. Patent Ductus Arteriosus c. Ventricular septal defect d. Tricuspid atresia Total 100 a b c d HAG(n=50) 0 02 08 40 LAG(n=50) 12 26 02 10

1. Facility Value (F. V) Difficulty index “The facility value of an item is defined as the proportion or percentage of the examinees who have answered the item correctly” J. P. Guilford

Calculation Proportion of respondents selecting the right answer to that item FV = No. of right answer (H + L) N (Total no. of students) X 100 a b c d HAG(n=50) 0 02 08 40 LAG(n=50) 12 26 02 10 40+10/100 = 50 %

RANGE n n n FV ranges from 0 – 100% Low FV – too difficult Question High FV – too easy Question Easy – ≥ 85% Medium – 51%-84% Hard - ≤ 50

Significance n n Determine that students have learned the concept Discriminate between students who knows and who don’t Help in designing better paper Start test with easy (high FV) items towards more difficult items (low FV)

2. Discrimination Index (D. I) “It is the ability of an item to discriminate between skilled and unskilled examinees” - Blood and Budd(1972)

Calculation Helps to differentiate who are knowledgeable DI = 2 x (H - L) N a b c d HAG(n=50) 0 02 08 40 LAG(n=50) 12 26 02 10 2 x(40 -10)/100 = 0. 6

RANGE n n n DI ranges from -> -1 to +1 DI = 1 indicates item perfectly discriminates between those who knows from those who don’t Negative – either ambiguous question or wrong key

RANGE n Item with high DI are preferred ≥ 0. 35 – GOOD <0. 2 – 0. 35> – INTERMEDIATE ≤ 0. 2 - UNACCEPTABLE

Significance n n Flaws in item can be identified Provide feedback for improvement Learning can be improved Misconception in learning can be identified

3. Distractor Efficiency n n Distractor should be PLAUSIBLE Any distractor that is not picked by at least 5% of student – is not a good distractor a b c d HAG(n=50) 0 02 08 40 LAG(n=50) 12 26 02 10 GOOD Change the distractor

QUESTION BANKING n n Large pool of question with set of information regarding question Goal – ability to deposit, discover and retrieve question

Information on Card Side A Content Area: Nerve Muscle Physiology Chapter: 21 (Muscle Blood Flow During Exercise) Item: Stem : main cause of increased blood flow to exercising muscles is Options: a. b. c. d. Raised BP Vasodilatation due to local metabolite Increased sympathetic discharge Increased Heart Rate Key: Vasodilatation due to local metabolite Reference: Guyton physiology 11/e p 247

Side B Year Class Options Q. No. FV a b c DI d 2009 PGET 122 21 18 61 25 49 0. 35 2010 PGET 135 11 16 72 13 64 0. 29 2011 PGET 148 28 21 69 14 52 0. 31 2012 PGET 133 25 15 78 08 62 0. 25