Item-analysis of the multiple-choice questions used in the formative assessment of introductory posting examination in Medicine and Surgery at a medical university in Southern Nigeria.
Keywords:
formative assessment, key assignment, distractor efficiency, difficulty index, discrimination index, item-analysisAbstract
Introduction: The aim of this study was to undertake quality assurance as post-examination analysis of multiple-choice questions (MCQs) used in formative assessment.
Materials & Methods: Classical Test theory (CTT) and Item Response Theory (IRT) of 500 items (100 keys & 400 distractors) in single-best answer MCQs (A-type) in introductory medicine (IM) and introductory surgery (IS) from 62 medical students was done post-examination. Anonymised answer-scripts had item responses made binary as 0 and 1 and analysed using Microsoft Excel
spreadsheet&JMetrik psychometric software to determine difficulty index, discrimination index, distractor efficiency and Cronbach's alpha coefficient.
Results: The mean score in IM was 60.83 ± 9.48 (95% C.I. 58.42 – 63.24)] Fifty-four students (87.10%) [95% C.I. 76.15 – 94.26] passed and 8 (12.9%0 [95% C.I 5.74 - 23.85] failed. Thirteen (20.7%) attained a score of 70 and above. The mean score in IS was 63.5 ± 7.1 (95% C.I 61.70 – 65.32). Sixty (96.77%) [95% C.I 88.83 – 99.61] passed, and 2 (3.23%) [95% C.I 0.39 – 11.17] failed. Twelve (19.3%) scored 70 and above. Difficulty index (DIF-I) of keys set at <0.3 (too hard) was 10% in IM & 14% in IS. DIF-I > 0.8 (too easy) was 22% in IM and 40% in IS respectively. Discrimination index (DI) of keys set <0.1 (poor) was 44% in IM & 48% in IS and >0.3(good) was 10% in IM & 2% in IS respectively. The Cronbach's alpha coefficient was 0.62 in IM & 0.45 in IS respectively. Nonfunctioning and ineffective distractors (NFD) with a score of zero (0) was 28.8% in IM & 45.2% in IS respectively.
Conclusion: Item analysis in this study showed many easy questions with poor discrimination, low reliability index and poor distractor efficiency. We recommend post-examination item-analysis as part of quality assurance matrix after formative assessment.
Downloads
References
1.Guidelines on minimum standards for undergraduate medical & dental education in Nigeria. Medical & Dental Council of Nigeria.
Accessed 23rd November 2024
2.Al-Wardy NM. Assessment methods in undergraduate medical education. Sultan Qaboos Univ Med J. Aug 2010;10(2):203-9.
3.Bhat SK, Prasad KHL. Item analysis and optimizing multiple-choice questions for a viable question bank in ophthalmology: A cross-
sectional study. Indian Journal of Ophthalmology. 2021;69(2)
4.Wood TJ, Cunnington JP, Norman GR. Assessing the measurement properties of a clinical reasoning exercise. Teach Learn Med. Fall 2000; 12(4): 196-200. doi:10.1207/S15328015TLM1204_6
5.Coughlin PA, Featherstone CR. How to Write a High Quality Multiple Choice Question (MCQ): A Guide for Clinicians. European Journal of
Vascular and Endovascular Surgery. 2017;54(5):654-658. doi:10.1016/j.ejvs.2017.07.012
6.Ventista O. Time to increase the quality of the multiple-choice questions you use! 2017; Accessed 23rd November 2024
7.Ndu IK, Ekwochi U, Di Osuorah C, et al. Negative Marking and the Student Physician--A Descriptive Study of Nigerian Medical Schools. J Med
Educ Curric Dev. Jan - Dec 2016;3doi:10.4137/jmecd.S40705
8.Lesage E, Valcke M, Sabbe E. Scoring methods for multiple choice assessment in higher education Is it still a matter of number right
scoring or negative marking? Studies in Educational Evaluation. 2013/09/01/ 2013;39(3):188-193.
9.Knowles M. The modern practice of adult education: From pedagogy to andragogy. Englewood Cliffs, NJ. Cambridge Adult Education; 1980.
10.Roine I, Molina Y, Caneo M. A psychometric appraisal of the dundee ready education environment measure in a medical school in Chile. Educ Health (Abingdon). Sep - Dec 2018; 31(3): 148 - 154. doi: 10. 4103/efh.EfH_17_18
11. Tavakol M, Dennick R. Post - examination analysis of objective tests. Medical Teacher. 2011/06/01 2011;33(6):447-458. doi:10.
3109/0142159X.2011.564682
12. Tavakol M, Dennick R. Post - examination interpretation of objective test data: Monitoring and improving the quality of high-stakes examinations: AMEE Guide No. 66. Medical Teacher. 2012/03/01 2012;34(3):e161-e175. doi:10. 3109/0142159X.2012.651178
13.Gasmalla HEE, Mohamed Tahir MEM. A-Type MCQs. In: Gasmalla HEE, Ibrahim AAM, Wadi MM, Taha MH, eds. Written Assessment in
Medical Education. Springer International Publishing; 2023:73-89.
14.Meyer JP. JMetrik (Version 4.1) Computer Software. 2014; Retrieved from.
15.Brown JD. Testing In Language Programs: A Comprehensive Guide To English Language Assessment. Upper Saddle River, NJ. McGraw-Hill College; 2005.
16.Etobro BA, Taiwo O, Alawaye M. Pragmatic Perspective of Item Analysis Using Microsoft Office Excel Data Analysis Tools. Ilorin Journal of Education. 04/08 2024;44(2):279 - 292.
17.Reynolds CR, Altmann RA, Allen DN. Item Analysis: Methods for Fitting the Right Items to the Right Test. In: Reynolds CR, Altmann RA,
Allen DN, eds. Mastering Modern Psychological Testing: Theory and Methods. Springer International Publishing; 2021:263-289.
18. Understanding Item Analyses. University Of Washington; Accessed 23rd November 2024
19. Tavakol M, Dennick R. Post - examination analysis of objective tests. Med Teach. 2011;33(6):447- 58. doi:10.3109/0142159X.2011.564682
20.Lord FM. The relation of the reliability of multiple-choice tests to the distribution of item difficulties. Psychometrika. 1952;17(2):181-
194. doi:10.1007/BF02288781
21.Ebel RL, Frisbie DA. Essentials of Educational Measurement. Prentice-Hall; 1986.
22.Testa S, Toscano A, Rosato R. Distractor Efficiency in an Item Pool for a Statistics Classroom Exam: Assessing Its Relation With Item
Cognitive Level Classified According to Blooms Taxonomy. Original Research. Frontiers in Psychology. 2018-August-28 2018;9doi:10.3389/fpsyg.2018.01585
23.Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16, 297-334
24.Biggs JB, Tang CKC. Teaching for quality learning at university: what the student does. Maidenhead. McGraw-Hill; 2011.
25.Vegi VAK, Sudhakar PV, Bhimarasetty DM, et al. Multiple-choice questions in assessment: Perceptions of medical students from low-resource setting. J Educ Health Promot. 2022;11:103. doi:10.4103/jehp.jehp_621_21
26.Loh KY, Elsayed I, Nurjahan MI, Roland GS. Item Difficulty and Discrimination Index in Single Best Answer MCQ: End of Semester Examinati-
ons at Taylors Clinical School. Springer Singapore; 2018:167-171.
27.Tarrant M, Ware J, Mohammed AM. An assessment of functioning and non-functioning distractors in multiple-choice questions: a
descriptive analysis. BMC Medical Education. 2009/07/07 2009;9 (1):40. doi:10.1186/1472-6920-9-40
28.Pan Q, Jiang Z. Examining distractor qualities of pediatrics subject tests from a national assessment. Front Med (Lausanne). 2022;9:921719.
doi:10.3389/fmed.2022.921719
29.Nnodim JO. Multiple-choice testing in anatomy. Med Educ. Jul 1992;26 (4):301-9. doi:10.1111/j.1365-2923.1992.tb00173.x
Downloads
Published
Issue
Section
License
All articles published in the journal are licensed under a Creative Commons Attribution (CC BY) license(i.e CC Attribution-NonCommercial-ShareAlike 4.0), allowing others to share, distribute, and build upon the work, provided the original author(s) and source are properly cited. Authors retain the copyright of their work.
