ShinyItemAnalysis: Analýza přijímacích a jiných znalostních či psychologických testů
No.9(2017)
Abstract
Keywords:
přijímací testy; analýza testů; položková analýza; teorie odpovědi na položku; odlišné fungování položek; R; Shiny; admission tests; test analysis; item analysis; item response theory; differential item functioning
Tento článek představuje aplikaci ShinyItemAnalysis pro psychometrickou analýzu testů a jejich položek. ShinyItemAnalysis nabízí psychometrické modely v rámci grafického rozhraní pro volně šiřitelné statistické prostředí R a zpřístupňuje tak jeho funkcionalitu širší veřejnosti. Aplikace pokrývá širokou škálu psychometrických metod, od tradiční položkové analýzy až po složitější latentní modely, nabízí cvičné datové soubory, uvádí rovnice modelů, odhady parametrů a jejich interpretaci, jakož i vybraný zdrojový kód, a je tak vhodným nástrojem pro výuku psychometrických konceptů a jejich implementace v R. Aplikace však také nabízí možnost analýzy vlastních dat a generování reportů a aspiruje tak na to být jednoduchým nástrojem pro rutinní analýzu testů a jejich položek. Závěr článku ukazuje, že ShinyItemAnalysis je dostupným, flexibilním a uživatelsky příjemným nástrojem, který může pomoci tomu, aby se statistická analýza přijímacích i jiných znalostních či psychologických testů stala v praxi samozřejmou záležitostí.
In this paper we introduce ShinyItemAnalysis application for psychometric analysis of educational and psychological tests and their items. ShinyItemAnalysis provides graphical interface and web framework to open source statistical software R and thus opens up its functionality to wide audience. Application covers broad range of methods and offers data examples, model equations, parameter estimates, interpretation of results, together with selected R code, and is thus suitable for teaching psychometric concepts with R. The application also aspires to be a simple tool for routine analysis by allowing the users to upload and analyze their own data and by generating analysis report. We conclude by arguing that psychometric analysis should be a routine part of test development in order to gather proofs of reliability and validity of the measurement. With example of admission test to medical faculty, we demonstrate how ShinyItemAnalysis may provide a simple and free tool to routinely analyze tests.
přijímací testy; analýza testů; položková analýza; teorie odpovědi na položku; odlišné fungování položek; R; Shiny; admission tests; test analysis; item analysis; item response theory; differential item functioning
Author biographies
Patrícia Martinková
Ústav informatiky AV ČR
Vědecký pracovník na Ústavu informatiky AV ČR.
Adéla Drabinová
Ústav informatiky AV ČR, Katedra pravděpodobnosti a matematické statistiky, Matematicko-fyzikální fakulta Univerzity Karlovy
doktorand
Jakub Houdek
Ústav informatiky AV ČR, Fakulta informatiky a statistiky, Vysoká škola ekonomická v Praze
student magisterského studia
References
AERA, APA, & NCME. (2014). Standards for educational and psychological testing. American Educational Research Association.
Agresti, A. (2013). Categorical Data Analysis. Wiley. Retrieved from http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470463635.html
Akaike, H. (1974). A new look at the statistical model identification. Automatic Control, IEEE Transactions on, 19(6), 716–723. https://doi.org/10.1109/tac.1974.1100705
Ames, A. J., & Penfield, R. D. (2015). An NCME Instructional Module on Item-Fit Statistics for Item Response Theory Models. Educational Measurement: Issues and Practice, 34(3), 39–48. https://doi.org/10.1111/emip.12067
Anděl, J., & Zvára, K. (2005). Přijímací zkouška z matematiky na MFF v roce 2004. Pokroky Matematiky, Fyziky a Astronomie, 50(2), 148–161. Retrieved from http://hdl.handle.net/10338.dmlcz/141263%0A
Andrich, D. (1982). An Index of Person Separation in Latent Trait Theory, the Traditional KR-20 Index, and the Guttman Scale Response Pattern. Education Research and Perspective, 9(1), 95–104.
Angoff, W. H., & Ford, S. F. (1973). Item-Race Interaction on a Test of Scholastic Aptitude. Journal of Educational Measurement, 10(2), 95–106. Retrieved from http://www.jstor.org/stable/1433905
Bock, D. R. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29–51. https://doi.org/10.1007/BF02291411
Byčkovský, P., & Zvára, K. (2007). Konstrukce a analýza testů pro přijímací řízení. Univerzita Karlova v Praze, Pedagogická fakulta. Retrieved from https://books.google.cz/books?id=mvvjtgAACAAJ
Cai, L., Thissen, D., & du Toit, S. H. C. (2011). IRTPRO for Windows. Lincolnwood, IL: Scientific Software International.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
ČŠI. (2015). Honocení výsledků vzdělávání didaktickými testy. Retrieved from http://www.csicr.cz/cz/Aktuality/Hodnoceni-vysledku-vzdelavani-didaktickymi-testy
de Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford Press.
Drabinová, A., Martinková, P., & Zvára, K. (2017). difNLR: Detection of Dichotomous Differential Item Functioning (DIF) and Differential Distractor Functioning (DDF) by Non-Linear Regression Models. Retrieved from https://cran.r-project.org/package=difNLR
Höschl, C., & Kožený, J. (1997). Predicting academic performance of medical students: The first three years. The American Journal of Psychiatry, 154(6), 86.
Chang, W., Cheng, J., Allaire, J. J., Xie, Y., & McPherson, J. (2017). shiny: Web Application Framework for R. Retrieved from https://cran.r-project.org/package=shiny
IBM Corp. Released. (2015). IBM SPSS Statistics for Windows, Version 23.0. 2015.
Jelínek, M., Květoň, P., & Vobořil, D. (2011). Testování v psychologii: Teorie odpovědi na položku a počítačové adaptivní testování. Praha: Grada.
Kingston, N., Leary, L., & Wightman, L. (1985). An Exploratory Study of the Applicability of Item Response Theory Methods to the Graduate Management Admission Test. ETS Research Report Series. https://doi.org/doi.org/10.1002/j.2330-8516.1985.tb00119.x
Kožený, J., Tišanská, L., & Höschl, C. (2001). Akademická úspěšnost na střední škole: prediktor absolvování studia medicíny. Československá Psychologie : Časopis pro Psychologickou Teorii a Praxi, 45(1), 1–6. Retrieved from http://www.medvik.cz/link/bmc01014269
Legewie, J., & DiPrete, T. A. (2014). The High School Environment and the Gender Gap in Science and Engineering. Sociology of Education, 87(4), 259–280. https://doi.org/10.1177/0038040714547770
Linacre, J. M. (2005). Rasch dichotomous model vs. one-parameter logistic model. Rasch Measurement Transactions, 19(3), 1032.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Routledge.
Magis, D., Béland, S., Tuerlinckx, F., & De Boeck, P. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42, 847–862. https://doi.org/10.3758/BRM.42.3.847
Mantel, N., & Haenszel, W. (1959). Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease. JNCI: Journal of the National Cancer Institute, 22(4), 719. https://doi.org/https://doi.org/10.1093/jnci/22.4.719
Martinková, P., Drabinová, A., Leder, O., Houdek, J. (2017). ShinyItemAnalysis: Test and Item Analysis via Shiny. Retrieved from https://cran.r-project.org/package=ShinyItemAnalysis
Martinková, P., Drabinová, A., Liaw, Y.-L., Sanders, E. A., McFarland, J., & Price, R. M. (2017). Checking equity: Why DIF analysis should be a routine part of developing conceptual assessments. In review.
Martinková, P., & Zvára, K. (2007). Reliability in the Rasch Model. Kybernetika, 43(3), 315–326. Retrieved from http://dml.cz/bitstream/handle/10338.dmlcz/135776/Kybernetika_43-2007-3_4.pdf
McFarland, J., Price, R. M., Wenderoth, M. P., Martinková, P., Cliff, W., Michael, J., Modell H., Wright, A. (in press). Development and validation of the Homeostasis Concept Inventory. CBE-Lifesciences.
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. ETS Research Report Series, 1992(1).
R Development Core Team. (2016). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing Vienna Austria, 0, {ISBN} 3-900051-07-0. https://doi.org/https://doi.org/10.1038/sj.hdy.6800737
Raju, N. S. (1990). Determining the Significance of Estimated Signed and Unsigned Areas Between Two Item Response Functions.pdf. Applied Psychological Measurement, 14, 197–207.
Revelle, W. (2009). An introduction to psychometric theory with applications in R. Retrieved from http://www.personality-project.org/r/book/
Rubešová, J. (2009). Souvisí úspěšnost studia na vysoké škole se středoškolským prospěchem? Pedagogická Orientace, 19(3), 89–103.
Rusch, T., Mair, P., & Hatzinger, R. (2013). Psychometrics With R: A Review Of CRAN Packages For Item Response Theory. Center for Empirical Research Methods, Discussion Paper Series, (November).
Řasová, K., Martinková, P., Vyskotová, J., & Šedová, M. (2012). Assessment set for evaluation of clinical outcomes in multiple sclerosis - psychometric properties. Patient Related Outcome Measures, 3, 59–70. Retrieved from https://www.dovepress.com/assessment-set-for-evaluation-of-clinical-outcomes-in-multiple-scleros-peer-reviewed-article-PROM
SABER. (n.d.). Biology Concept Inventories and Assessments. Retrieved March 9, 2017, from http://saber-biologyeducationresearch.wikispaces.com/DBER-Concept+Inventories
Salvatori, P. (2001). Reliability and Validity of Admissions Tools Used to Select Students for the Health Professions. Advances in Health Sciences Education, 6(2), 159–175. https://doi.org/10.1023/A:1011489618208
SAS Institute Inc. (2013). SAS 9.4 Language Reference: Concepts. Cary, NC, USA: SAS Institute Inc.
Schwarz, G. (1978). Estimating the Dimension of a Model. The Annals of Statistics, 6(2), 461–464. https://doi.org/10.2307/2958889
StataCorp. (2015). Stata Statistical Software: Release 14. 2015. https://doi.org/10.2307/2234838
Swaminathan, H., & Rogers, H. J. (1990). Detecting Differential Item Functioning Using Logistic Regression Procedures. Source Journal of Educational Measurement, 27(4), 361–370. Retrieved from http://www.jstor.org/stable/1434855
Štuka, Č., Martinková, P., Vejražka, M., Trnka, J., & Komenda, M. (2013). Testování při výuce medicíny. Konstrukce a analýza testů na lékařských fakultách. (Vyd. 1.). Praha: Karolinum. Retrieved from http://www.wikiskripta.eu/Testy
Štuka, Č., Martinková, P., Zvára, K., & Zvárová, J. (2012). The prediction and probability for successful completion in medical study based on tests and pre-admission grades. The New Educational Review, 28, 138–152. Retrieved from http://www.educationalrev.us.edu.pl/dok/volumes/tner_2_2012.pdf
Štuka, Č., Vejražka, M., Martinková, P., Komenda, M., & Štěpánek, L. (2016). The use of test and item analysis for improvment of tests. In Mefanet. Brno. Retrieved from http://www.mefanet.cz/index.php?pg=konference--prezentace
Urbánek, T., Denglerová, D., & Širůček, J. (2011). Psychometrika: měření v psychologii. Portál.
van der Linden, W. J. (2017). Handbook of Item Response Theory, Three Volume Set. CRC Press.
Wilson, M. (2005). Constructing measures: An item response modeling approach. Taylor & Francis. Retrieved from https://doi.org/10.4324/9781410611697
Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago: Mesa Press.
Wu, M. L., Adams, R. J., & Wilson, M. R. (2008). ConQuest: Multi-Aspect Test Software. Camberwell: Australian Council for Educational Research.
Zvára, K., & Anděl, J. (2001). Connections between the results of entrance examinations and successful completion of studies at the Faculty of Mathematics and Physics. Pokroky Mat. Fyz. Astron., 46(4), 304–312. Retrieved from http://dml.cz/dmlcz/141097
Zwick, R. (2006). Higher education admission tests. In Educational Measurement (4th ed.). Westport, CT: American Council on Education/Praeger.
Agresti, A. (2013). Categorical Data Analysis. Wiley. Retrieved from http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470463635.html
Akaike, H. (1974). A new look at the statistical model identification. Automatic Control, IEEE Transactions on, 19(6), 716–723. https://doi.org/10.1109/tac.1974.1100705
Ames, A. J., & Penfield, R. D. (2015). An NCME Instructional Module on Item-Fit Statistics for Item Response Theory Models. Educational Measurement: Issues and Practice, 34(3), 39–48. https://doi.org/10.1111/emip.12067
Anděl, J., & Zvára, K. (2005). Přijímací zkouška z matematiky na MFF v roce 2004. Pokroky Matematiky, Fyziky a Astronomie, 50(2), 148–161. Retrieved from http://hdl.handle.net/10338.dmlcz/141263%0A
Andrich, D. (1982). An Index of Person Separation in Latent Trait Theory, the Traditional KR-20 Index, and the Guttman Scale Response Pattern. Education Research and Perspective, 9(1), 95–104.
Angoff, W. H., & Ford, S. F. (1973). Item-Race Interaction on a Test of Scholastic Aptitude. Journal of Educational Measurement, 10(2), 95–106. Retrieved from http://www.jstor.org/stable/1433905
Bock, D. R. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29–51. https://doi.org/10.1007/BF02291411
Byčkovský, P., & Zvára, K. (2007). Konstrukce a analýza testů pro přijímací řízení. Univerzita Karlova v Praze, Pedagogická fakulta. Retrieved from https://books.google.cz/books?id=mvvjtgAACAAJ
Cai, L., Thissen, D., & du Toit, S. H. C. (2011). IRTPRO for Windows. Lincolnwood, IL: Scientific Software International.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
ČŠI. (2015). Honocení výsledků vzdělávání didaktickými testy. Retrieved from http://www.csicr.cz/cz/Aktuality/Hodnoceni-vysledku-vzdelavani-didaktickymi-testy
de Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford Press.
Drabinová, A., Martinková, P., & Zvára, K. (2017). difNLR: Detection of Dichotomous Differential Item Functioning (DIF) and Differential Distractor Functioning (DDF) by Non-Linear Regression Models. Retrieved from https://cran.r-project.org/package=difNLR
Höschl, C., & Kožený, J. (1997). Predicting academic performance of medical students: The first three years. The American Journal of Psychiatry, 154(6), 86.
Chang, W., Cheng, J., Allaire, J. J., Xie, Y., & McPherson, J. (2017). shiny: Web Application Framework for R. Retrieved from https://cran.r-project.org/package=shiny
IBM Corp. Released. (2015). IBM SPSS Statistics for Windows, Version 23.0. 2015.
Jelínek, M., Květoň, P., & Vobořil, D. (2011). Testování v psychologii: Teorie odpovědi na položku a počítačové adaptivní testování. Praha: Grada.
Kingston, N., Leary, L., & Wightman, L. (1985). An Exploratory Study of the Applicability of Item Response Theory Methods to the Graduate Management Admission Test. ETS Research Report Series. https://doi.org/doi.org/10.1002/j.2330-8516.1985.tb00119.x
Kožený, J., Tišanská, L., & Höschl, C. (2001). Akademická úspěšnost na střední škole: prediktor absolvování studia medicíny. Československá Psychologie : Časopis pro Psychologickou Teorii a Praxi, 45(1), 1–6. Retrieved from http://www.medvik.cz/link/bmc01014269
Legewie, J., & DiPrete, T. A. (2014). The High School Environment and the Gender Gap in Science and Engineering. Sociology of Education, 87(4), 259–280. https://doi.org/10.1177/0038040714547770
Linacre, J. M. (2005). Rasch dichotomous model vs. one-parameter logistic model. Rasch Measurement Transactions, 19(3), 1032.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Routledge.
Magis, D., Béland, S., Tuerlinckx, F., & De Boeck, P. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42, 847–862. https://doi.org/10.3758/BRM.42.3.847
Mantel, N., & Haenszel, W. (1959). Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease. JNCI: Journal of the National Cancer Institute, 22(4), 719. https://doi.org/https://doi.org/10.1093/jnci/22.4.719
Martinková, P., Drabinová, A., Leder, O., Houdek, J. (2017). ShinyItemAnalysis: Test and Item Analysis via Shiny. Retrieved from https://cran.r-project.org/package=ShinyItemAnalysis
Martinková, P., Drabinová, A., Liaw, Y.-L., Sanders, E. A., McFarland, J., & Price, R. M. (2017). Checking equity: Why DIF analysis should be a routine part of developing conceptual assessments. In review.
Martinková, P., & Zvára, K. (2007). Reliability in the Rasch Model. Kybernetika, 43(3), 315–326. Retrieved from http://dml.cz/bitstream/handle/10338.dmlcz/135776/Kybernetika_43-2007-3_4.pdf
McFarland, J., Price, R. M., Wenderoth, M. P., Martinková, P., Cliff, W., Michael, J., Modell H., Wright, A. (in press). Development and validation of the Homeostasis Concept Inventory. CBE-Lifesciences.
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. ETS Research Report Series, 1992(1).
R Development Core Team. (2016). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing Vienna Austria, 0, {ISBN} 3-900051-07-0. https://doi.org/https://doi.org/10.1038/sj.hdy.6800737
Raju, N. S. (1990). Determining the Significance of Estimated Signed and Unsigned Areas Between Two Item Response Functions.pdf. Applied Psychological Measurement, 14, 197–207.
Revelle, W. (2009). An introduction to psychometric theory with applications in R. Retrieved from http://www.personality-project.org/r/book/
Rubešová, J. (2009). Souvisí úspěšnost studia na vysoké škole se středoškolským prospěchem? Pedagogická Orientace, 19(3), 89–103.
Rusch, T., Mair, P., & Hatzinger, R. (2013). Psychometrics With R: A Review Of CRAN Packages For Item Response Theory. Center for Empirical Research Methods, Discussion Paper Series, (November).
Řasová, K., Martinková, P., Vyskotová, J., & Šedová, M. (2012). Assessment set for evaluation of clinical outcomes in multiple sclerosis - psychometric properties. Patient Related Outcome Measures, 3, 59–70. Retrieved from https://www.dovepress.com/assessment-set-for-evaluation-of-clinical-outcomes-in-multiple-scleros-peer-reviewed-article-PROM
SABER. (n.d.). Biology Concept Inventories and Assessments. Retrieved March 9, 2017, from http://saber-biologyeducationresearch.wikispaces.com/DBER-Concept+Inventories
Salvatori, P. (2001). Reliability and Validity of Admissions Tools Used to Select Students for the Health Professions. Advances in Health Sciences Education, 6(2), 159–175. https://doi.org/10.1023/A:1011489618208
SAS Institute Inc. (2013). SAS 9.4 Language Reference: Concepts. Cary, NC, USA: SAS Institute Inc.
Schwarz, G. (1978). Estimating the Dimension of a Model. The Annals of Statistics, 6(2), 461–464. https://doi.org/10.2307/2958889
StataCorp. (2015). Stata Statistical Software: Release 14. 2015. https://doi.org/10.2307/2234838
Swaminathan, H., & Rogers, H. J. (1990). Detecting Differential Item Functioning Using Logistic Regression Procedures. Source Journal of Educational Measurement, 27(4), 361–370. Retrieved from http://www.jstor.org/stable/1434855
Štuka, Č., Martinková, P., Vejražka, M., Trnka, J., & Komenda, M. (2013). Testování při výuce medicíny. Konstrukce a analýza testů na lékařských fakultách. (Vyd. 1.). Praha: Karolinum. Retrieved from http://www.wikiskripta.eu/Testy
Štuka, Č., Martinková, P., Zvára, K., & Zvárová, J. (2012). The prediction and probability for successful completion in medical study based on tests and pre-admission grades. The New Educational Review, 28, 138–152. Retrieved from http://www.educationalrev.us.edu.pl/dok/volumes/tner_2_2012.pdf
Štuka, Č., Vejražka, M., Martinková, P., Komenda, M., & Štěpánek, L. (2016). The use of test and item analysis for improvment of tests. In Mefanet. Brno. Retrieved from http://www.mefanet.cz/index.php?pg=konference--prezentace
Urbánek, T., Denglerová, D., & Širůček, J. (2011). Psychometrika: měření v psychologii. Portál.
van der Linden, W. J. (2017). Handbook of Item Response Theory, Three Volume Set. CRC Press.
Wilson, M. (2005). Constructing measures: An item response modeling approach. Taylor & Francis. Retrieved from https://doi.org/10.4324/9781410611697
Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago: Mesa Press.
Wu, M. L., Adams, R. J., & Wilson, M. R. (2008). ConQuest: Multi-Aspect Test Software. Camberwell: Australian Council for Educational Research.
Zvára, K., & Anděl, J. (2001). Connections between the results of entrance examinations and successful completion of studies at the Faculty of Mathematics and Physics. Pokroky Mat. Fyz. Astron., 46(4), 304–312. Retrieved from http://dml.cz/dmlcz/141097
Zwick, R. (2006). Higher education admission tests. In Educational Measurement (4th ed.). Westport, CT: American Council on Education/Praeger.
