Presenters: Patrícia Martinková Ph.D., Adéla Hladká (née Drabinová), M.Sc.
Differential Item Functioning (DIF) analysis is an analytic method useful for identifying potentially biased items in assessments. While simply comparing two groups’ total scores can lead to incorrect conclusions about test fairness, many DIF detection methods have been proposed in the past, those based on total scores as well as those based on Item Response Theory (IRT) models (Martinková, Drabinová et al., 2017).
The workshop will offer an introduction into DIF detection from a practical point of view. We will first provide psychometric background, from Classical Test Theory (CTT) to IRT models. Then, we will introduce the mostly used DIF detection methods including i) Delta plots, ii) Mantel-Haenszel test, iii) Logistic regression, iv) Nonlinear regression, v) Lord/Wald test, vi) Raju’s area between item characteristic curves, and vii) SIBTEST. We will also discuss multinomial regression model for detection of Differential Distractor Functioning (DDF). Further, we will discuss other technical aspects, such as item purification, correction for multiple comparisons etc. We will discuss pros and cons of each method and we will focus on their application in practice on real data examples.
The free statistical software R will be used throughout the sessions using its packages difNLR, difR, deltaPlotR, and mirt. Moreover, the ShinyItemAnalysis package will provide interactive user-friendly interface helpful for those who are new to R (Martinková & Drabinová, 2018).
The workshop is intended for researchers, graduate students and practitioners and all with interest on how to conduct DIF analysis in practice. An introductory statistical background is expected. Some experience in R is helpful, but not required.
Attendees are expected to bring their own laptop with R installed together with the latest versions of the R package ShinyItemAnalysis and its dependencies, including packages difNLR, difR, mirt and deltaPlot. Electronic training materials will be provided to the attendees.
Tentative Schedule
| 09.00 |
Coffee and registration |
|
| 09.30 |
Welcome & introductions
Outline of the Workshop |
Patrícia, Adéla |
| 09.45 |
Background to DIF analysis: from CTT to IRT |
Patrícia |
| 11.00 |
Break |
|
| 11.30 |
Examples in R and ShinyItemAnalysis |
Adéla |
| 13.00 |
Lunch |
|
| 14.00 |
Differential item functioning: Methods and approaches |
Patrícia |
| 15.30 |
Break |
|
| 15.45 |
Examples of DIF analyses in R and ShinyItemAnalysis |
Adéla |
| 16.30 |
Workshop close |
– |
Presenters’ Bios:
Patrícia Martinková is a researcher at the Department of Statistical Modelling, Institute of Computer Science of the Czech Academy of Sciences and the principal investigator of the Center for Educational Measurement and Psychometrics at Faculty of Education, Charles University, Prague. She is also a Fulbright alumna and 2013-2015 visiting research scholar with the Center for Statistics and the Social Sciences (CSSS), and an affiliate assistant professor at Department of Statistics and CSSS, University of Washington. Her research focuses on developing models and estimators for detection of disparities in rating and for better understanding of the differences among different groups. She has a long history in analyzing achievement tests, including those used in university admissions, grant selection or hiring. She taught Item Response Theory Models of Testing at University of Washington, and she teaches Selected topics in psychometrics and Seminar in Psychometrics at Charles University, Prague. Martinková has published number of innovative papers on detection of disparities in rating and assessment. She is the maintainer of the R package ShinyItemAnalysis and she initiated the development of the R package difNLR. For more information about Martinková, visit the webpage: http://www.cs.cas.cz/martinkova/
Adéla Hladká (née Drabinová) is a PhD student at Department of Probability and Mathematical Statistics, Charles University and a PhD fellow at Institute of Computer Science of the Czech Academy of Sciences. Her field of interest is developing statistical tools in psychometrics with main focus on detection of differential item functioning (DIF). Hladká is the main author of difNLR R package – tool for DIF detection using generalized logistic regression models, and one of the main developers of the ShinyItemAnalysis R package and application. She is a teaching assistant for Selected topics in psychometrics at Charles University. For more information about Hladká, see the webpage: http://www.cs.cas.cz/hladka/
Martinková and Hladká organized 11th workshop on Psychometric computing, Psychoco 2019, in Prague. They are both authors of the difNLR package with almost 20,000 downloads from CRAN and of the ShinyItemAnalysis R package and interactive online application, now with more than 13,000 downloads from CRAN and 10,000 online accesses from almost 100 countries around the world. ShinyItemAnalysis package has been featured in the December 2018 issue of The R journal.
Papers about DIF and detection of rating disparities, published by the presenters:
- Martinková, P., & Drabinová, A. (2018). ShinyItemAnalysis for Teaching Psychometrics and to Enforce Routine Analysis of Educational Tests. The R Journal, 10(2), 503-515. https://doi.org/10.32614/RJ-2018-074
- Martinková, P., Goldhaber, D., & Erosheva, E. (2018). Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability. PloS one, 13(10), e0203002. https://doi.org/10.1371/journal.pone.0203002
- Martinková, P., Drabinová, A., Liaw, Y.-L., Sanders, E. A., McFarland, J. L., & Price, R. M. (2017). Checking equity: Why DIF analysis should be a routine part of developing conceptual assessments. CBE Life Sciences Education, 16(2), rm2. https://doi.org/10.1187/cbe.16-10-0307
- Drabinová, A., & Martinková, P. (2017). Detection of Differential Item Functioning with Nonlinear Regression: A Non‐IRT Approach Accounting for Guessing. Journal of Educational Measurement, 54(4), 498-517. https://doi.org/10.1111/jedm.12158
- Martinková, P., Drabinová, A., & Houdek, J. (2017). ShinyItemAnalysis: Analýza přijímacích a jiných znalostních či psychologických testů [ShinyItemAnalysis: Analyzing admission and other educational and psychological tests. In Czech]. TESTFÓRUM, 6(9), 16-35.
- Martinková, P., Štěpánek, L., Drabinová, A., Houdek, J., Vejražka, M., & Štuka, Č. (2017). Semi-real-time analyses of item characteristics for medical school admission tests. In 2017 Federated Conference on Computer Science and Information Systems (FedCSIS) (pp. 189-194). IEEE. https://doi.org/10.15439/2017F380
- McFarland, J. L.*, Price, R. M.*, Wenderoth, M. P.*, Martinková, P.*, Cliff, W., Michael, J., … & Wright, A. (2017). Development and validation of the homeostasis concept inventory. CBE—Life Sciences Education, 16(2), ar35. (*contr. equally) http://dx.doi.org/10.1187/cbe.16-10-0305
- Martinková, P., Drabinová, A., Leder, O., Houdek, J. (2019). ShinyItemAnalysis: Test and Item Analysis via Shiny. R package version 1.3.0. http://www.ShinyItemAnalysis.org
- Drabinová A, Martinková P, Zvára K. (2018) difNLR: Detection of differential item functioning (DIF) and differential distractor functioning (DDF) by non-linear regression models. R package version 1.2.2.
References:
Martinková, P., Drabinová, A., Liaw, Y.-L., Sanders, E. A., McFarland, J. L., & Price, R. M. (2017) Checking equity: Why DIF analysis should be a routine part of developing conceptual assessments. CBE Life Sciences Education, 16(2), rm2. https://doi.org/10.1187/cbe.16-10-0307
Martinková, P., & Drabinová, A. (2018) ShinyItemAnalysis for Teaching Psychometrics and to Enforce Routine Analysis of Educational Tests. The R Journal, 10(2), 503-515. https://doi.org/10.32614/RJ-2018-074
Drabinová, A., & Martinková, P. (2017). Detection of Differential Item Functioning with Nonlinear Regression: A Non‐IRT Approach Accounting for Guessing. Journal of Educational Measurement, 54(4), 498-517. https://doi.org/10.1111/jedm.12158
