Prácticas Cuestionables en Estudios de Validez de Instrumentos de Medición PsicológicaComunalidades y Unicidades de la Crisis de Replicabilidad en el Campo de la Psicometría
- David Paniagua 1
- Iván Sánchez-Iglesias 1
- Alejandro Miguel-Alvaro 1
- Nieves Casas-Aragonez 1
- Marta Evelia Aparicio-Garcia 2
- Raimundo Aguayo-Estremera 1
- 1 Universidad Complutense de Madrid, Facultad de Psicología
- 2 Universidad Complutense de Madrid, Instituto de Estudios Feministas
ISSN: 1135-3848
Año de publicación: 2022
Título del ejemplar: Avances en Medición en Psicología
Volumen: 5
Número: 66
Páginas: 23-34
Tipo: Artículo
Otras publicaciones en: Revista iberoamericana de diagnóstico y evaluación psicológica
Resumen
En la última década se ha estudiado la crisis de replicabilidad: tanto los procesos como las soluciones. Dicho estudio ha supuesto una reflexión sobre el modo de hacer ciencia en psicología. En este trabajo se pretende resumir el estado actual de la crisis en replicabilidad enfocándolo hacia los estudios de validación de instrumentos de medición en psicología. Se explorarán las características que hacen que los estudios de validez tengan unas prácticas cuestionables específicas: Las Prácticas Cuestionables en estudios de Validez (PCV). Distinguimos tres grandes bloques de PCV: Teoría y Diseño, Ejecución y Redacción. Es necesario que se exploren la incidencia de las PCV, que los autores tomen conciencia de que son una mala práctica y que revistas y entidades aúnen esfuerzos en reducir su aparición.
Referencias bibliográficas
- Abad, F. J., Olea, J., Ponsoda, V., & García., C. (2011). Medición en Ciencias Sociales y de la Salud. Síntesis.
- Agha, R. A., Lee, S. Y., Jeong, K. J. L., Fowler, A. J., & Orgill, D. P. (2016). Reporting quality of observational studies in plastic surgery needs improvement: A systematic review. Annals of Plastic Surgery, 76(5), 585-589. https://doi.org/doi:10.1097/SAP.0000000000000419
- Agnoli, F., Wicherts, J., Veldkamp, C., Albiero, P., & Cubelli, R. (2017). Questionable research practices among italian research psychologists. PLOS ONE 12(3). https://doi.org/10.1371/journal.pone.0172792
- Aguayo, R. (2018). La investigación en el síndrome de burnout: Reflexión crítica desde una perspectiva metodológica. Apuntes de Psicología, 36(1-2), 93-100. https://doi.org/10.55414/ap.v36i1-2.715
- Aleksic, J., Alexa, A., Attwood, T. K., Hong, N. C., Dahlö, M., Davey, R., Dinkel, H., Förstner, K. U., Grigorov, I., Hériché, J.-K., Lahti, L., MacLean, D., Markie, M. L., Molloy, J., Schneider, M. V., Scott, C., Smith-Unna, R., & Vieira, B. M. (2014). An open science peer review oath. F1000Research, 3. https://doi.org/10.12688/f1000research.5686.1
- Alister, M., Vickers-Jones, R., Sewell, D. K., & Ballard, T. (2021). How do we choose our giants? Perceptions of replicability in psychological science. Advances in Methods and Practices in Psychological Science, 4(2). https://doi.org/10.1177/25152459211018199
- Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report, 73, American Psychologist3–25. https://doi.org/10.1037/amp0000191.
- Asendorpf, J., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J., Fiedler, K., Fiedler, S., Funder, D., Kliegl, R., Nosek, B., Perugini, M., Roberts, B., Schmitt, M., Van Aken, M., Weber, H., Wicherts, J. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108-119. https://doi.org/10.1002/per.1919
- Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543-554. https://doi.org/10.1177/1745691612459060
- Bandalos, D., & Gerstner, J. J. (2016). Using factor analysis in test construction. En K. Schweizer, K. y DiStefano, C. (Eds.), Principles and methods of test construction: Standards and recent advances (pp. 26-51). Hogrefe Publishing.
- Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., & Rupp, D. E. (2016). Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology, 31(3), 323-338. https://doi.org/10.1007/s10869-016-9456-7
- Bem, D. J. (1987). Writing the empirical journal article. In M. P. Zanna & J. M. Darley (Eds.), The complete academic: A practical guide for the beginning social scientist (pp. 171-201). Random House.
- Begley, G. C., & Ioannidis, J. P. A. (2015). Reproducibility in science: Improving the standard for basic and preclinical research. Circulation Research, 116-126.
- Blanco, F., Perales, J.C., & Vadillo,M.A. (2017). Pot la psicologia rescatar‐se a si mateixa? Incentius, biaix i replicabilitat. Anuari de psicologia de la Societat Valenciana de Psicologia, 18(2), 231‐252.
- Chambers, C. (2017). The seven deadly sins of psychology. In The Seven Deadly Sins of Psychology. Princeton University Press.
- Collins, F. S., & Tabak, L. A. (2014). NIH plans to enhance reproducibility. Nature, 505, 612-613.
- Cristea, I. O., & Naudet, F. (2019). Increase value and reduce waste in research on psychological therapies. Behaviour Research and Therapy.
- Cudeck, R. (2007). Factor analysis in the year 2004: Still spry at 100. In Factor Analysis at 100 (pp. 15-22). Routledge.
- Dubois, J. M., Anderson, E. E., Chibnall, J., Carroll, K., Gibb, T., Ogbuka, C., & Rubbelke, T. (2013). Understanding research misconduct: A comparative analysis of 120 cases of professional wrongdoing. Accountability in Research, 20(5-6), 320-338.
- D'Urso, E. D., Maassen, E., van Assen, M. A., Nuijten, M. B., De Roover, K., & Wicherts, J. (2022). The dire disregard of measurement invariance testing in psychological science.
- Dunn, B., O’Mahen, H., Wright, K., & Brown, G. (2019). A commentary on research rigour in clinical psychological science: How to avoid throwing out the innovation baby with the research credibility bath water in the depression field. Behaviour Research and Therapy.
- Fabrigar, L., Wegener, D., MacCallum, R., & Strahan, E. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272-299. https://doi.org/10.1037/1082-989X.4.3.272
- Fabrigar, L., & Wegener, D. (2012). Exploratory factor analysis. Oxford University Press.
- Ferrando, P., Lorenzo-Seva, U., Hernández-Dorado, A., & Muñiz, J. (2022). Decalogue for the Factor Analysis of Test Items. Psicothema 34(1), 7-17. https://doi.org/10.7334/psicothema2021.456
- Fiedler, K., & Schwarz, N. (2016). Questionable research practices revisited. Social Psychological and Personality Science, 7(1), 45-52. https://doi.org/10.1177/1948550615612150
- Flake, J., & Fried, E. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456-465. https://doi.org/10.1177/2515245920952393
- Freiberg-Hoffmann, A., & Romero-Medina, A. (2021). Approaches and Study Skills Inventory for Students: Comparación de las propiedades psicométricas entre las versiones de lápiz-papel y online en estudiantes universitarios. Revista Iberoamericana de Diagnóstico y Evaluación – e Avaliação Psicológica, 4(64), 17-30. https://doi.org/10.21865/RIDEP61.4.11
- Ford, J., MacCallum, R., & Tait, M. (1986). The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology, 39(2), 291-314. https://doi.org/10.1111/j.1744-6570.1986.tb00583.x
- García-Garzón, E., Lecuona, O., & Carbajal, G. (2018). Estudios de replicación, pre-registros y ciencia abierta en Psicología. Apuntes de Psicología, 36 (1-2), 75-83. https://doi.org/10.55414/ap.v36i1-2.713
- Garfield, E. (1996). What is the primordial reference for the phrase ‘publish or perish’. The Scientist, 10(12), 11.
- Goretzko, D., Pham, T. T. H., & Bühner, M. (2021). Exploratory factor analysis: Current use, methodological developments and recommendations for good practice. Current Psychology, 40(7), 3510-3521. https://doi.org/10.1007/s12144-019-00300-2
- Haig, B. (2014). Investigating the psychological world: Scientific method in the behavioral sciences. MIT press.
- Heilmayr, D. (2022). A course unit and presentation assignment to teach students about open science and replicability in psychology. Scholarship of Teaching and Learning in Psychology. https://doi.org/10.1037/stl0000324
- Henson, R., & Roberts, J. (2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational and Psychological Measurement, 66(3), 393-416.
- Hofmann, B., Myhr, A., & Holm, S. (2013). Scientific dishonesty - A nationwide survey of doctoral students in Norway. BMC Med Ethics 14, 3. https://doi.org/10.1186/1472-6939-14-3
- Ioannidis, J. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
- Izquierdo, I., Olea, J., & Abad, F. (2014). Exploratory factor analysis in validation studies: Uses and recommendations. Psicothema, 26(3), 395-400. https://doi.org/10.7334/psicothema2013.349
- John, L., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532. https://doi.org/10.1177/0956797611430953
- Kuffner, T., & Walker, S. (2019). Why are p-values controversial?. The American Statistician, 73(1), 1-3. https://doi.org/10.1080/00031305.2016.1277161
- Ledesma, R., Ferrando, P., & Tosi, J. (2019). Uso del Análisis Factorial Exploratorio en RIDEP. Recomendaciones para Autores y Revisores. Revista Iberoamericana de Diagnóstico y Evaluación – e Avaliação Psicológica, 52(3), 173-180. https://doi.org/10.21865/RIDEP52.3.13
- LeVeque, R. J., Mitchell, I. M., & Stodden, V. (2012). Reproducible research for scientific computing. Computing in Science & Engineering, 13-17.
- Lilienfeld, S., & Waldman, I. (Eds.). (2017). Psychological science under scrutiny: Recent challenges and proposed solutions. John Wiley & Sons.
- Lilienfeld, S., & Strother, A. (2020). Psychological measurement and the replication crisis: Four sacred cows. Canadian Psychology / Psychologie canadienne, 61(4), 281-288. https://doi.org/10.1037/cap0000236
- Lloret, S., Ferreres, A., Hernández, A., & Tomás, I. (2017). El análisis factorial exploratorio de los ítems: Análisis guiado según los datos empíricos y el software. Anales de Psicología/Annals of Psychology, 33(2), 417-432. https://doi.org/10.6018/analesps.33.2.270211
- MacLeod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., et al. (2014). Biomedical research: Increasing value, reducing waste. The Lancet, 383(9912), 101-104. https://doi.org/10.1016/s0140-6736(13)62329-6.
- Martínez-Arias, M. R., Hernández-Lloreda, M. J. y Hernández-Lloreda, M. V. (2006). Psicometría. Alianza Editorial.
- Muñiz, J. (2018). Introducción a la Psicometría. Teoría Clásica y TRI. Pirámide.
- Norris, M., & Lecavalier, L. (2010). Evaluating the use of exploratory factor analysis in developmental disability psychological research. Journal of Autism and Developmental Disorders, 40(1), 8-20. https://doi.org/10.1007/s10803-009-0816-2
- Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results [Editorial]. Social Psychology, 45(3), 137-141. https://doi.org/10.1027/1864-9335/a000192
- Nosek, B., Hardwicke, T., Moshontz, H., Allard, A., Corker, K., Dreber, A., Fidler, F., Hilgard, J., Struhl, M., Nuijten, M., Rohrer, J., Romero, F., Scheel, A., Scherer, L., Schönbrodt, F., & Vazire, S. Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology 2022 73:1, 719-748
- Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). https://doi.org/10.1126/science.aac4716
- Pearson, M., Schwebel, F., Richards, D., & Witkiewitz, K. (2022). Examining replicability in addictions research: How to assess and ways forward. Psychology of Addictive Behaviors, 36(3), 260-270. https://doi.org/10.1037/adb0000730
- Picho, K., Maggio, L. A., & Artino, A. R. (2016). Science: The slow march of accumulating evidence. Perspectives on Medical Education, 5(6), 350-353. https://doi.org/10.1007/s40037-016-0305-1
- Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71-90. https://doi.org/10.1016/j.dr.2016.06.004
- Rabelo, A., Farias, J., Sarmet, M., Joaquim, T., Hoersting, R., Victorino L., Modesto, J., & Pilati, R. (2019). Questionable research practices among Brazilian psychological researchers: Result from replication study and an international comparison. International Journal of Psychology, 55(4), 674-683. https://doi.org/10.1002/ijop.12632
- Ryan, J., & Tipu, S. (2022). Business and management research: Low instances of replication studies and a lack of author independence in replications. Research Policy, 51(1), 104408. https://doi.org/10.1016/j.respol.2021.104408
- Saunders, R., & Savulescu, J. (2008). Research ethics and lessons from Hwanggate: What can we learn from the Korean cloning fraud?. Journal of Medical Ethics, 34(3), 214-221. http://doi.org/10.1136/jme.2007.023721
- Schimmel, L. M. C., & Van Koppen, P. J. (2017). Verdachten testen: Testgebruik in de forensische psychologie [testing suspects: Use of tests in forensic psychology]. De Psycholoog, 52(10), 34-42. https://www.tijdschriftdepsycholoog.nl/wetenschap/verdachten-testen/
- Shrout, P. E., & Rodgers, J. L. (2018). Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annual Review of Psychology, 69, 487-510.
- Simmons, J., Nelson, L., & Simonsohn, U. (2018). False-positive citations. Perspectives on Psychological Science, 13(2), 255-259. https://doi.org/10.1177/1745691617698146
- Toro, R., Peña-Sarmiento, M., Avendaño-Prieto, B. L., Mejía-Vélez, S., & Bernal-Torres, A. (2022). Análisis empírico del Coeficiente Alfa de Cronbach según opciones de respuesta, muestra y observaciones atípicas. Revista Iberoamericana de Diagnóstico y Evaluación – e Avaliaçãol Psicológica, 63(2), 17-30. https://doi.org/10.21865/RIDEP63.2.02
- Van Dijk, D., Manor, O., & Carey, L. (2014). Publication metrics and success on the academic job market. Current Biology, 24(11), R516-R517. https://doi.org/10.1016/j.cub.2014.04.039
- Vanier, S., Schiavone, S., & Bottesini, J. (2022). Credibility beyond replicability: Improving the four validities in psychological science. Current Directions in Psychological Science, 31(2), 162-168. https://doi.org/10.1177/09637214211067779
- Waters, A. M., LeBeau, R. T., Young, K. S., Dowell, T. L., & Ryan, K. M. (2020). Towards the enhancement of quality publication practices in clinical psychological science. Behaviour Research and Therapy, 124, 103499. https://doi.org/10.1016/j.brat.2019.103499
- Wigboldus, D.H.J., Dotsch, R. (2016). Encourage playing with data and discourage questionable reporting practices. Psychometrika 81, 27-36). https://doi.org/10.1007/s11336-015-9445-1
- Xie, Y., Wang, K., & Kong, Y. (2021). Prevalence of research misconduct and questionable research practices: A systematic review and meta-analysis. Science and Engineering Ethics, 27(4), 1-28. https://doi.org/10.1007/s11948-021-00314-9
- Zuckerman, H. (1977). Deviant behavior and social control in science. In E. Savarin (Ed.), Deviance and social change (pp. 87-138). Sage.
- Zuckerman, H. (1984). Norms and deviant behavior in science. Science, Technology, & Human Values, 9(1), 7-13.