Orchestrating Evaluation of Complex Educational TechnologiesA Case Study of a CSCL System

  1. Prieto, Luis P. 1
  2. Dimitriadis, Yannis 2
  3. Asensio-Pérez, Juan I. 2
  1. 1 École Polytechnique Fédérale de Lausanne
    info

    École Polytechnique Fédérale de Lausanne

    Lausana, Suiza

  2. 2 Universidad de Valladolid
    info

    Universidad de Valladolid

    Valladolid, España

    ROR https://ror.org/01fvbaw18

Revista:
Qualitative Research in Education

ISSN: 2014-6418

Año de publicación: 2014

Título del ejemplar: June. Special Issue

Volumen: 3

Número: 2

Páginas: 175-205

Tipo: Artículo

DIALNET GOOGLE SCHOLAR lock_openDialnet editor

Otras publicaciones en: Qualitative Research in Education

Resumen

A medida que las tecnologías digitales penetran en nuestra vida diaria, la complejidad de los entornos educativos, y del soporte tecnológico que usamos en ellos, aumenta. Esta mayor complejidad, y la necesidad de los educadores de aplicar nuevas tecnologías en entornos auténticos (con sus múltiples restricciones), han dado lugar a la noción de la práctica educativa con tecnología como una “orquestación del aprendizaje”. Por otro lado, paralelamente, la complejidad de evaluar los beneficios de estas tecnologías educativas también ha aumentado, y no está claro cómo los investigadores-evaluadores pueden hacer frente a la multiplicidad de lugares, tecnologías, informantes y cuestiones que conlleva esta actividad evaluadora. Mediante la proposición de la noción de “orquestación de la evaluación”, este artículo intenta reconciliar las “descripciones de cara al público” que se dan en las publicaciones científicas y la “práctica a pie de planta” de la evaluación de tecnologías educativas, a través del estudio de un caso en el que los autores evaluaron un sistema (GLUE!-PS) para ayudar a profesores a coordinar escenarios colaborativos soportados por ordenador. En este artículo se reutiliza un marco conceptual sobre “aspectos de orquestación”, evaluado internacionalmente, para estructurar la narrativa del caso, mostrando, por ejemplo, cómo las preguntas de evaluación y métodos planeados originalmente fueron modulados para hacer frente a las múltiples restricciones que planteó la evaluación en entornos educativos auténticos.

Referencias bibliográficas

  • Bote-Lorenzo, M. L., Gómez-Sánchez, E., Vega-Gorgojo, G., Dimitriadis, Y., Asensio-Pérez, J. I., & Jorrín-Abellán, I. M. (2008). Gridcole: a tailorable grid service based system that supports scripted collaborative learning. Computers & Education, 51(1), 155–172.
  • Bruce, B. C. (2008). Ubiquitous Learning, Ubiquitous Computing, and life experience. In Proceedings of the 6th International Conference on Networked Learning (NLC2008) (pp. 583–590).
  • Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches. Sage.
  • Dillenbourg, P. (2013). Design for classroom orchestration. Computers & Education, 69, 485–492. doi:10.1016/j.compedu.2013.04.013
  • Dillenbourg, P., Järvelä, S., & Fischer, F. (2009). The Evolution of Research in Computer-Supported Collaborative Learning: from design to orchestration. In N. Balacheff, S. Ludvigsen, T. de Jong, A. Lazonder, & S. Barnes (Eds.), Technology-Enhanced Learning: Principles and Products (pp. 3–19). Springer.
  • Draper, S. W. (1997). Prospects for summative evaluation of CAL in higher education. Research in Learning Technology, 5(1).
  • Economides, A. A. (2005). Collaborative Learning Evaluation (CLE) framework. WSEAS Transactions on Advances in Engineering Education, 2(4), 339–346.
  • Ewing, J., & Miller, D. (2002). A framework for evaluating computer supported collaborative learning. Educational Technology & Society, 5(1), 112–118.
  • Fischer, F., & Dillenbourg, P. (2006). Challenges of orchestrating computer-supported collaborative learning. In Paper presented at the 87th Annual Meeting of the American Educational Research Association (AERA).
  • Garfinkel, H. (2002). Ethnomethodology’s program: Working out Durkheim's aphorism (Vol. 6). Rowman & Littlefield.
  • Glass, R. L. (1995). A structure-based critique of contemporary computing research. Journal of Systems and Software, 28(1), 3–7.
  • Gómez-Sánchez, E., Bote-Lorenzo, M. L., Jorrín-Abellán, I. M., Vega-Gorgojo, G., Asensio-Pérez, J. I., & Dimitriadis, Y. (2009). Conceptual framework for design, technological support and evaluation of collaborative learning. International Journal of Engineering Education, 25(3), 557–568.
  • Guba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. A Journal of Theory, Research, and Development on Educational Communication and Technology, 29(2), 75–91.
  • Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Sage Publications, Inc.
  • Hernández-Leo, D., Asensio-Pérez, J. I., & Dimitriadis, Y. (2005). Computational Representation of Collaborative Learning Flow Patterns Using IMS Learning Desing. Educational Technology & Society, 8(4), 75–89.
  • Jorrín-Abellán, I. M., & Stake, R. E. (2009). Does Ubiquitous Learning Call for Ubiquitous Forms of Formal Evaluation?: An Evaluand oriented Responsive Evaluation Model. Ubiquitous Learning: An International Journal, 1(3), 71–82.
  • Jorrín-Abellán, I. M., Stake, R. E., & Martínez-Monés, A. (2009). The Needlework in evaluating a CSCL system: The Evaluand-oriented Responsive Evaluation Model. In Proceedings of the 9th International Conference on Computer Supported Collaborative Learning (pp. 68–72).
  • Kovalainen, M., Kumpulainen, K., & Satu, V. (2001). Orchestrating classroom interaction in a community of inquiry: Modes of teacher participation. Journal of Classroom Interaction, 36(2), 17–28.
  • Luckin, R. (2008). The Learner Centric Ecology of Resources: a Framework for using Technology to Scaffold Learning. Computers & Education, 50, 449–462.
  • MacNeill, S., & Kraan, W. (2010). Distributed Learning Environments: A Briefing Paper. Retrieved from http://publications.cetis.ac.uk/wp-content/uploads/2011/02/Distributed_Learning.pdf
  • Martı́nez, A., Dimitriadis, Y., Rubia, B., Gómez, E., & de la Fuente, P. (2003). Combining qualitative evaluation and social network analysis for the study of classroom social interactions. Computers & Education, 41(4), 353–368. Retrieved from http://www.sciencedirect.com/science/article/pii/S0360131503000824
  • Martínez-Monés, A., Gómez-Sánchez, E., Dimitriadis, Y., Jorrín-Abellán, I. M., Rubia-Avi, B., & Vega-Gorgojo, G. (2005). Multiple Case Studies to Enhance Project-Based Learning in a Computer Architecture Course. IEEE Transactions on Education, 48(3), 482–489.
  • McKenney, S. (2013). Designing and researching technology-enhanced learning for the zone of proximal implementation. Research in Learning Technology, 21. doi:http://dx.doi.org/10.3402/rlt.v21i0.17374
  • Miles, M. B., & Huberman, M. (1994). Qualitative Data Analysis: An Expanded Sourcebook. Newbury Park, CA, USA: Sage Publications.
  • Oliver, M. (2000). An introduction to the Evaluation of Learning Technology. Educational Technology & Society, 3(4), 20–30.
  • Oliver, M., & Conole, G. (2004). Evaluating information and communications technology: a tool kit for practitioners. In R. Holliman & E. Scanlon (Eds.), Mediating Science Learning through Information and Communications Technology (pp. 139–150). Routledge.
  • Orlikowski, W. J., & Baroudi, J. J. (1991). Studying information technology in organizations: Research approaches and assumptions. Information Systems Research, 2(1), 1–28.
  • Pozzi, F., Manca, S., Persico, D., & Sarti, L. (2007). A general framework for tracking and analysing learning processes in computer-supported collaborative learning environments. Innovations in Education and Teaching International, 44(2), 169–179.
  • Prieto, L. P. (2012). Supporting orchestration of blended CSCL scenarios in Distributed Learning Environments. School of Telecommunications Engineering, University of Valladolid, Spain.
  • Prieto, L. P., Asensio-Pérez, J. I., Dimitriadis, Y., Gómez-Sánchez, E., & Muñoz-Cristóbal, J. A. (2011). GLUE!-PS: A multi-language architecture and data model to deploy TEL designs to multiple learning environments. In Proceedings of the European Conference on Technology-Enhanced Learning (EC-TEL 2011) (pp. 285–298).
  • Prieto, L. P., Asensio-Pérez, J. I., Muñoz-Cristóbal, J. A., Dimitriadis, Y., Jorrín-Abellán, I. M., & Gómez-Sánchez, E. (2013). Enabling Teachers to Deploy CSCL Designs Across Distributed Learning Environments. IEEE Transactions on Learning Technologies, 6(4), 324–336. doi:http://doi.ieeecomputersociety.org/10.1109/TLT.2013.22
  • Prieto, L. P., Asensio-Pérez, J. I., Muñoz-Cristóbal, J. A., Jorrín-Abellán, I. M., Dimitriadis, Y., & Gómez-Sánchez, E. (2014). Supporting orchestration of CSCL scenarios in web-based Distributed Learning Environments. Computers & Education, 73, 9–25. Retrieved from http://www.sciencedirect.com/science/article/pii/S0360131513003321
  • Prieto, L. P., Holenko-Dlab, M., Abdulwahed, M., Gutiérrez, I., & Balid, W. (2011). Orchestrating Technology Enhanced Learning: a literature review and a conceptual framework. International Journal of Technology-Enhanced Learning (IJTEL), 3(6), 583–598.
  • Prieto, L. P., Villagrá-Sobrino, S., Dimitriadis, Y., Schank, P., Penuel, W., & H., D. A. (2011). Mind the Gaps: Using Patterns to Change Everyday Classroom Practice Towards Contingent CSCL Teaching. In Proceedings of the International Conference of Computer-Supported Collaborative Learning (CSCL2011) (Vol. 1, pp. 518–525).
  • Roschelle, J., Dimitriadis, Y., & Hoppe, U. (2013). Classroom orchestration: Synthesis. Computers & Education, 69, 523–526. doi:http://dx.doi.org/10.1016/j.compedu.2013.04.010
  • Schön, D. A. (1983). The Reflective Practitioner - How Professionals Think in Action. Basic Books.
  • Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge Handbook of the Learning Sciences. Cambridge University Press.
  • Stake, R. (2010). Qualitative Research: Studying How Things Work. The Guilford Press.
  • Stake, R. E. (1983). Program Evaluation, Particularly Responsive Evaluation. In Evaluation Models (Vol. 6, pp. 287–310). Springer Netherlands. doi:10.1007/978-94-009-6669-7_17
  • Stake, R. E. (2004). Standards-based and responsive evaluation. Sage.
  • Strijbos, J. W., & Fischer, F. (2007). Methodological challenges for collaborative learning research. Learning and Instruction, 17(4), 389–393.
  • Sutherland, R., & Joubert, M. (2009). D1.1: The STELLAR vision and strategy statement. Retrieved from http://www.stellarnet.eu/kmi/deliverables/20090929_d1-1___vision-and-strategy.pdf
  • Treleaven, L. (2004). A new taxonomy for evaluation studies of online collaborative learning. In Online collaborative learning: Theory and practice (pp. 160–179). Idea Group Inc. (IGI).
  • Tsiatsos, T., Andreas, K., & Pomportsis, A. (2010). Evaluation Framework for Collaborative Educational Virtual Environments. Educational Technology & Society, 13(2), 65–77.
  • Vatrapu, R., Suthers, D. D., & Medina, R. (2008). Usability, sociability, and learnability: A CSCL design evaluation framework. In Proceedings of the 16th International Conference on Computers in Education. Taipei, Taiwan.
  • Villasclaras-Fernández, E., Hernández-Leo, D., Asensio-Pérez, J. I., & Dimitriadis, Y. (2013). Web Collage: An implementation of support for assessment design in CSCL macro-scripts. Computers & Education, 67, 79–97. doi:10.1016/j.compedu.2013.03.002
Fuente de los datos: Dialnet