Orchestrating Evaluation of Complex Educational TechnologiesA Case Study of a CSCL System

  1. Prieto, Luis P. 1
  2. Dimitriadis, Yannis 2
  3. Asensio-Pérez, Juan I. 2
  1. 1 École Polytechnique Fédérale de Lausanne
    info

    École Polytechnique Fédérale de Lausanne

    Lausana, Suiza

  2. 2 Universidad de Valladolid
    info

    Universidad de Valladolid

    Valladolid, España

    ROR https://ror.org/01fvbaw18

Journal:
Qualitative Research in Education

ISSN: 2014-6418

Year of publication: 2014

Issue Title: June. Special Issue

Volume: 3

Issue: 2

Pages: 175-205

Type: Article

DIALNET GOOGLE SCHOLAR lock_openDialnet editor

More publications in: Qualitative Research in Education

Abstract

As digital technologies permeate every aspect of our lives, the complexity of the educational settings, and of the technological support we use within them, unceasingly rises. This increased complexity, along with the need for educational practitioners to apply such technologies within multi-constraint authentic settings, has given rise to the notion of technology-enhanced learning practice as “orchestration of learning”. However, at the same time, the complexity involved in evaluating the benefits of such educational technologies has also increased, prompting questions about the way evaluators can cope with the different places, technologies, informants and issues involved in their evaluation activity. By proposing the notion of “orchestrating evaluation”, this paper tries to reconcile the often disparate “front office accounts” of research publications and the “shop floor practice” of evaluation of educational technology, through the case study of evaluating a system to help teachers in coordinating computer-supported collaborative learning scenarios (GLUE!-PS). We reuse an internationally-evaluated conceptual framework of “orchestration aspects” (design, management, adaptation, pragmatism, etc.) to structure the case’s narrative, showing how the original evaluation questions and methods were modulated in the face of the multiple (authentic) evaluation setting constraints.

Bibliographic References

  • Bote-Lorenzo, M. L., Gómez-Sánchez, E., Vega-Gorgojo, G., Dimitriadis, Y., Asensio-Pérez, J. I., & Jorrín-Abellán, I. M. (2008). Gridcole: a tailorable grid service based system that supports scripted collaborative learning. Computers & Education, 51(1), 155–172.
  • Bruce, B. C. (2008). Ubiquitous Learning, Ubiquitous Computing, and life experience. In Proceedings of the 6th International Conference on Networked Learning (NLC2008) (pp. 583–590).
  • Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches. Sage.
  • Dillenbourg, P. (2013). Design for classroom orchestration. Computers & Education, 69, 485–492. doi:10.1016/j.compedu.2013.04.013
  • Dillenbourg, P., Järvelä, S., & Fischer, F. (2009). The Evolution of Research in Computer-Supported Collaborative Learning: from design to orchestration. In N. Balacheff, S. Ludvigsen, T. de Jong, A. Lazonder, & S. Barnes (Eds.), Technology-Enhanced Learning: Principles and Products (pp. 3–19). Springer.
  • Draper, S. W. (1997). Prospects for summative evaluation of CAL in higher education. Research in Learning Technology, 5(1).
  • Economides, A. A. (2005). Collaborative Learning Evaluation (CLE) framework. WSEAS Transactions on Advances in Engineering Education, 2(4), 339–346.
  • Ewing, J., & Miller, D. (2002). A framework for evaluating computer supported collaborative learning. Educational Technology & Society, 5(1), 112–118.
  • Fischer, F., & Dillenbourg, P. (2006). Challenges of orchestrating computer-supported collaborative learning. In Paper presented at the 87th Annual Meeting of the American Educational Research Association (AERA).
  • Garfinkel, H. (2002). Ethnomethodology’s program: Working out Durkheim's aphorism (Vol. 6). Rowman & Littlefield.
  • Glass, R. L. (1995). A structure-based critique of contemporary computing research. Journal of Systems and Software, 28(1), 3–7.
  • Gómez-Sánchez, E., Bote-Lorenzo, M. L., Jorrín-Abellán, I. M., Vega-Gorgojo, G., Asensio-Pérez, J. I., & Dimitriadis, Y. (2009). Conceptual framework for design, technological support and evaluation of collaborative learning. International Journal of Engineering Education, 25(3), 557–568.
  • Guba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. A Journal of Theory, Research, and Development on Educational Communication and Technology, 29(2), 75–91.
  • Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Sage Publications, Inc.
  • Hernández-Leo, D., Asensio-Pérez, J. I., & Dimitriadis, Y. (2005). Computational Representation of Collaborative Learning Flow Patterns Using IMS Learning Desing. Educational Technology & Society, 8(4), 75–89.
  • Jorrín-Abellán, I. M., & Stake, R. E. (2009). Does Ubiquitous Learning Call for Ubiquitous Forms of Formal Evaluation?: An Evaluand oriented Responsive Evaluation Model. Ubiquitous Learning: An International Journal, 1(3), 71–82.
  • Jorrín-Abellán, I. M., Stake, R. E., & Martínez-Monés, A. (2009). The Needlework in evaluating a CSCL system: The Evaluand-oriented Responsive Evaluation Model. In Proceedings of the 9th International Conference on Computer Supported Collaborative Learning (pp. 68–72).
  • Kovalainen, M., Kumpulainen, K., & Satu, V. (2001). Orchestrating classroom interaction in a community of inquiry: Modes of teacher participation. Journal of Classroom Interaction, 36(2), 17–28.
  • Luckin, R. (2008). The Learner Centric Ecology of Resources: a Framework for using Technology to Scaffold Learning. Computers & Education, 50, 449–462.
  • MacNeill, S., & Kraan, W. (2010). Distributed Learning Environments: A Briefing Paper. Retrieved from http://publications.cetis.ac.uk/wp-content/uploads/2011/02/Distributed_Learning.pdf
  • Martı́nez, A., Dimitriadis, Y., Rubia, B., Gómez, E., & de la Fuente, P. (2003). Combining qualitative evaluation and social network analysis for the study of classroom social interactions. Computers & Education, 41(4), 353–368. Retrieved from http://www.sciencedirect.com/science/article/pii/S0360131503000824
  • Martínez-Monés, A., Gómez-Sánchez, E., Dimitriadis, Y., Jorrín-Abellán, I. M., Rubia-Avi, B., & Vega-Gorgojo, G. (2005). Multiple Case Studies to Enhance Project-Based Learning in a Computer Architecture Course. IEEE Transactions on Education, 48(3), 482–489.
  • McKenney, S. (2013). Designing and researching technology-enhanced learning for the zone of proximal implementation. Research in Learning Technology, 21. doi:http://dx.doi.org/10.3402/rlt.v21i0.17374
  • Miles, M. B., & Huberman, M. (1994). Qualitative Data Analysis: An Expanded Sourcebook. Newbury Park, CA, USA: Sage Publications.
  • Oliver, M. (2000). An introduction to the Evaluation of Learning Technology. Educational Technology & Society, 3(4), 20–30.
  • Oliver, M., & Conole, G. (2004). Evaluating information and communications technology: a tool kit for practitioners. In R. Holliman & E. Scanlon (Eds.), Mediating Science Learning through Information and Communications Technology (pp. 139–150). Routledge.
  • Orlikowski, W. J., & Baroudi, J. J. (1991). Studying information technology in organizations: Research approaches and assumptions. Information Systems Research, 2(1), 1–28.
  • Pozzi, F., Manca, S., Persico, D., & Sarti, L. (2007). A general framework for tracking and analysing learning processes in computer-supported collaborative learning environments. Innovations in Education and Teaching International, 44(2), 169–179.
  • Prieto, L. P. (2012). Supporting orchestration of blended CSCL scenarios in Distributed Learning Environments. School of Telecommunications Engineering, University of Valladolid, Spain.
  • Prieto, L. P., Asensio-Pérez, J. I., Dimitriadis, Y., Gómez-Sánchez, E., & Muñoz-Cristóbal, J. A. (2011). GLUE!-PS: A multi-language architecture and data model to deploy TEL designs to multiple learning environments. In Proceedings of the European Conference on Technology-Enhanced Learning (EC-TEL 2011) (pp. 285–298).
  • Prieto, L. P., Asensio-Pérez, J. I., Muñoz-Cristóbal, J. A., Dimitriadis, Y., Jorrín-Abellán, I. M., & Gómez-Sánchez, E. (2013). Enabling Teachers to Deploy CSCL Designs Across Distributed Learning Environments. IEEE Transactions on Learning Technologies, 6(4), 324–336. doi:http://doi.ieeecomputersociety.org/10.1109/TLT.2013.22
  • Prieto, L. P., Asensio-Pérez, J. I., Muñoz-Cristóbal, J. A., Jorrín-Abellán, I. M., Dimitriadis, Y., & Gómez-Sánchez, E. (2014). Supporting orchestration of CSCL scenarios in web-based Distributed Learning Environments. Computers & Education, 73, 9–25. Retrieved from http://www.sciencedirect.com/science/article/pii/S0360131513003321
  • Prieto, L. P., Holenko-Dlab, M., Abdulwahed, M., Gutiérrez, I., & Balid, W. (2011). Orchestrating Technology Enhanced Learning: a literature review and a conceptual framework. International Journal of Technology-Enhanced Learning (IJTEL), 3(6), 583–598.
  • Prieto, L. P., Villagrá-Sobrino, S., Dimitriadis, Y., Schank, P., Penuel, W., & H., D. A. (2011). Mind the Gaps: Using Patterns to Change Everyday Classroom Practice Towards Contingent CSCL Teaching. In Proceedings of the International Conference of Computer-Supported Collaborative Learning (CSCL2011) (Vol. 1, pp. 518–525).
  • Roschelle, J., Dimitriadis, Y., & Hoppe, U. (2013). Classroom orchestration: Synthesis. Computers & Education, 69, 523–526. doi:http://dx.doi.org/10.1016/j.compedu.2013.04.010
  • Schön, D. A. (1983). The Reflective Practitioner - How Professionals Think in Action. Basic Books.
  • Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge Handbook of the Learning Sciences. Cambridge University Press.
  • Stake, R. (2010). Qualitative Research: Studying How Things Work. The Guilford Press.
  • Stake, R. E. (1983). Program Evaluation, Particularly Responsive Evaluation. In Evaluation Models (Vol. 6, pp. 287–310). Springer Netherlands. doi:10.1007/978-94-009-6669-7_17
  • Stake, R. E. (2004). Standards-based and responsive evaluation. Sage.
  • Strijbos, J. W., & Fischer, F. (2007). Methodological challenges for collaborative learning research. Learning and Instruction, 17(4), 389–393.
  • Sutherland, R., & Joubert, M. (2009). D1.1: The STELLAR vision and strategy statement. Retrieved from http://www.stellarnet.eu/kmi/deliverables/20090929_d1-1___vision-and-strategy.pdf
  • Treleaven, L. (2004). A new taxonomy for evaluation studies of online collaborative learning. In Online collaborative learning: Theory and practice (pp. 160–179). Idea Group Inc. (IGI).
  • Tsiatsos, T., Andreas, K., & Pomportsis, A. (2010). Evaluation Framework for Collaborative Educational Virtual Environments. Educational Technology & Society, 13(2), 65–77.
  • Vatrapu, R., Suthers, D. D., & Medina, R. (2008). Usability, sociability, and learnability: A CSCL design evaluation framework. In Proceedings of the 16th International Conference on Computers in Education. Taipei, Taiwan.
  • Villasclaras-Fernández, E., Hernández-Leo, D., Asensio-Pérez, J. I., & Dimitriadis, Y. (2013). Web Collage: An implementation of support for assessment design in CSCL macro-scripts. Computers & Education, 67, 79–97. doi:10.1016/j.compedu.2013.03.002
Data source: Dialnet