Data management in favor of teaching quality: learning analytics

As users of technology and as teachers in the area of educational technology, we use a multitude of platforms to support our daily work. Some of them are provided by the university where we work and others are external tools to which, with a greater or lesser awareness, we offer data with which in many cases we do not know what to do with them (Camino, 2022). When we work with our students, both in face-to-face and virtual classrooms, we are constantly exposed, all of us, to leaving a trace of our activity, but what can all this data offer us? How can we use it to improve our Teaching work?

To answer these questions and others, we present the research project “Use of learning analytics in university digital environments: impact on improving teaching performance” (LAxDigTeach – PID2020-115115GB-I00) coordinated by Montse Guitert Catasús from Universitat Oberta de Catalunya. The starting point is the definition made by Siemens (2013: 1382) about learning analytics as “the measurement, collection, analysis and report of data about students and their contexts, in order to understand and optimize the learning and the environments in which it occurs”. Another initial reference to the project has been Cerro’s doctoral thesis “Monitoring and evaluation of online collaborative activities through learning analytics. Analysis of the impact on teachers and students of higher education” (2021) where a tool based on web technologies for learning analytics and for teachers was designed and implemented (DIANA 2.0 for Moodle). Thus, the research project has the specific objective of exploring, defining, and adjusting learning analytics indicators and metrics, based on those defined in Cerro, Guitert and Romeu (2020); that report on the interaction processes of students in shared spaces during the teaching in digital environments.

In the main research team, there are researchers from the Open University of Catalonia and Edul@b research group, the University of the Basque Country – Euskal Herriko Unibertsitatea (Elkarrikertuz research group) and the University of Extremadura (CILEM research group). 6 universities participate in this research, 4 at the state level (University of La Laguna, University of Extremadura, Universitat Oberta de Catalunya, Universitat Rovira i Virgili) and 2 internationals (University of Padua, Federal University of Rio Grande do Sul). The methodology, based in design-based research, is divided into four phases: (1) Defining learning analytics technologies, (2) Learning analytics use cases, (3) Adjusting and validating learning analytics indicators and metrics, and (4) Recommendations and dissemination of results.

During the 2021-22 academic year, the metrics were validated, a plugin was created to use the tool on the Moodle platform and began the training of the teaching team that was going to participate. During the 2022-2023 academic year, the pilot phase is being carried out within some of the subjects that are taught in all the online master’s degrees about education and ICT in Spain with control and experimental groups. One of the cases at the University of Extremadura, has been carried out within the Master of Digital Education. Two subjects have participated in this master’s degree between September and December 2022 that have allowed the pilot phase. The first one has been “Curricular Integration of Educational Technology” where the group of 20 students has formed the control group, and the second “Research Perspectives in Educational Technology” where the group has become part of the experimental group.The coordination between both subjects in terms of design has been essential, they follow the same way of organizing student activities, based on a webquest and which also implies a collaborative discussion forum activity that serves as the axis for the development of the learning content.

---

 It is important to note that the objective in this pilot phase is focused on analyzing the communicative interaction of the students trying to understand the performance they do and thus be able to act in the teaching process, promoting learning and the acquisition of the competences defined in the training action.

---

In the following lines we would like to describe how the experience with the experimental group in the subject “Research Perspectives in Educational Technology” has been, showing some previous results offered by the DIANA 2.0 tool, waiting for a more exhaustive analysis in the subsequent months.

The subject is taught at the end of the first semester and lasts three and a half weeks of. In this subject the students make a first approach to the perspectives of research in educational technology. The students should make three tasks based on three readings. After these readings, they participated in an asynchronous debate forum activity to debate, reflect on, and contrast ideas through the interaction with the classmates that make possible to consolidate ideas and build knowledge to later apply them in the following two tasks in a little groups and individual way.

This debate has been divided into three discussion threads with a limited duration for each thread in which the following aspects have been addressed: (1) Identification of the main characteristics of the perspectives of research in education (December 5-9). (2) Pros and cons of each of the perspectives (December 10-14). (3) Conclusions: what is the perspective that can provide me a good development for the Master Essay (December 15-18). In order to carry out the analysis of the learning with the DIANA 2.0 tool, in relation to communicative interaction, each teacher must configure some initial parameters: the semantic field of the conversation (estimated words that should appear in the conversation); the degree of severity of the conversation; the minimum and maximum participation messages; the maximum degree of dispersion of the conversation and the maximum days of inactivity (Image 1).

Once these parameters have been defined, it is possible to carry out a preliminary analysis at any time of what is happening throughout the days in the debate. Consulting this pre-analysis regularly, as well as reading what is being discussed in the forum, facilitates the monitoring of the activity: it is possible to redirect it, encourage people to participate if they have not done and give individual feedback, among other options.

The tool allows to visualize in a different way the participation of the students in relation to different indicators such as: participation in the communicative interaction with the number of messages they contribute; promotion of dialogue and negotiation through the answers sent and the degree of popularity among colleagues; the communicative style considering the number of written words; the constancy and regularity in the group interaction throughout the days; and the exchange of information within the group with contributions from external links and/or attached documents (Image 2).

The tool offers different possibilities to visualize the data collected during the debate forum, in relation to the previously defined parameters (graphs, tables and word clouds for example). Beyond the numerical information offered, it is possible to carry out a transversal reading of what happens in the forum and the involvement that the students themselves show with the activity and in the communicative interaction. In conclusion, this pilot experience has enabled us:

---

1.- To have for the first time a graphic view of the communicative interaction.

2.- To do a most sustainable activity follow up reducing considerably the time dedicated to it by the teacher.

3.- To quickly identify the students who have not participated in the debate threads.

4.- To provide individualized feedback to the students on the participation they are carrying out, giving them the opportunity to improve in the following interventions and threads. This feedback has been received very positively by the students.

5.- To carry out a more objective and adjusted evaluation of the activity.

---

Although the design of the subject is reviewed annually making updates and adjustments to the tasks and readings, participation in the research project has made possible to think better about the design of the subject and the distribution of tasks, giving more importance to collaboration and teamwork.

References