Here, we present a protocol for the behavioral analysis of a project-based learning methodology for health sciences students (20-56 years old). The protocol facilitates the comparison of the participants’ performance in e-Learning versus blended-Learning (b-Learning) through a monitoring tool. The results are analyzed using Educational Data Mining and qualitative techniques.
Academic leaders all over the world are encouraging the use of active methodologies in teaching, especially in higher education. The reason for this is that social changes are happening at an ever-increasing rate, and they require students and teachers to develop digital skills. This is especially significant for health sciences degrees, in which future graduates must have effective problem-solving skills. To respond to this challenge, the use of a project-based learning (PBL) methodology, together with various monitoring techniques based on the use of Educational Data Mining (EDM) and mixed methods, will provide teachers with information about the effectiveness of the methodology and guide the implementation of personalized educational responses.
This study provides a protocol for the application of the PBL methodology in e-Learning and blended-Learning (b-Learning) teaching modalities for health sciences students studying occupational therapy in higher education. In addition, statistical techniques for the analysis of covariance and unsupervised learning allow differences to be detected between the two teaching modalities, thus specifying their effectiveness in terms of a range of variables related to behavioral patterns, performance, and satisfaction. Data visualization also helps in understanding the qualitative aspects of the learning process. These data will help teachers to produce more effective proposals for the implementation of the PBL methodology based on the based on the context of the teaching-learning process. Therefore, this protocol offers many resources and materials to help teachers implement the PBL methodology in e-Learning and b-Learning teaching methods.
Characteristics of the project-based learning methodology
Nowadays, professionals in various sectors face numerous (technical, sociopolitical, and economic) challenges arising from globalization in a variety of spheres, such as the environment1. These issues influence the global spread of diseases, thus diminishing resources, increasing poverty, and necessitating the responsibility to create sustainable societies1. Health experts have the potential and the skills to ensure the viability of communities by advancing care strategies and techniques on a large scale while contributing to the improvement of people's quality of life2. The United Nations has highlighted the need to address these challenges with the active participation of all countries in its Sustainable Development Goals (SDGs) 1. Health professionals' activities can be linked to many of the proposed goals. Therefore, those who train healthcare students must acquire pedagogical approaches that help students to gain the competencies that will allow them to face these new challenges1,2.
Among the various existing pedagogical methodologies, project-based learning (PBL)3 stands out as a methodology that helps students develop the competencies they will need to safely and effectively deal with the future challenges of their professions3. PBL is an innovative pedagogical approach that overcomes the limitations of traditional methodologies by transforming the teaching-learning process to make students the protagonists of their own learning. This change helps students acquire problem-solving skills in real-life contexts4. PBL is based on the teacher producing research questions5 that are included in practical scenarios related to the discipline. The students have to solve the problems in collaborative groups. The final objective is the acquisition of knowledge (conceptual, procedural, and attitudinal) through group work in the resolution of the task or problem6.
This approach originated in the fields of education and constructivist psychology, and from those beginnings, it has been adapted to different disciplines6. However, implementing this methodology can be affected by a lack of educational resources and infrastructure, and because of this, a hybrid model is sometimes used that combines traditional teaching with PBL7,8. However, these models have exhibited limitations in the learning experience1. For this reason, it is important to develop pedagogical protocols that guide and facilitate the use and interpretation of these methodologies. In addition, in recent years, the PBL methodology has been implemented through virtual learning platforms – Learning Management Systems (LMS)9 - in what is called Online Project-Based Learning (OPBL)10.
Project-based learning methodology for university students in the health sciences field
The protocol is important for helping teachers to apply these methodologies and interpret the results arising from the teaching and learning process9. The protocol can be used from the beginning of schooling to higher education and can be tailored to different mental, physical, and socioeconomic conditions11. The presented protocol focuses on working with university students in the health sciences. The teaching model is based on organizing learning around projects prepared by the teacher, who serves as an initial stimulus, and from this framework, the student becomes the main focus of their own learning11. The approach is an active, comprehensive, and constructive way for students to acquire basic competencies (conceptual, procedural, and attitudinal) that are closely related to their future professional practice4,12.
Learning methods are combined to stimulate the acquisition of cognitive and metacognitive competencies related to conceptual and procedural knowledge4,13,14, all of which contribute to students developing critical thinking, increased motivation toward learning, and autonomy in decision-making and problem-solving13. However, one of the keys to the successful application of this methodology is that the projects the teacher prepares must be directly related to real practice, and the students must be given autonomy in how they approach the problems based on minimum initial guidelines4,9. This means that the teachers need to clearly define the tools for evaluating the students' competencies and to give them guidance about the evaluation criteria that will be used to shape the PBL, as well as to assess how well the evaluation is done15,16. Furthermore, students benefit from collaborating with their peers to learn to argue and be assertive through debate, thus developing communication and teamwork skills4,16.
The main aim for university students is to "learn to learn" by dealing with the challenges the teacher sets to facilitate the acquisition of these competencies14 (which will later be implemented in the future working lives of the students). The results of using this methodology indicate that it encourages autonomy in learning and solving practical problems4. An added advantage of this methodology is that it is very effective in helping students acquire competencies, specifically in health sciences, in online teaching and blended-Learning (b-Learning)17,18,19. This has become especially important due to the COVID-19 pandemic20,21.
The common elements of PBL can be summed up as follows: (1) first, the concepts related to the project are explained to the students (in online teaching environments, flipped learning experiences can be used); (2) the project plan is defined and analyzed; (3) a review of the supporting theory is carried out, and a plan is drawn up to help understand the object of study; (4) objectives and questions are proposed for addressing the problem; (5) possible solutions are formulated, and the most viable options are evaluated; (6) these solutions are implemented; (7) the results are reported; (8) critical reflection is conducted, feedback is provided, and the process is evaluated, including suggesting new questions; and (9) the process is finished by presenting the work4,6,13.
It should also be kept in mind that not all students respond in the same way to this type of methodology. Students who are more actively involved in their learning, those who are more critical thinkers, and those who have better interpersonal relationship skills tend to achieve better results22. It is also possible that students who are used to traditional learning methodologies may feel frustrated. This is why students should be given clear working rules and a timetable for the implementation of the project phases from the beginning8. As noted above, PBL has been widely used within health sciences degrees, and there is extensive evidence supporting it in the literature18,23,24. However, some aspects that could be improved have been identified, which are related to clinical skills seminars and the dissemination of experience in the scientific community25,26.
This methodology seems to have better results for degrees with a higher proportion of internships (for example, nursing vs. medicine, occupational therapy versus psychology or nutrition)3. In addition, recent studies have suggested implementing PBL methodology through joint training projects between different degrees to work on interaction in real contexts. The objective would be the acquisition of multidisciplinary and interdisciplinary competencies27,28. However, to implement this type of methodology well, the teaching staff must be qualified to implement it, and the students must develop critical and divergent thinking with respect to clinical practice, especially in the case of health sciences3,15,18,29.
Another important aspect of the implementation is the need to evaluate student satisfaction with the PBL methodology training and analyze their ideas for improvement19,30,31. More specifically, it has been reported that occupational therapy students have found this methodology essential for their learning as it allows them to learn how to solve similar problems to those they will face in their professional lives, which enhances their motivation31. Similarly, a longitudinal study that began in 1993-in which a hybrid model of PBL was evaluated in occupational therapy students-demonstrated that the students had very positive opinions of group learning for their future professional practice7. Another study on occupational therapy students found a high degree of satisfaction with the methodology. This is because students think that this way of learning helps them acquire knowledge through practice and enables them to respond appropriately in their future work13.
In summary, using the PBL methodology in health sciences courses is key to students' acquisition of professional competencies. Various studies have indicated the steps to follow to prioritize the interventions and the design of the materials to achieve effective, satisfactory learning. Therefore, it is very important to monitor students' learning processes to detect early or potential problems with learning and address them. Educational Data Mining (EDM) and mixed-methods analysis techniques should be used for monitoring, the essential fundamentals of which are discussed below.
Educational Data Mining techniques
When LMSs are used in either e-Learning or b-Learning teaching models, a series of logs or log files are generated that capture each of the student and teacher interactions. These logs are evidence of the learning behaviors and can be analyzed using EDM or Learning Analytics (LA)32 techniques. These techniques allow the teacher to learn what student interaction exists, how much interaction exists, and what is the quality of the interaction33. In particular, EDM techniques make it easier to discover students' learning behavior patterns and their interactions4. EDM techniques can be used to study different profiles35,36 (oriented toward students or oriented toward educators).
The final objective is to provide feedback for the instruction, evaluate the course content structure, analyze elements that have been effective in the learning processes, classify the type of students, and identify needs for the guidance and monitoring of learning. This helps in determining each student's most common learning patterns and the frequency of errors, which can then be used to tailor the most appropriate educational responses. One of the things that EDM techniques can be used for is monitoring students' learning processes to offer appropriate help via mentoring. EDM techniques include supervised learning techniques (prediction or classification techniques)37 and unsupervised learning techniques36 (clustering techniques)38. Using EDM techniques in teaching processes that include active teaching methodologies, such as PBL, has been shown to be very effective for studying individual student behavior and student behavior in collaborative groups10.
The application of computer-assisted qualitative analysis techniques
In the last two decades, quantitative and qualitative techniques, or a combination of the two, have been applied in research in what has been called mixed methods39. The use of mixed methods for data analysis is especially useful when dealing with complex problems of analysis, such as monitoring students' learning behaviors. The use of these methods allows for the transformation of qualitative data, such as student feedback from open-ended questions in a survey, into qualitative data, and the analysis of the data with different data visualization techniques40. A variety of computer-assisted (or -aided) qualitative data analysis software (CAQDAS) can be used to facilitate data transformation and analysis41.
A summary of the procedure for teaching through the PBL methodology, the analysis of learning behaviors, the use of EDM, and computer-based qualitative analysis techniques is shown in Figure 1.
Figure 1: PBL work and the use of EDM and computer-based qualitative analysis techniques. Data collection and processing applying EDM and text mining techniques in e-Learning and b-Learning teaching environments. Abbreviations: PBL = project-based learning; EDM = Educational Data Mining; DB = database; e-Learning = online classes; b-Learning = blended classes (online and physical classes). Please click here to view a larger version of this figure.
Three research questions were posed in this study: RQ1: Are there significant differences in the learning outcomes and satisfaction of health sciences students studying occupational therapy depending on whether the PBL methodology is implemented via e-Learning versus b-Learning, considering the effects of students' prior knowledge? RQ2: Do the clusters of participants found match the learning outcomes, learning behaviors, and perceived satisfaction as a function of teaching modality (e-Learning vs. b-Learning)? RQ3: Are the students' suggestions for the improvement of the PBL methodology different depending on the teaching modality, e-Learning versus b-Learning?
The following protocol can be used by teachers in healthcare and can also be modified to work with students in other knowledge areas.
This protocol was executed in compliance with the procedural regulations of the Bioethics Committee of the University of Burgos (Spain) number IO 03/2022. Before participating, the respondents were made fully aware of the research objectives and provided their informed consent. They received no financial compensation for their participation.
1. Participant recruitment
Table 1: Sample characteristics. Sample characteristics (age, gender) in both groups (e-Learning teaching method vs. b-Learning teaching method). Abbreviations: e-Learning = online classes; b-Learning = blended classes (online and physical classes); M age =Mean age; SD age=Standard deviation age. Please click here to download this Table.
2. Experimental procedure
Table 2: Prior knowledge questionnaire13. Open and closed questions from the prior knowledge questionnaire. Please click here to download this Table.
Table 3: Scale of satisfaction with PBL13. Open and closed questions from the PBL satisfaction scale. Abbreviation: PBL = project-based learning. Please click here to download this Table.
Figure 2: Analysis of a single student's behavior on the UBUVirtual platform. Heat map of the behaviors carried out by the student on the virtual learning platform. Please click here to view a larger version of this figure.
Figure 3: Analysis of all student behaviors on the UBUVirtual platform. Analysis of the learning behaviors performed by the students on the virtual learning platform with respect to task completion. Please click here to view a larger version of this figure.
Figure 4: Analysis of student behavior in a group on the UBUVirtual platform. Analysis of the learning behaviors performed by the students in work groups on the virtual learning platform with respect to the completion of tasks. Please click here to view a larger version of this figure.
The 98 participants recruited for the present study were undergraduate occupational therapy students aged between 21 years old and 56 years old (Table 1). The protocol was tested over 2 academic years at the University of Burgos. The delivery of the protocol is summarized in Table 4.
Table 4: Summary of the application of the protocol for project-based learning for university students. Abbreviation: LMS = Learning Management Systems. Please click here to download this Table.
The first research question (RQ1) was as follows: are there significant differences in the learning outcomes and satisfaction of health sciences students studying occupational therapy depending on whether the PBL methodology is implemented via e-Learning versus b-Learning, considering the effects of students' prior knowledge?
As Table 5 shows, there were significant differences in the elaboration (LOEPBL) scores, with the e-Learning group scoring higher. The b-Learning group scored higher in presentation (LOEXPBL) and LMS access (LMSA). No significant differences were found in the total learning outcomes (LOT), and no effect was found for prior knowledge as the covariate.
Table 5: ANCOVA of a fixed effects factor (teaching modality, e-Learning vs. b-Learning), the covariate (level of prior knowledge), and the effect value, η2. ANCOVA of a fixed effects factor (teaching modality, e-Learning vs. b-Learning), the covariate (level of prior knowledge), and the effect value, η2. Abbreviations: PBL = project-based learning; LOEPBL = elaboration of PBL; LOEXPBL = presentation of PBL; LOT = total learning outcomes; LMS = Learning Management Systems; LMSA = LMS access; SPBL = satisfaction with PBL; e-Learning = online classes; b-Learning = blended classes (online and physical classes); M = mean; SD = standard deviation; ANCOVA = analysis of covariance. Please click here to download this Table.
The second research question (RQ2) was as follows: do the clusters of participants found match the learning outcomes, learning behaviors, and perceived satisfaction as a function of teaching modality (e-Learning vs. b-Learning)?
Two clusters were found in which differences were detected in the different variables, but it could not be established that one was superior to the other in all the variables (Table 6).
Table 6: Final cluster centers. Abbreviations: PBL = project-based learning; LOEPBL = elaboration of PBL; LOEXPBL = presentation of PBL; LOT = total learning outcomes; LMS = Learning Management Systems; LMSA = LMS access; SPBL = satisfaction with PBL. Please click here to download this Table.
Subsequently, a cross table was prepared between the values of the clusters of belonging assigned to each participant with respect to the teaching modality variable (e-Learning vs. b-Learning) (Table 6) and the percentage of belonging to each of the groups. A contingency coefficient of C = 0.40, p < 0.001, was obtained (Table 7).
Table 7: Cross-tabulation of participant by cluster number Abbreviations: e-Learning = online classes; b-Learning = blended classes (online and physical classes). Please click here to download this Table.
In addition, an ANOVA was performed between clusters for the variables of elaboration (LOEPBL), presentation (LOEXPBL), total learning outcomes (LOT), LMS access (LMSA), and students' satisfaction with teaching (SLPBL). Significant differences were found in presentation (LOEXPBL, p = 0.03) and LMS access (LMSA, p < 0.001) (see Table 8).
Table 8: ANOVA clusters. Abbreviations: PBL = project-based learning; LOEPBL = elaboration of PBL; LOEXPBL = presentation of PBL; LOT = total learning outcomes; LMS = Learning Management Systems; LMSA = LMS access; SPBL = satisfaction with PBL; df = degrees of freedom; ANOVA = analysis of variance. Please click here to download this Table.
A visual cluster analysis was performed by means of the k-means technique using visualization software43, taking teaching modality, e-Learning versus b-Learning, as a variable with respect to the different variables studied: elaboration (LOEPBL), presentation (LOEXPBL), total learning outcomes (LOT), access to the LMS (LMSA), and student satisfaction with teaching (SPBL) (Figure 5).
Figure 5: Cluster analysis of the variables LOEPBL, LOEXPBL, LOT, LMSA, and SPBL with respect to the variable "Teaching modality, e-Learning versus b-Learning". Visualization of the clusters found with respect to the variables LOEPBL, LOEXPBL, LOT, LMSA, and SPBL. Abbreviations: PBL = project-based learning; LOEPBL = elaboration of PBL; LOEXPBL = presentation of PBL; LOT = total learning outcomes; LMS = Learning Management Systems; LMSA = LMS access; SPBL = satisfaction with PBL; e-Learning = online classes; b-Learning = blended classes (online and physical classes). Please click here to view a larger version of this figure.
Next, a heat map was constructed for the behavior of the variables in the clusters. Greater discrimination was found in cluster 1 (the students included in this cluster had less access to the virtual learning platform) and cluster 2 (the students included in this cluster had more access to the virtual learning platform) in the behaviors of students on the virtual learning platform (UBUVirtual is a virtual learning platform based on Moodle) and in the type of teaching (e-Learning vs. b-Learning) (Figure 6).
Figure 6: Heat map visualization of the clusters in the variables LOEPBL, LOEXPBL, LOT, LMSA, and SPBL. Abbreviations: PBL = project-based learning; LOEPBL = elaboration of PBL; LOEXPBL = presentation of PBL; LOT = total learning outcomes; LMS = Learning Management Systems; LMSA = LMS access; SPBL = satisfaction with PBL. Please click here to view a larger version of this figure.
The third research question (RQ3) was as follows: are the students' suggestions for the improvement of the PBL methodology different depending on the teaching modality (e-Learning vs. b-Learning)?
The answers were categorized into two groups, the e-Learning versus b-Learning groups, using the qualitative data analysis software44. The frequencies by categorization code are shown in Table 9, and Figure 7 shows an analysis of the students' responses in the two groups by applying a Sankey diagram.
Table 9: Frequency analysis of the categorized responses of the students in the two intervention groups (Group 1, e-Learning teaching, and Group 2, b-Learning teaching). Abbreviations: e-Learning = online classes; b-Learning = blended classes (online and physical classes). Please click here to download this Table.
Figure 7: Sankey plot of the categorized responses in the two intervention groups (Group 1, e-Learning teaching, and Group 2, b-Learning teaching). Sankey plot of the categorized responses in the two intervention groups. Abbreviations: e-Learning = online classes; b-Learning = blended classes (online and physical classes). Please click here to view a larger version of this figure.
How do these results indicate that the suggestions for the improvement of the teaching were different depending on whether the teaching was via e-Learning or b-Learning? The students in both groups agreed that the teaching had been satisfactory. Nonetheless, the b-Learning group noted the importance of using the simulation center and their desire for real practice with children, thereby increasing the teaching in classroom practice. The teaching modality used (e-Learning vs. b-Learning) led to differences in the results of applying the PBL methodology.
More specifically, the students in the e-Learning group scored higher in the execution of the project, while the b-Learning group scored higher in presenting the project. There was also increased access to the virtual platform in the b-Learning group. Two clusters were found, with cluster 2 showing better results in all the variables studied except for two (total learning outcomes, for which the results were the same between the two groups, and satisfaction with the teaching methodology, for which the results were very similar in both groups).
Similarly, in the e-Learning group, 82.6% of the students were in cluster 1, and 17.39% were in cluster 2. In the b-Learning group, 59.62% were in cluster 2, and 40.38% were in cluster 1. Therefore, it can be concluded that the teaching modality used seems to have an effect on the student learning outcomes and student activity in the LMS, with b-Learning being more effective than e-Learning in project presentation and student activity in the LMS. In contrast, e-Learning seems to be more effective with regard to creating the project. Nonetheless, the PBL methodology is very effective in both cases since the means of the final learning outcomes were high in both groups with very low variability (e-Learning mean = 8.89, standard deviation = 0.48; b-Learning mean = 8.65, standard deviation = 0.92). In addition, satisfaction was very high in both modalities, with no significant differences between the groups.
These results support the first question from RQ1 and indicate the directions for future research to examine these aspects more deeply. The differences noted above can also be seen by analyzing the qualitative data regarding student satisfaction within the two teaching modalities. It is important to remember that the e-Learning was delivered at the most critical point in the state of emergency due to COVID-19, during which neither in-person nor blended teaching could be carried out. The students' evaluations of how the PBL methodology was implemented in the online mode showed that it was very satisfactory, as it allowed them to continue their training in a way that was close to normal and to relate it to their professional practice. This may be why the e-Learning group did not make many suggestions for improvements. In contrast, the b-Learning group, who were taught during the second year of the pandemic, felt that the PBL methodology imparted via the LMS and in the face-to-face simulation center was beneficial for their training. However, this group suggested many more ideas for improvements to the teaching, which were mainly focused on increasing the in-person classes, specifically in terms of practical activities with real users.
Supplemental Material Appendix 1. This includes Table 1 and Table 2. (Table 1) Document including the relationship between metacognitive strategies, sub-strategies, and the activities to develop them. Project-based learning is described from the current state of the field. In addition, information is presented on the relationship between the steps of PBL and the development of different metacognitive strategies. This is a very useful document for teachers who want to apply this educational methodology. (Table 2) Rubric for the evaluation of the project resolution process. A rubric is presented for monitoring the project development process with respect to the use of different metacognitive strategies. Please click here to download this File.
Supplemental Material Appendix 2: This includes Table 1 and Table 2. (Table 1) Rubric for the evaluation of the presentation of the PBL. A rubric is presented for the evaluation of the project development process with respect to the presentation and defense of the project. (Table 2) Rubric for the evaluation of the elaboration of the PBL. A rubric is presented for the assessment of the project development process with respect to project evaluation. Please click here to download this File.
Supplementary Material Appendix 3: An example of a case study. Please click here to download this File.
Supplementary Material Appendix 4: Examples of work produced by the students. Please click here to download this File.
Supplementary Material Appendix 5: Examples of the virtual laboratories developed. Please click here to download this File.
This study leads to the conclusion that the teaching modality (e-Learning vs. b-Learning) may affect the results in different elements of PBL17,18. In future studies, this aspect will be explored in greater depth to see if the same pattern is found with students from other (particularly health sciences) courses, since this is the subject of this protocol. In contrast, no differences were found in the total learning outcomes or in student satisfaction with the teaching methodology. This is in line with results from other studies4,13,14,15,18,29.
In summary, the use of a PBL teaching methodology has been shown to be effective both for students' learning outcomes and their motivation for learning, which is consistent with the findings from other studies2,3,4. However, the students in the b-Learning group expressed an interest in more in-person teaching. This aspect supports a hybrid use of the methodology8. Similarly, the students in both groups appreciated that this methodology guides them toward the practical aspects of their profession4,13. It was also found that the social situation in which the teaching was delivered affected the students' satisfaction and critical reflections about their learning. The learning was more participative in the second part of the pandemic, where there were greater possibilities for in-person learning4,14,15,18,29. Not all students were found to respond in the same way or experience the same amount of participation22. This is an important aspect to be considered by teachers with respect to using this methodology, and supporting the methodology requires teachers with digital competencies along with the use of EDM techniques32,33,36 and the analysis of students' open responses using mixed techniques39,40,41.
Finally, one limitation of this study is related to the PBL methodology, which was used only with students studying for a single degree-occupational therapy. The methodology was not implemented with multi- or interdisciplinary teaching designs31.
The authors have nothing to disclose.
The study was carried out as part of the research project "SmartLearnUni", funded by the Spanish Ministry of Science and Innovation 2020 I+D+i Projects – RTI Type B. Reference: PID2020-117111RB-I00. The authors also gratefully acknowledge the collaboration of the health sciences students at the University of Burgos, especially the students studying for their Occupational Therapy and Nursing degrees.
Atlas.ti v.9 | Atlas.ti | ||
Orange v. 3.30 | Orange | ||
SPSS v.24 | SPSS | ||
UBUVirtual | UBU |