This protocol guides researchers and educators through implementation of the Problem-Solving before Instruction approach (PS-I) in an undergraduate statistics class. It also describes an embedded experimental evaluation of this implementation, where the efficacy of PS-I is measured in terms of learning and motivation in students with different cognitive and affective predispositions.
Nowadays, how to encourage students’ reflective thinking is one of the main concerns for teachers at various educational levels. Many students have difficulties when facing tasks that involve high levels of reflection, such as on STEM (Science, Technology, Engineering and Mathematics) courses. Many also have deep-rooted anxiety and demotivation towards such courses. In order to overcome these cognitive and affective challenges, researchers have suggested the use of “Problem-Solving before Instruction” (PS-I) approaches. PS-I consists of giving students the opportunity to generate individual solutions to problems that are later solved in class. These solutions are compared with the canonical solution in the following phase of instruction, together with the presentation of the lesson content. It has been suggested that with this approach students can increase their conceptual understanding, transfer their learning to different tasks and contexts, become more aware of the gaps in their knowledge, and generate a personal construct of previous knowledge that can help maintain their motivation. Despite the advantages, this approach has been criticized, as students might spend a lot of time on aimless trial and error during the initial phase of solution generation or they may even feel frustrated in this process, which might be detrimental to future learning. More importantly, there is little research about how pre-existing student characteristics can help them to benefit (or not) from this approach. The aim of the current study is to present the design and implementation of the PS-I approach applied to statistics learning in undergraduate students, as well as a methodological approach used to evaluate its efficacy considering students’ pre-existing differences.
One of the questions that teachers are most concerned about currently is how to stimulate students' reflection. This concern is common in courses of a mathematical nature, such as STEM courses (Science, Technology, Engineering and Mathematics), in which the abstraction of many concepts requires a high degree of reflection, yet many students report approaching these courses purely through memory-based methods1. In addition, students often show superficial learning of the concepts1,2,3. The difficulties that students experience applying reflection and deep learning processes, however, are not only cognitive. Many students feel anxiety and demotivation faced with these courses4,5. In fact, these difficulties tend to persist throughout students' educations6. It is therefore important to explore educational strategies that motivationally and cognitively prepare students for deep learning, regardless of their differing predispositions.
It is particularly useful to find strategies that complement typical instructional approaches. One of the most typical being direct instruction. Direct instruction means fully guiding students from the introduction of novel concepts with explicit information about these concepts, then following that with consolidation strategies such as problem-solving activities, feedback, discussions, or further explanations7,8. Direct instruction can be effective for easily transmitting content8,9,10. However, students often do not reflect on important aspects, such as how the content relates to their personal knowledge, or potential procedures that could work and do not11. It is therefore important to introduce complementary strategies to make students think critically.
One such strategy is the Problem-Solving before Instruction (PS-I) approach12, also referred to as the Invention approach11 or the Productive Failure approach13. PS-I is different to direct instruction in the sense that students are not directly introduced to the concepts, instead there is a problem-solving phase prior to the typical direct instruction activities in which students seek individual solutions to problems before getting any explanation about procedures for solving them.
In this initial problem, students are not expected to fully discover the target concepts13. Students may also feel cognitive overload14,15,16 and even negative affect17 with the uncertainty and the many aspects to consider. However, this experience can be productive in the long term because it can facilitate critical thinking about important features. Specifically, the initial problem can help students to become more aware of the gaps in their knowledge18,activate prior knowledge related to the content to cover13, and increase motivation because of the opportunity to base their learning on personal knowledge7,17,19.
In terms of learning, the effects of PS-I are generally seen when the results are evaluated with deep learning indicators20,21. In general no differences have been found between students who learned through PS-I and those who learned through direct instruction in terms of procedural knowledge20,22, which refers to the ability to reproduce learned procedures. However, students who go through PS-I generally exhibit higher learning in conceptual knowledge7,19,23, which refers to understanding the content covered, and transfer7,15,19,24, which refers to capacity to apply this understanding to novel situations. For example, a recent study in a class about statistical variability showed that students who were given the opportunity to invent their own solutions to measure statistical variability before receiving explanations about the general concepts and procedures in this topic demostrated better understanding at the end of the class than those who were able to directly study the relevant concepts and procedures before getting involved in any problem-solving activity23. However, some studies have shown no differences in learning16,25,26 or motivation19,26 between PS-I and direct instruction alternatives, or even better learning in direct instruction alternatives14,26, and it is important to consider potential sources of variability.
The design features underlying the implementation of PS-I are an important feature20. A systematic review20 found that there was more likely to be a learning advantage for PS-I over direct instruction alternatives when the PS-I interventions were implemented with at least one of two strategies, either formulating the initial problem with contrasting cases, or building the subsequent instruction with detailed feedback about the students' solutions. Contrasting cases consist of simplified examples that differ in a few important characteristics11 (see Figure 1 for an example), and can help students identify relevant features and evaluate their own solutions during the initial problem11,20. The second strategy, providing explanations that build on the students' solutions13, consist of explaining the canonical concept while giving feedback about the affordances and limitations of solutions generated by students, which can also help students focus on relevant features and evaluate the gaps in their own knowledge20, but after the initial problem-solving phase is completed (see Figure 3 for an example of the scaffolding from students' typical solutions).
Given the support in the literature for these two strategies, contrasting cases and building instruction on students' solutions, it is important consider them when promoting the inclusion of PS-I in real educational practice. This is the first goal of our protocol. The protocol provides materials for a PS-I intervention that incorporate these two principles. It is a protocol that, while adaptable, it is contextualized for a lesson on statistical variability, a very common lesson for university and high school students, who are generally the target populations in the literature on PS-I29. The initial problem-solving phase consists of inventing variability measures for income distributions in countries, which is a controversial topic30 that may be familiar to students in many learning areas. Then materials are provided for students to study solutions to this problem in a worked example, and for a lecture that incorporates discussion of common solutions produced by students along with embedded practice problems.
The second goal of our protocol is to make the experimental evaluation of PS-I accessible to educators and researchers, which can facilitate the investigation of PS-I from a greater variety of perspectives while maintaining some conditions constant across the literature. Yet conditions of this experimental evaluation are flexible to modifications. The experimental evaluation described in the protocol can be applied in ordinary lessons, since students in a single class can be assigned the materials for the PS-I condition or the materials for a direct instruction condition at the same time (Figure 4). This direct instruction condition is also adaptable to research and education needs, but as originally described in the protocol students start by getting the initial explanations about the target concept with the worked example, and then consolidate this knowledge with a practice problem (only presented in this condition to compensate for the time PS-I students spend on the initial problem), and with the lecture23. Potential adaptations include starting with the lecture and then having students to do the problem-solving activity, which is a typical control condition for comparing PS-I that has often led to better learning for the PS-I condition7,13,19,26. Alternatively, the control condition can be reduced to the exploration of a worked example followed by the lecture phase, which, although a more simplified version of direct instruction approaches than originally proposed, is more common in the literature and has led to varied results, with some studies indicating better learning in PS-I15,24, and others indicating better learning from this type of direct instruction condition14,26.
Finally, a third goal of the protocol is to provide resources for evaluating how students with different predispositions and cognitive abilities can benefit from PS-I15. The evaluation of these predispositions is especially important if we consider the negative predispositions that some students often have with STEM courses, and the fact that PS-I can still produce negative reactions in some cases14. There is, however, little research on this.
On the one hand, since PS-I facilitates the association of learning with individual ideas, rather than just formal knowledge, PS-I can be hypothesized as being able to help motivate students from low academic levels, those who have low feelings of competence, or low motivation about the subject13,27. One study showed that students with low mastery orientation, i.e., fewer goals related to personal learning, benefited more from PS-I than those with higher motivation to learn27. On the other hand, students with other profiles might encounter difficulties when involved in PS-I. More specifically, metacognition plays an important role in PS-I31, and students with low metacognition skills might not benefit from PS-I due to difficulties in being aware of their knowledge gaps or discerning relevant content15. In addition, as the initial phase of PS-I is based on the production of individual solutions, students with low divergent abilities, difficulties generating a variety of responses in a given situation, might benefit less from PS-I than other students. The protocol presents reliable instruments to assess for these predispositions (Table 1) although others may be considered.
In summary, this protocol aims to make an implementation of a PS-I intervention that follows accepted principles in the PS-I literature accessible to educators and researchers. Additionally, the protocols provide an experimental evaluation of this intervention, and facilitate the evaluation of students' cognitive and motivational predispositions. It is a protocol that does not require access to new technologies or specific resources, and one that can be modified based on research and educational needs.
This protocol follows the Helsinki Declaration of Ethical Principles for Research with Humans, but applies these principles to the added difficulties of integrating research within real-life settings in education32. Specifically, neither the assignment of learning conditions nor the decision to participate can have consequences for students' learning opportunities. In addition, confidentiality and the anonymity of students is maintained even when it is the teachers who are in charge of the evaluation. The aims, scope, and procedures of the protocol have been approved by the Research Ethics Committee of the Principality of Asturias (Spain) (Reference: 242/19).
Please note that if the user is only interested in implementing the PS-I approach, only Step 6 (without assigning participants to the control condition) and Step 7 are relevant. Despite that, Steps 5 and 9 can be added as practice exercises for students. If the user is also interested in the experimental evaluation, it is important that students work individually during Steps 4, 5, 6, and 9. It is therefore recommended that during these steps, student seating is arranged so that there is an empty space beside each student.
Depending on convenience, the steps can be implemented continuously within a single class session or with subsequent steps in a different class session.
1. Information for students about the purpose and procedures of the study
2. Providing students with an identification number disassociated from other records
3. Completion of questionnaires about cognitive and affective predispositions and basic demographic data
4. Administration of the divergent thinking test
5. Completion of the pre-test of previous academic knowledge
6. Assignment to and administration of the two learning conditions
7. Administration of the lecture content
8. Completion of the curiosity questionnaire
9. Administration of the learning post-test
10. Providing students with feedback and all learning materials
11. Coding the data
12. Analysis of the data
Please note that references in this section refer to practical manuals on how to perform the analyses with SPSS and PROCESS software but other programs may also be used.
This protocol was satisfactorily implemented in a previous study23, with the exception of the measures of students' predispositions in terms of their sense of competence, mastery approach goals, metacognition, and divergent thinking.
To address these predispositions, this protocol includes measures that have been previously validated and that have shown high levels of reliability (Table 1).
Typical solutions generated by students in the invention problem of the PS-I condition can be seen in Figure 3A-D. Students do not usually produce the canonical solution of standard deviation. However, the sub-optimal solutions they do produce reveal reflection about relevant aspects of standard deviation (e.g., range, summing deviations, or averaging deviations). Previous research has shown that the variety of solutions in the initial problem in PS-I was associated with higher learning, regardless of the correctness of the response44. Nonetheless, it is important to note that the absence of response in this problem is not an indicator of students not benefiting from it, since students can critically reflect about the problem without producing a visible result.
A typical solution produced by students in the practice problem used in the control condition (Figure 2) is shown in Figure 3 E. These solutions are more homogeneous and in line with the canonical concept of standard deviation because it is a problem that was presented after they had studied the concepts and procedures in the Worked Example (Appendix D-2).
Figure 5 reproduces an example for reporting the general differences between PS-I and direct instruction in the experimental evaluation. It is based on results of a previous study that followed this protocol23 in which students in the PS-I condition did not differ in procedural knowledge, transfer of knowledge, curiosity, or previous knowledge, but did differ in conceptual knowledge.
Figure 6 shows an example for reporting the moderating effect of one of the proposed student predispositions, metacognitive abilities. In this hypothetical example, students with lower metacognitive abilities learned more from direct instruction than from PS-I, while those with higher metacognitive abilities benefited more from PS-I than from direct instruction.
Figure 1: Invention Problem in the PS-I Condition. In this problem23 students in the PS-I condition are asked to invent quantitative indexes to measure inequality across the four countries. It is formulated with the technique of Contrasting Cases11: the countries show consistencies and variations regarding the relevant features, and these variations are easy to calculate. For example, Pinpanpun and Toveo have the same mean (5), same number of cases (7), same range (10), but different distribution. Please click here to view a larger version of this figure.
Figure 2: Practice Problem in the Direct Instruction Condition. In this problem23 students in the direct instruction condition are asked to apply the concepts and procedures learned in the Worked Example. Please click here to view a larger version of this figure.
Figure 3: Common Solutions in the Invention Problem and in the Practice Problem. Images A-D show common solutions in the Invention Problem, which can be used in the posterior direct instruction phase to scaffold contents: (A) The range – easy to calculate, but does not account for differences across all inhabitants-; (B) Range based measure – considers more inhabitants than the range as it becomes amplified when maximums values are repeated, but does not consider all values-; (C) Average of deviations – it accounts for differences across all inhabitants, but it is confusing because negative deviations subtract from positive deviations-; (D) Average of absolute deviations -a conceptually complete solution similar to the canonical solution of the standard deviation-; (E) A typical solution to the practice problem of the control condition. Students in this condition have already studied the Worked Example, and therefore most of them are able to reproduce and interpret correctly the canonical solutions of the standard deviation. Please click here to view a larger version of this figure.
Figure 4: Design of Experimental Evaluation. After the completion of the questionnaires and tests to measure students' predispositions, students are randomly assigned to the activities of the two learning conditions (all students remain in the same class). Once students complete these activities, all of them receive the same lecture about statistical variability. Curiosity and learning are measured at the end of the learning process. Please click here to view a larger version of this figure.
Figure 5: Results about Efficacy of PS-I versus Direct Instruction. The graphics display a typical result of the comparison between the PS-I condition and the direct instruction condition within each dependent variable, using data of a previous study that used this protocol23. The two bars in each graphic represent the means for the two conditions, while their corresponding error bars represent +/- 1 standard errors of those means. * indicates significant results at the .05 significance level. Please click here to view a larger version of this figure.
Figure 6: Hypothetical Results about the Moderating Effects of Students' Predispositions. The graphics display an hypothetical result about the moderating effect of metacognitive abilities on the relative efficacy of PS-I to promote learning, in which PS-I is more effective than direct instruction only for students who report medium and high metacognitive abilities. Following recommendations in43, the 16th , 50th, and 86th percentiles have been used to respectively represent students with low, medium, and high metacogntive abilities. Please click here to view a larger version of this figure.
Construct | Measure and Description | ||
Sense of Competence | The Cognitive Competence Scale in the Survey of Attitudes towards Statistics (SATS-28)33 can be used (Appendix B2). It is composed of 6 items that ask students how much they agree with statements about their sense of competence learning statistics (e.g. “I can learn statistics”). It has shown internal, convergent and predictive validity, and high reliability (α = .71 – .93)45. | ||
Mastery Approach Goals | The Mastery Approach Scale in the Achievement Goal Questionnaire-Revised 34 can be used (Appendix B3). It is composed of 3 items that ask students how much they agree with statements about having learning goals that focus on personal learning (e.g., “I am striving to understand the content of this course as thoroughly as possible”). It has shown internal, convergent and predictive validity, and high internal reliability (α =.84)34. | ||
Metacognitive Regulation | The Regulation of Cognition Scale of the Metacognitive Awareness Inventory46 can be used (Appendix B4). It consists of 35 items that ask students how typical it is for them to use different metacognitive strategies (e.g., “I reevaluate my assumptions when I get confused”). It has shown internal and predictive validity, and high reliability (α = .88)46. | ||
Divergent Thinking | The Fluency score from the Alternative Uses Task36 can be used. It consists of presenting students with several objects (e.g., a paper clip), and asking them to provide as many uncommon uses for each object within a given time. It is a reliable score (H = .631) that has been internally validated 47 and has shown predictive validity in versions with different extensions, varying between 1 to 20 objects presented, and between 1 to 3 minutes given for each object37,48,49. For time restrictions within educational settings, a short version of two objects and two minutes per object37 is proposed in this protocol. | ||
Previous Academic Knowledge | To adapt to the specific contents covered in this protocol, a learning pre-test has been adapted (Appendix C) from a reliable (α =.75) pre-test used in a previous study 7. It consists of 5 items that ask students about central tendency measures that are relevant to the assimilation of variability contents. |
Table 1: Proposed Constructs and Measures to Evaluate Students' Predispositions. Five constructs about students' predispositions are proposed to be evaluated as moderators in the efficacy of PS-I. A proposed measure for each construct is described regarding the number of items, description of the items, and evidence about validity and reliability.
Construct | Measure and Description | ||
Curiosity | The Curiosity Scale in the Epistemically-Related Emotions Questionnaire40 can be used (Appendix F). It consists of three items that ask students to rate the intensity they felt curious, interested, and inquisitive. It has shown internal and predictive validity, and high reliability (α = .88)40. | ||
Learning (procedural, conceptual, and transfer) | To evaluate learning about the specific variability contents covered in this protocol, a learning post-test has been adapted (Appendix G) from a reliable (α =.84) post-test used in a previous study7. It consists of 12 items: three items referred to procedural learning (e.g., item 1 where students have to calculate the standard deviation), six items referred to conceptual learning (e.g., item 4 where students have to reason about components of the standard deviation formula), and three items referred to transfer (e.g., item 10 where students have to infer a procedure to compare scores with different measurements). |
Table 2: Proposed Constructs and Measures to Evaluate the efficacy of PS-I. The proposed instruments to measure curiosity and three types of learning (procedural, conceptual, and transfer) are described, including information about number of items, description of the items, and evidence about validity and reliability.
The aim of this protocol is to guide researchers and educators in the implementation and evaluation of the PS-I approach in real classroom contexts. According to some previous experiences, PS-I can help promote deep learning and motivation in students19,21,24, but there is a need for more research about its efficacy in students with different abilities and motivational predispositions14,27. More specifically, using this document, educators can follow a PS-I implementation protocol for a statistics class designed according to the most widely-accepted principles in the PS-I literature11,13,20,50 (Steps 6-7). Additionally, educators and researchers can follow an embedded experimental evaluation about the efficacy of this implementation in students with different motivational and/or cognitive predispositions (all Steps). This experimentation does not conflict with the educational principles of equality of opportunities, free consent to participate, or respecting student confidentiality, nor is it necessary to use any new technologies.
The protocol is flexible and may be modified or applied according to new research or educational needs. Nevertheless, as described in this document, the protocol allows the evaluation of the efficacy of PS-I in terms of curiosity and different types of learning, including learning measures that require deep learning, such as conceptual knowledge and transfer of knowledge, as well as learning measures that do not necessarily require deep learning, such as procedural knowledge. Both motivation and deep learning are significant concerns for all instructors. STEM course designers are especially concerned with these topics as a large proportion of students have difficulties understanding those courses1,2,3 and experience various motivational issues4,5. The protocol also provides guidance for the evaluation of the efficacy of PS-I in students in terms of some cognitive and/or motivational predispositions, which are also a concern in STEM education, and in the relative efficacy of PS-I. The predispositions proposed in the protocol include previous academic knowledge, mastery-approach goals, sense of competence learning the subject, metacognition, and divergent thinking.
Examples of modification to the protocol based on ideas proposed in the literature include increasing the number of problems in the conditions15, giving students more time for problem exploration44, and including different variables to account for mediational learning processes14,15,24. The protocol is also flexible about the application of the different steps over different class sessions. Each step can be performed in the same class period as the previous step, and researchers and educators can decide how to organize the steps to their own convenience.
Nevertheless, a critical factor for the evaluation is that students collaborate in respecting the evaluation rules. For example, in some steps they are supposed to work individually so that possible interactions between them do not contaminate the results. In order to achieve that, it is important for students to be informed about the procedures, and for them to be equally involved in the learning activities regardless of whether they want to participate in the experimental evaluation or not32, as described in Step 1 of the protocol. For the activities that require individual work, we also recommended ensuring that there are spaces left between students.
In summary, this protocol may be useful in making PS-I and its experimental evaluation more accessible to educators and researchers, providing them with materials and guidance, giving them the flexibility to apply it according to their research and educational needs, and proposing analysis options that adapt to different sample sizes. However, one possible limitation here might be the time required to complete the questionnaires and tests about student predispositions. When the user is interested in evaluating these predispositions but there is no available time to do so during class, these questionnaires could be completed as an assignment outside class. A second limitation is the potential measurement error of some of the proposed predisposition measures that are not specifically contextualized in the learning of variability measures, but rather in general learning (metacognition and divergent thinking) or general statistics learning (mastery approach goals and sense of competence). This error should be considered as a potential limitation of any studies conducted with this protocol. A final limitation is that the previous knowledge pre-test and the learning post-test are not validated measures in the previous literature so far since the content of the implementation is very specific and validated measures for them are not available. However, it is expected that the future implementation of this protocol will advance their validation.
On similar lines, future application of the protocol will also define new research needs and new variations to be applied. Having the protocol as a common source may contribute to provide a certain systematic structure across different studies. In addition, as long as the educators find the experimental evaluation of this protocol compatible with their educational practice, this protocol may encourage involvement of educators with PS-I research, which would mean a broader professional perspective in the research process and better access to samples32.
The authors have nothing to disclose.
This work was supported by a project of the Principality of Asturias (FC-GRUPIN-IDI/2018/000199) and a predoctoral grant from the Ministry of Education, Culture, and Sports of Spain (FPU16/05802). We would like to thank Stephanie Jun for her help editing the English in the learning materials.
SPSS Program | International Business Machines Corporation (IBM) | Other programs for general data analysis might be used instead | |
PROCESS program | Andrew F. Hayes (Ohio State University) | Freely accesible at: http://www.processmacro.org. Other programs for mediation, moderation, or conditional process analyses might be used instead | |
Cognitive Competence Scale in the Survey of Attitudes towards Statistics (SATS-28) | Candace Schau (Arizona State University) | In case it is used, request should be requested from the author, who whold the copyright | |
Mastery Approach Scale in the Achievement Goal Questionnaire-Revised | Andrew J. Elliot (University of Rochester) | In case it is used, request should be requested from the author | |
Regulation of Cognition Scale of the Metacognitive Awareness Inventory | Gregory Schraw (University of Nevada Las Vegas) | In case it is used, request should be requested from the creator |