Using Learning Outcome Measures to assess Doctoral Nursing Education

The following protocol describes the development of an assessment rubric for the purposes of measuring student learning outcomes. The process is applied to a doctoral nursing education program.

1. Introduction. Over the past two decades, teachers in healthcare education have been increasingly challenged to demonstrate through their educational programs how students are adequately prepared to function in complex service environments upon graduation (^{1, 2, 3,4, 5, 6}). The concept of a student learning outcome measures is built on a desire to demonstrate through a measurement process how a student who successfully completes the learning process related to an area of knowledge. It is different from one who does not complete the same learning process. Grades alone do not demonstrate a student s mastery of the learning outcome but rather successful or unsuccessful achievement in response to a testing procedure at a point in time. Focus on grading at the expense of learning contributes to culture of testing rather than a culture of learning (⁷). Further, most testing procedures measure various concepts quantitatively that have been or are easily quantifiable. But what about those aspects of a science or skill that relate more to quality? How should educators measure these aspects? How should they be reflected in expected student learning outcomes? How should they be quantified so they can be appropriately measured? As a society we do not typically value what we cannot measure. This protocol provides a method for quantifying concepts and constructs in any discipline or skill through the use of what are known as “rubrics”(⁸). Once these concepts are quantified, they can then be measured, altered, and refined to demonstrate that knowledge has been attained by the student.
2. Structure. The following core concepts are visually defined as they relate to the assessment process. First, the development of the concept of a learning outcome is illustrated as follows (see Table 1). A learning outcome is the embodiment of the knowledge a student should have upon successful completion of the learning process related to some type of content. Learning outcomes should encompass knowledge that is grounded in measurement (intrinsically quantitative) and knowledge that is grounded in meaning (intrinsically qualitative).
   1. Structure-methodology. The next core concept of the assessment process is measurement methodology. Many educators confuse successful attainment of student learning outcomes with successful scores on a grade. Use of a grade as the only measurement methodology only reflects half of the potential assessment of the learning process. Perhaps this is best illustrated with the following question, which student has successfully grasped the learning outcome, the one who scores a 90% on an exam but cannot actualize the knowledge in a real-world situation versus the student who scores a 70% on the same exam and can actualize the knowledge when needed in a real-world situation? Grades alone provide a baseline, point in time measurement of knowledge verification and synthesis. They do not provide measurement of knowledge translation or application.
   2. Structure-rubric Definition. Now that we understand the underpinnings of a learning outcome, we need to understand the best way to measure it. One good way is through the use of a matrix or assessment rubric. To properly develop a student learning outcome assessment rubric, one must first define what a rubric is. It is a scoring tool that, when properly constructed, allows the person using it to assess a piece of work using a defined set of criteria. Further, it provides a mechanism for defining quality by moving those “gut perceptions” into measurable phenomena. This has been shown to be of valuable to both teacher and learner especially in adult learning environments (^{9, 10}).
   3. Structure-rubric Composition. A rubric typically takes the physical shape of a grid with rows and columns. The rows delineate the various categories that define the concept or construct that you are trying to measure while the columns represent the varying degrees of each criteria you are measuring within each category. Table 2 provides the framework for a basic rubric.
   4. Structure-rubric Overview. Once you understand what a rubric is and its function, you will need to think about what those aspects are of the concept or construct that you are trying to define. Table 3 provides an example of quantitative and qualitative concepts that could be applied to and measured in a learning outcomes assessment rubric.
3. Process. The method for developing a successful rubric is grounded in the rubrics’ ability to quickly and accurately measure the concepts or constructs you want to assess. In its most rudimentary state, it is the embodiment of the assessor s “gut feeling”. Ideally, the measurement process should take into account quantitative aspects (things that are routinely, easily measured) and qualitative aspects (things that have meaning, not easily measured) and quantify both in the form of an assessment rubric.
4. Outcome. The desired outcome of developing and using an assessment rubric in the education process is to accurately measure student learning outcomes. The learning outcomes are measurable statements of what the teacher expects the student to be able to know, do, value, and think about, etc. upon completion of the course or program.
5. Application. Harden noted that the learning outcomes could be quantified in a way that allows for measurement of competencies necessary for practice in medicine (¹¹). Well designed learning outcomes that measure performance in capstone courses have been shown to be an effective tool in correlating nursing education preparation for successful transition into practice (¹²). Properly constructed rubrics to assess learning outcomes are a core component of a well designed learning outcomes measurement program (^{13, 14}). They have significant application in professional education programs where qualitative concepts are central to practice. This protocol is applied to a doctoral nursing education program for the purposes of demonstrating how to develop and use the assessment rubrics in course and program level evaluation of desired student learning outcomes.
   1. Application-scenario. The Doctor of Nursing Practice (DNP) is a relatively new doctoral degree. It is the practice doctorate for nursing. It is the sister doctorate to other practice doctorates in healthcare such as the MD for physicians, the PharmD for pharmacists, and the PsyD for psychologists. At its core, is the expectation of graduates to be able to apply techniques such as synthesis and ethical decision making for the purposes of knowledge translation of research into nursing practice. The accreditation organization that certifies DNP programs requires that student learning outcomes measure the ability of the student to demonstrate these competencies previously described and others through application of skills and knowledge at a course and program level.
   2. Application-learning Outcome Development. A teacher in the DNP program wishes to teach the knowledge and skills to students so they can demonstrate the ability to make ethical decisions. First, the teacher should use the following question to guide their creation of a learning outcome measure: What is it that I would want to say distinguishes a student who has grasped the concept of X than someone who has not grasped the concept of X. In this application, the concept is ethical decision making (see Table 1 for example).
   3. Application-concept Development. Next, the teacher should write down these concepts that underpin the structure of whatever the outcome is that you are trying to measure. In our case, ethical decision making is what we want to assess. Again, Table 1 provides an example of these key concepts.
   4. Application-quantitative Rubric Section Development. Third, the teacher should determine which of these concepts are intrinsically quantitative and may be measured as learning outcomes that encompass knowledge grounded through measurement. Table 4 represents the quantitative measurements of ethical decision making in a rubric that would measure our proposed learning outcome.
   5. Application-qualitative Rubric Section Development. Fourth, the teacher should determine which of these concepts are intrinsically qualitative and may be measured as learning outcomes that encompass knowledge grounded in meaning. This section of rubric development may be implemented in one of two ways: either as a pure-qualitative descriptor section rubric or a quasi-qualitative descriptor section of the rubric. Once this section of the rubric is completed, the rubric is ready for use in assessing learning outcomes.
   6. Application-qualitative Rubric Section Development Explicated. It is important to describe in additional detail the importance of clarifying how a teacher intends to measure the qualitative concepts of a learning outcome. The benefit of the pure-qualitative level rubric is the articulation of previously ill-defined or un-defined concepts into words which helps a teacher begin to qualify “gut” perceptions of student learning. However, the price of using the pure-qualitative approach to rubric measurement is the sacrifice of a more reliable assessment rubric because the application of the score is still in the eye of the assessor. Conversely, development of the concepts into a quasi-qualitative descriptor assessment facilitates reliable measurement when it is used by other assessors because it quantifies the concepts in a simple, objectively measurable format.
   7. Application-rubric Testing. Now that the rubric is created, it is important to determine how it will be applied to measure the outcome. In general, learning outcome measurement rubrics are applied to learning activities in a course and have a grade component associated with them. The example in Table 5 shows how the rubric we just created is applied to an ethical decision-making exercise.