Evaluation: Our Purpose and Approach

Overview

To this point in the course, you have taken a deep dive into the discovery of the learner’s needs, challenges, and opportunities. Our focus for the rest of this course is on developmental evaluation. Before we review our developmental evaluation approach, let's take a moment to compare and contrast its purpose to other types of evaluation, namely formative and summative evaluation.

Formative and Summative Evaluation

There are many types of instructional evaluation approaches. In fact, entire college degree programs delve into the vast range of evaluation protocols. If you are in the field of education, you likely know about the distinction between formative and summative evaluation. As defined by Tessmer (1993), formative evaluation is a process of determining the worth or value of instruction for the purpose of revising and improving the instructional materials as it is being created. This is in contrast to summative evaluation that focuses on the assessment of the extent to which the lesson objectives were achieved at the end of the course, as compared in the figure below.

Formative vs Summative Evaluation Compared

Developmental Evaluation Purpose

Unlike the formative and summative approaches described above, developmental evaluation is conducted for a different purpose. As described by Michael Quinn Patton (1994, p. 133) Links to an external site.:

"Formative evaluation carries a bias about making something better rather than just making it different. From a developmental perspective, you do something different because something has changed - your understanding, the characteristics of participants, technology, or the world. Those changes are dictated by your current perceptions, but the commitment to change doesn’t carry a judgment that what was done before was inadequate or less effective. Change is not necessarily progress. Change is adaptation."

Patton (2011, p.4) elaborates on his perspective of the purpose of developmental evaluation in his book Developmental evaluation: Applying complexity concepts to enhance innovation and use (see book preview) Links to an external site.:

"As I’ve discovered over the last decade, developmental evaluation as a distinct approach to evaluation has proven especially relevant and attractive to social innovators. These people are trying to bring about major social change by fighting poverty, homelessness, community and family violence, and by helping people with AIDS, severe disabilities, chronic diseases, and victims of natural disasters and war. Some of the daunting challenges social innovators face include skepticism, criticism, naysayers, disbelievers, and the ever-present very real possibility of failure, perhaps even the likelihood of failure ... Moreover, social innovators don’t follow a linear pathway of change; there are ups and downs, roller-coaster rides along cascades of dynamic interactions, unexpected and unanticipated divergences, tipping points and critical mass momentum shifts. Indeed, things often get worse before they get better as systems change creates resistance to and pushback against the new. Traditional evaluation approaches are not well suited for such turbulence. Traditional evaluation aims to control and predict, to bring order to chaos. Developmental evaluation accepts such turbulence as the way the world of social innovation unfolds in the face of complexity. Developmental evaluation adapts to the realities of complex nonlinear dynamics rather than trying to impose order and certainty on a disorderly and uncertain world."

Our Developmental Evaluation Approach

We are taking a two-fold approach to developmental evaluation in this course:

  1. First, you will review and evaluate existing instructional materials designed by course participants in one of our prior Designers for Learning instructional design cohorts to examine the extent to which the materials meet our evolving understanding of this instructional design project's needs, challenges, and opportunities.
  2. Second, you will redesign and develop a new lesson to either:
  • make modest adjustments to the lesson's content (e.g., different instructional activities, new assessment, sharpened objectives, etc.) to better align the lesson with the assumed needs, challenges, and opportunities, or
  • make more substantial changes (e.g., redesigning the lesson for an entirely different learner, context, and/or different learning environment).

Ultimately, we are conducting this developmental evaluation as a means of refining and expanding our roster of open educational resources based on the evaluation and design criteria we will cover in these final two modules of the course. In this module, you will conduct a deep dive into a previously designed lesson to consider various design factors. You can evaluate and redesign any lesson from the roster of open educational resources in Designers for Learning's Adult Learning Zone group on the OER Commons website Links to an external site., either the lesson you considered in the Design Practice: First Impressions exercise or Reflect | M1 | Persona Discovery assignment in Module 1, or a different lesson if you have changed your desired focus.

At various points within this module, you will be prompted to consider various evaluation and design criteria in a series of reflection questions about the lesson you have chosen to evaluate and redesign. At the end of this module, your evaluation decisions will influence and inform your redesign decisions to make your learning experience effective, efficient, and engaging to our target audience of adult learners. Again, the changes you make to the lesson could be modest or substantial based on the decisions you (as the designer) chose to make about the new direction of your lesson.

Pause.png Before we move on ...

At this point, you may be saying, "But ... I'm not an expert in adult basic education." That's ok! Most educators and designers volunteering on this project aren't experts in adult basic education, either. That is why we enlisted the help of SMEs in adult education to help us design this course. Their perspectives are peppered throughout our course material. Also, during a prior implementation of this course, we conducted webcasts in which adult education instructors described their experiences working in an Adult Education instructional setting. During the March 31, 2016 webcast (55:43 minutes) posted in the Supplementary Readings & Exercises section in this course, we asked several instructors to share their perspectives on adult education. If you don't have time to review the entire webcast, we have bookmarked several points in the webcast where the adult educators offered specific advice to designers for instruction for adult learners. Their comments about the context and the needs will help you as you evaluate and redesign instruction for their learners:

Our Goals: Effective, Efficient & Engaging Learning Experiences

The adult education experts who worked with us to develop this course offered fantastic advice to get us thinking about our evaluation goals. Their advice centered on ways to design effective, efficient, and engaging learning experiences for adult learners in an Adult Education learning context. As they noted,

  • Effective instruction helps learners achieve the desired outcomes that are focused on "real world" problems and tasks they will face in life, not merely the academic problems tested within high school equivalency exams.
  • Efficient instruction avoids wasting the time of instructors and learners, and works within the available resources and constraints present in the learning environment (e.g., lack of access to computers and the Internet).
  • Engaging instruction involves your learners in their own learning.

Image

Design Checkbox.png Your Evaluation Decisions

We begin this evaluation process with your first evaluation decisions. As you embark on this challenge to both evaluate and redesign effective, efficient, and engaging instruction, begin by reflecting on the following:

  1. Think back to Module 1. What do my learners need and want to be able to do once they complete this learning experience?
  2. How can I help to make the learning experience I evaluate and redesign more effective at helping learners achieve these desired outcomes?
  3. How can I find, create, and use resources to redesign the learning experience in a more timely and cost-effective manner that considers the constraints inherent in the adult education learning context?
  4. How can I redesign the learning experience to be more relevant to my learners' past and present experiences, as well as future aspirations, and motivates my learners to participate in their own learning?