Module 1: How: Methods to address and answer assessment questions

Overview: 

  • How will you answer your assessment questions and measure change? Direct versus indirect evidence? Qualitative or quantitative methods?
  • How will this information be used to effect change?

How you will measure your outcomes?  Depending on wording of your outcomes, you may elect to use qualitative, quantitative, and/or mixed methods data collection. You may decide to use existing data, rather than collect new data. You will need to decide if your outcomes require direct or indirect evidence, or formative or summative information.

Quantitative, Qualitative, or Mixed Methods Quantitative data collection has these characteristics:

Quantitative data

  • involves the use of predetermined instruments where numeric data is collected;
  • typically measures a large number of individuals; and
  • involves statistical analysis, description of trends, and comparison of groups, relationships among variables, a comparison of results with predictions and past studies.

Qualitative data

  • involves open-ended questions, observations/field research, interviews, document analysis, audiovisual materials;
  • involves typically a small number of individuals; and
  • involves text analysis, description, analysis, and thematic development, searching for larger meaning

Mixed methods:

  • allow for both breadth and depth in participant responding.

Quant Qual homegrown.PNG

Do we want to collect direct or indirect evidence?

Direct methods

Direct methods of collecting information require the students to display their knowledge and skills.

Direct measures of learning are usually accomplished through assessment methods such as “quiz” type survey, rubric, document analysis, observation, portfolio, visual methods, one-minute assessment, and/or case study

  • Example: Where on campus would you go, or whom would you consult with if you had questions about which courses to register for in the fall?

Indirect methods

Indirect methods require that students or someone else to reflect student learning, behavior, and attitudes rather than to demonstrate it. Indirect measures of learning are usually accomplished through assessment methods such as survey, focus group, document analysis, and/or one-minute assessment.

  • Example: I know where to go on campus if I have questions about which courses to register for in the fall. (Strongly agree, Moderately agree, Neither agree nor disagree, Moderately disagree, Strongly disagree)

Closing the loop on assessment takes programs beyond data collection and interpretation to use. There is a transparency in this process that helps to foster trust and ensures participants that their voices were heard.

 

In thinking though the use of data, please consider these key points to guide your work:

  • how results provide evidence for your outcomes
  • what results say about program processes
  • what results say about the impact of the program on students’ learning and development
  • what decisions will be made or actions taken regarding programs functioning and/or assessment processes

 

Feedback loops Information from your data analysis and interpretation back into the assessment cycle at various points along the way. For example, if your results indicated that your assessment measure needed to be revised, then you would not need to revisit the stage of mission, values, and objectives clarification.

Based on the results, what decisions will you make or what action will you take regarding programs, policies, and services as well as improvements/refinements to the assessment process? Make sure to assess the effectiveness of these decisions or actions.

If results were as you hoped, then note this as well. If results did not meet expectations: Are outcomes well matched to the program? Are measures aligned with the outcomes? Do changes or improvements need to be made to the program in order to reach the goals of the program? Is there differential performance?

Activity

  1. Please review this Assessment Methodology video Links to an external site. [31:06]
  2. Ask yourself: What skills do I possess to effectively do assessment? What areas do I want to improve or to partner with others?