Module 6: Additional ideas in data analysis
Knowing how to compare groups of respondents appropriately is a terrific start to understanding data analysis within the context of student affairs assessment. However, this topic is only scratching the surface of all the possibilities within this field. This section provides a brief overview of some of the other data possibilities that will likely arise within student affairs assessment.
Quantitative and Qualitative Data
Student affairs assessment practitioners use data from a variety of sources. For example, a practitioner might collect written feedback from students about a program or an event on campus. Students might write their thoughts about whether they liked a speaker that a group brought to campus, or whether they benefitted from a wellness campaign during a particular time of the year. This feedback will vary and will be by nature open-ended and descriptive. This kind of data is qualitative data. A good indicator that something is qualitative is that it is predominantly text, or even a recorded interview.
Often, student affairs assessment practitioners will collect data through a survey. Surveys can have both quantitative and qualitative components, but they generally collect quantitative data. For example, a survey that asks a respondent to answer a series of multiple choice questions is collecting quantitative data—data that we can quantify, or put a number on it. We can count how many respondents answered “Strongly Agree” to a question, or how many respondents said they are first-year students.
For both quantitative and qualitative data, there are a number of ways to analyze the data. It is usually helpful to have more than one person take a look at the data in order to identify the best way to analyze the data, especially when analyzing qualitative data. And whether we should use quantitative or qualitative data is largely up to the questions we ask. Module 1 of this course has a helpful comparison between these two types of data. It is included here.
Statistical Significance
How do we know that a pattern in our data is worth paying attention to? Say, for example, that two groups of students have different GPA averages, maybe one group has a 3.2 average and another has a 3.4 average. These are different averages, but is the difference meaningful? One way we can know if a difference is meaningful is through the concept of statistical significance.
Many student affairs practitioners use hypothesis tests like t tests, chi squared tests, and ANOVA to test whether the differences between groups in a dataset are statistically significant. Without diving deep into the technical ideas behind hypothesis testing, if one of these tests reports a statistically significant difference between groups, we can conclude with a certain amount of certainty (usually with 95% certainty or more) that the difference is not due to random chance.
Say that in the example of the students with different average GPAs we find that 3.2 and 3.4 are statistically significantly different GPA averages. This means that the difference in GPA averages are due to differences between the groups and not because they just happen to be different averages. That is, students in Group A with the 3.2 GPA average have lower grades for some reason that is not represented in Group B with the 3.4 GPA average. If we did not observe a statistically significant difference between the two groups, that would suggest that the difference in GPA averages is not meaningful, that they just happen to be randomly different GPAs.
It is important to note here, however, that sometimes we are limited in our ability to find statistically significant differences because of the limitations of our sample, something we discuss in the next section. Furthermore, while we are looking for some general differences within the context of assessment, we are not making larger claims to broader audiences like research might. Meaning we can understand our findings within the assessment context, but cannot extend our findings to other student groups or different contexts.
To learn more about significance tests see t-tests Links to an external site. and chi-squared tests Links to an external site..
Sampling
Part of running tests that tell us reliable results related to statistical significance is making sure we have an appropriate sample. Just like we would get a skewed understanding of college students’ health and wellbeing if we only surveyed students who went to the gym five days a week, the way we put together a sample will have tremendous effects on the outcome of our studies or assessments. If at all possible, seek to put your samples together randomly, selecting at random from a list of students that all qualify for your survey. There are a number of good resources Links to an external site. to show you how to do this.
Furthermore, if you are interested in trying to understand student patterns among a large group of students—say an entire student body—it is important to have a large enough sample to make claims about the broader student population. We can never control who responds to surveys or calls for interviews, however, so even our samples will likely have limitations to what we can say about the population we are trying to study. Nevertheless, making sure our sample size is large enough to adequately represent the population is a key first step to take when creating a random sample. Use a sample size calculator Links to an external site. to calculate how large is large enough (make sure to type “5” in the confidence interval blank).
This section has covered a range of basic data analysis concepts. However, we have only scratched the surface in terms of having a fuller understanding of statistical analyses. Consider this section as a simple start to a robust field and explore several methods of data analysis to fit your student affairs assessment needs.