Full Scale Evaluation of an Introductory Statistics Course -- An Example

Beth Chance
Cal Poly -- San Luis Obispo

Newsletter for the Section on Statistical Education
Volume 7, Number 1 (Winter 2001)

Recently, I undertook an evaluation of the introductory course (Math 37) that I had been teaching at the University of the Pacific for 5 years. My goals were to examine the changes in student reactions to the course over time, to measure long-term retention of key concepts through follow-up surveys and interviews, and to obtain external reviews of student portfolios to assess my course goals and quality of student work. Here, I outline the process I undertook with the help of an Irvine Assessment Grant administered by UOP. A copy of the full report, with results, can be obtained by request. Preliminary efforts, e.g. photocopying of assignments and compilation of course evaluations, were completed during the fall semester. Interpretation of data and surveys was conducted during the Spring semester. Two students assisted throughout the project.

Longitudinal examination of student evaluations and performance
Compiling five years worth of course evaluations, I compared student opinion on several measures:

These results allowed me to see if there was a change in student opinion over time and if adjustments made to the materials in response to student evaluations appeared to improve these assessments.

In a separate evaluation, also completed at the end of each term, students were asked several questions about the project component of the course including:

Responses were classified as positive, negative, and lukewarm. Again, this allowed us to examine the consistency of opinions over time.

Follow-up survey of student attitudes and understanding of key statistical ideas
To see if students maintained these opinions a year after taking the course, a questionnaire was designed and sent to all students, with known addresses, who took Math 37 from me during the 1997/98 school year. We felt it would be more beneficial to hear from most of these students before possibly conducting more in-depth interviews with just a few students. The students from these three sections were asked to respond to several attitudinal questions and one content-based question. (Several of the attitudinal questions were put on the course evaluation for the Spring 99 course and the content question was also put on the final exam of all three sections of Math 37 this Spring for comparison.) Students received a $10 bookstore gift certificate for replying to the survey. A reminder postcard was sent out one week after the first mailing.

Of the 53 surveys sent out, we received 24 responses. While there was diversity of majors and semester course was taken, it is highly important to note the voluntary nature of the responses received. In an attempt to conduct some more in-depth interviews, the students could also indicate if they were willing to participate in a second survey for an additional $20. We were able to conduct such surveys with three students. The timing of the surveys at the end of the semester was a detriment to our response rates.

Content: The first question asked students to list the three most important ideas or topics of the course. This was also compared to a UOP faculty survey I had conducted earlier of what topics they hoped their students were learning in this survey course. The second question of the study survey asked which statistical tools and concepts students had used in their subsequent coursework.

Learning Aids: Students were asked which components of the course were most helpful in learning the material. They were to choose from lecture, homeworks, in-class activities, lab activities, lab write-ups, office hours, and the textbook. Students could also enter other choices on their own. Several students volunteered the term project as a valuable learning aid. This was a regrettable omission on the survey itself.

Value of Statistics: We compared some measures of student interest level in statistics for the follow-up survey and the course evaluation at the end of the Spring semester. Students were also asked to rate their level of agreement on a scale of 1-5 (with 5 presenting strongest agreement) for 2 statements: "It is important to know something about statistics" and "I suspect that I will use statistics in my chosen line of work".

Application of knowledge: Students were also given an applied contextual question on the follow-up survey and for all three sections of Math 37 Spring 1999. This question was adapted from questions 7 and 16 of the Statistical Reasoning Assessment (SRA) instrument developed by Joan Garfield and Clifford Konold (Garfield, 98):

For one month, 500 students, randomly selected from 80 elementary schools around the country, kept a daily record of the hours they spent watching television. The average number of hours per week spent watching television was 28. The researchers conducting the study also obtained report cards for each of the students. They found that the students who did well in school spent less time watching television than those students who did poorly.

Listed below were several possible statements concerning the results of this research. Students were asked to agree or disagree with each statement. For example:

a. The sample of 500 is too small to permit drawing conclusions about all elementary children at these schools.

Three students completed the full Statistical Reasoning Assessment questionnaire. Due to the nonrandom nature of the sample, it's difficult to generalize from these results. Continued use of questionnaires such as the Statistical Reasoning Assessment Tool would allow more comparisons over time, as well as to national norms.

External reviews of student portfolios: Three external evaluators from similar institutions were identified and contacted: Carolyn Pillars, Gustavus Adolphus University; Rosemary Roberts, Bowdoin College; Katherine Halverson, Smith College. A (very) modest stipend was paid to each evaluator.

During the Fall 1998 semester, copies of primary handouts and assignments were made, as well as copies of samples of student work (homeworks, quizzes, lab reports). These samples covered a wide range of student performance. Copies of all course handouts and assignments, along with a statement of goals of the course, were also made. This material was sent to the evaluators for their perusal and reflection. Several guiding questions were also submitted to help direct their review.

Observation of student interaction in classroom and lab: We attempted to track student activities during lecture and during the common hour in the computer lab. A standardized observation form was developed and student assistants recorded the activities of a random selection of students every 5 minutes. The form was divided into two subsections -- one focusing on "people interaction" and one focusing on "classroom interaction." Our goal was to classify whether students were interacting with other students and in what way, and to measure students' level of engagement during the class.

Discussion Seminar: One component of the proposed study that was not completed was a discussion seminar involving UOP professors with interests in statistics education. Such a discussion would be extraordinarily valuable for sharing ideas, examples, and concerns about the content and pedagogy of the introductory courses across campus. To be most effective, this discussion could be moderated by an external expert in statistics education.

Conclusion: This overview has demonstrated how an evaluation could be conducted to analyze time trends in student opinion and performance, and to put a more critical eye to on-going practices. While this was a very experimental study, I feel I obtained some very useful insights about my course development. The budget for this project, including student assistant, stipends for students and external reviewers but minus the campus-wide discussion group, came to less than $1500.

Return to V7 N1 Contents
Return to Newsletter Home Page