Deborah A. Curtis

San Francisco State University

Michael Harwell

University of Pittsburgh

Journal of Statistics Education v.6, n.1 (1998)

Copyright (c) 1998 by Deborah A. Curtis and Michael Harwell, all rights reserved. This text may be freely shared among individuals, but it may not be republished in any medium without express written consent from the authors and advance notification of the editor.

**Key Words:**
Doctoral student preparation; Statistics
education; Survey research.

Although numerous research studies have focused on issues related to the teaching of statistics, few studies have focused on the training of people who may become statistics teachers. The purpose of this study was to examine doctoral students' preparation in statistics in the field of education. A national survey was conducted of twenty-seven quantitative methods (QM) programs. One QM professor from each program was identified and asked to describe and evaluate the training of QM and non-QM doctoral students at his or her institution. The vast majority of professors indicated that most or all of the students in their QM programs received training in the "old standard" procedures -- ANOVA, multiple regression, and traditional multivariate procedures, whereas fewer than half of the professors indicated that most or all of their QM students received training in more recent procedures such as bootstrapping and multilevel models. Professors were also asked to rate the skills of their QM students in areas such as mathematical statistics and computing on a scale from "Weak" to "Strong." Most professors gave high ratings to their QM students' skills with statistical packages, but gave much more mixed ratings of their QM students' training in mathematical statistics. Nearly half of the professors thought that most of their QM students could have benefited from one or two additional statistics courses. Results are discussed in terms of training future doctoral students.

1 Much of the literature on statistics education focuses on basic issues of designing and delivering statistics courses. Thus, a good deal of attention has been paid to issues such as what topics to teach in statistics courses, how to teach specific topics in statistics, how to organize classrooms for effective instruction, which textbooks to use, which computer programs to use, and what kinds of examples to use. Substantially less attention has been paid to examining the graduate school preparation of the statistics educators themselves. This seems unfortunate, because the knowledge and skills of these statistics educators will likely have a direct impact on their effectiveness as classroom instructors, statistical consultants, and mentors. The purpose of this paper is to address this topic by taking a closer look at doctoral training in statistics in one particular academic discipline, namely the field of education.

2 Consider doctoral students who are enrolled in quantitative methods (QM) programs in the U.S. Most doctoral programs in education have QM programs, loosely defined here to include doctoral students interested in research design, educational measurement, and educational statistics. Many of the graduates of these QM doctoral programs will go on to teach introductory and advanced statistics, measurement, and research design courses in colleges and universities across the nation. Moreover, these graduates may serve as statistical educators in ways that are not related to direct classroom instruction, including serving as consultants with colleagues; serving on thesis and doctoral committees; serving on editorial boards of research journals; and training and supervising future doctoral students. Although there may be disagreement among faculty members over what should be the specific content of doctoral training programs in quantitative methods, there is substantial agreement on the importance of adequately preparing students for their future roles as teachers, researchers, and consultants. As faculty members who are responsible for training QM doctoral students in research methods and statistics and who recognize the variety of skills our graduates need, we have a special responsibility to train students in techniques that will allow them to expertly discharge their professional responsibilities.

3 Next consider doctoral students in education in the U.S. who
are *not* enrolled in QM programs. These non-QM doctoral
students in education vary widely in their areas of
specialization. Typical specializations offered in U.S.
universities include educational leadership, educational
psychology, curriculum and instruction, special education,
mathematics education, science education, bilingual
education, language and literacy, and counseling. When
studying the statistics preparation of doctoral students in
education, we considered these non-QM students to be an
important group to study because non-QM graduates who take
post-secondary faculty positions in education may also play
a part in the statistical education of students at their
institutions, either directly, as classroom instructors, or
indirectly, as advisors, mentors, and consultants.

4 This paper presents the results of a survey addressing one particular aspect of doctoral student preparation, namely, their skills and knowledge in the area of statistics. A national survey was conducted of twenty-seven quantitative methods (QM) programs. One QM professor from each program was identified and given a questionnaire to complete. These professors were asked to describe and evaluate the training of both QM and non-QM doctoral students at their institutions.

5 The literature on training doctoral students in educational statistics is quite sparse. A survey of statistics course content for doctoral students has not been conducted in the field of education, although studies of statistical preparation have been conducted in psychology (see Aiken, West, Sechrest, and Reno 1990) and medicine (see Dawson-Saunders, Azen, Greenberg, and Reed 1987). The lack of systematic research on what doctoral students are learning in their statistics classes is unfortunate because this information could help facilitate discussions of whether this training should be restructured and, if so, how it should be restructured. This is not to suggest that discussions of what should be taught do not take place. For example, Tukey (1980), among others, has argued that students need much more experience with exploratory data analysis in their statistics training. Similarly, Noether (1980) argued for teaching nonparametric methods in introductory statistics classes because nonparametrics can provide a strong conceptual framework for students. Brogan (1986) also suggested that nonparametrics should be taught, especially in social science and nursing disciplines. Brogan further recommended that in a required two-course statistics sequence, graduate students should learn about general linear models, including multiple linear regression, ANOVA, and possibly categorical data analysis, and should be trained to use statistical computer packages.

6 Content analyses of the statistical procedures and research
designs currently being used in published research can serve
as a foundation for discussions of what we should be
teaching. A number of such content analyses have been
performed over the years. Elmore and Woehlke
(1988, 1996),
for example, conducted a content analysis of statistical
methods used in the *American Educational Research
Journal* (AERJ), *Educational Researcher*, and *Review of
Educational Research* for all articles published. In their
1988 publication, these researchers focused on articles
published between 1978 and 1987, and in their 1996 paper,
they focused on articles published between 1988 and 1995.
Consider their findings for AERJ articles published between
1988 and 1995. Here, Elmore and Woehlke
(1996) found that
the predominant statistical method used was ANOVA/ANCOVA,
which was used in 64 out of 245 articles that they coded.
The next most frequent procedures were multivariate
procedures, which were used in 12% of the articles, followed
by bivariate correlation (12%), multiple
regression/correlation (11%), t-tests (9%), structural
equation modeling (9%), and nonparametrics (8%).

7 Aiken, West, Sechrest, and Reno (1990) examined the adequacy of doctoral student preparation in statistics in psychology programs. Because our research is similar to theirs, we will describe their study in detail. Aiken et al. surveyed all 222 psychology departments or schools identified by the American Psychological Association as granting doctoral degrees. From these 222 units, 186 responded, for a response rate of 84%. The goals of their research were (a) to describe the current content of the statistics, measurement, and research design courses; (b) to look at differences in requirements between sub-disciplines in psychology; and (c) to gather data on professors' perceptions of students' abilities to apply statistical and measurement techniques in their own research. In addition, these researchers distinguished between "elite" and "other" institutions, and presented data comparing the two types of institutions.

8 In terms of demographics, Aiken et al. (1990) found that 89% of the departments offered an introductory graduate statistics sequence, and of the departments offering this sequence, 77% were one year in length. A minority of institutions -- 17% -- offered doctorates in a quantitative area of specialization, although one-third of these did not have any first-year students. In terms of curriculum offerings (including optional and required courses), they found that the majority of programs offered at least a partial course (defined as at least a half a semester or full quarter) in ANOVA (88% of the programs), multiple regression (58%), and MANOVA (53%). More recent statistical topics such as structural equation models and time series models were less frequently covered (18% and 6%, respectively).

9 In terms of the content of the required introductory statistics course sequence, the authors distinguished between "old standards of statistics" and "more advanced statistical considerations," and collected information on the percent of institutions offering in-depth coverage of specific procedures. ("In-depth" was defined as coverage to the point that students could perform the analysis in question themselves). In-depth coverage of the "old standards" was provided in the majority of institutions: multifactor ANOVA, multiple comparison procedures, repeated measures via traditional factorial ANOVA, and multiple regression were covered in-depth in the required course sequence in 73%, 69%, 73% and 63% of the institutions, respectively. Topics considered more advanced were covered in-depth in required courses less often. For example, 39% of the institutions provided in-depth coverage of ANCOVA in their required course sequence, 38% provided in-depth coverage of ANOVA as a special case of regression, 20% provided in-depth coverage of exploratory data analysis, 21% provided in-depth coverage of multivariate procedures, and 18% provided in-depth coverage of statistical power analysis.

10 Aiken et al. (1990) interpreted these findings to mean that the statistical and methodological curriculum in psychology has changed little in 20 years. They also concluded that typical first-year courses only serve those students who undertake traditional laboratory research. They summarized their work by stating that new Ph.D. recipients are competent to handle traditional statistical techniques but not newer and often more useful procedures (p. 721).

11 Guo and Nitko (1996) surveyed 80 programs that offer graduate training in educational measurement. Among the 48 programs that responded and award doctorates, 19 offer training in measurement alone, 16 offer degrees in the broader area of research methodology, and 13 offer degrees in other areas. Guo and Nitko's focus was on surveying the content of courses offered to QM students and assessing agreement on what is taught. Although their focus was on educational measurement, they also presented aggregated information that reflected the frequency of coverage of various statistical topics. At the doctoral level, students in the 19 programs in measurement take 21 credits of statistics on average, and those in research methodology programs take 24 credits of statistics. The statistics courses include regression, factor analysis, multivariate analysis of variance (MANOVA), and other unspecified advanced topics.

12 Like Aiken et al. (1990), our general purpose was to survey the course-related training of doctoral students in statistics. One goal was to assess the statistical methods and procedures used in training QM and non-QM students in education. Another goal was to learn about the perceptions of QM professors regarding the competence of QM and non-QM doctoral students with various statistical techniques. This information should add to the statistical education literature on training doctoral students in QM.

13 The universities we initially targeted were educational
research institutions identified in a University of Illinois
study as constituting the top thirty-one Schools and
Colleges of Education, ranked in terms of academic
productivity and prestige (West and Rhee 1995).
A list of
these institutions is provided in Table 1.
Our target
population consisted of those universities that had QM
programs, and our plan was to identify one QM professor from
each institution as a contact person who would complete a
questionnaire. The use of the 31 schools listed in
West and
Rhee (1995) as our initial sample necessarily narrows our
inferences to these or similar institutions. In order to
identify these contacts, several strategies were used.
First, we simply identified people that we knew personally
at specific universities, with the belief that these people
would be most likely to complete questionnaires. Second, we
used the membership roster of the American Educational
Research Association (AERA) Educational Statisticians
Special Interest Group. Because the initial results of this
study would be presented at an AERA conference, we believed
that using this membership roster would increase our chance
of having the questionnaires completed and returned. Third,
we used a book entitled *Graduate Study in Educational and
Psychological Measurement, Quantitative Psychology, and
Related Fields*, published by the Pennsylvania State
University, which lists quantitatively-oriented programs in
the United States and provides names of appropriate contact
persons. Finally, in a few instances, we used personal
contacts to identify a non-quantitative faculty member at a
particular university who then gave us the name of a QM
professor at that university. There were only two instances
where we could not identify a contact person at a given
university, leaving an effective sample of 29 universities.

** Table 1.** Universities Initially Contacted for this Study

1. University of Wisconsin -- Madison |

2. University of Illinois |

3. Ohio State University |

4. Stanford University |

5. University of Minnesota |

6. Indiana University -- Bloomington |

7. Michigan State University |

8. Columbia -- Teachers College |

9. University of Georgia |

10. Pennsylvania State University |

11. University of Maryland |

12. University of Texas -- Austin |

13. University of Michigan |

14. Arizona State University |

15. University of California -- Los Angeles |

16. University of Washington |

17. University of California -- Berkeley |

18. University of Chicago |

19. Harvard University |

20. University of Virginia |

21. Vanderbilt University -- Peabody |

22. University of North Carolina |

23. University of Florida |

24. Florida State University |

25. Syracuse University |

26. University of Arizona |

27. University of Nebraska |

28. Virginia Polytechnic |

29. SUNY-Buffalo |

30. University of Missouri |

31. University of Pittsburgh |

14 Because our goal was to survey universities with QM programs, the next step was to determine whether each university had a QM program. To do this, QM programs were broadly defined -- thus, some programs might place a heavy emphasis on measurement, whereas others might place a heavy emphasis on educational statistics. To figure out which universities had a QM program, we again used our personal knowledge of various universities. In cases where we were not sure, we contacted someone at that university. Of the 29 universities, all but two had QM programs. Thus, our final sample consisted of 27 universities. (The two Schools/Colleges without QM programs still offered doctoral training in statistics.)

15 In November 1995, all participants received a copy of the questionnaire (described below) to complete. A second mailing was conducted approximately one month later, in December 1995. In January 1996, this second mailing was followed by e-mail and/or phone calls asking that participants return their questionnaires. All questionnaires received within three months of the initial mailing were then analyzed. Twenty-three of the 27 universities with QM programs responded for a response rate of 85%. This rate is consistent with that reported by Aiken et al. (1990) and Guo and Nitko (1996). We believe that this response rate is large enough to generate credible results, although it is possible that there is some unknown but systematic difference between QM faculty members who responded to the survey and those who did not respond.

16 A questionnaire (see Appendix) was used to collect information about the statistics courses offered in QM programs and the perceptions of professors of QM. The questionnaire was divided into two sections, one directed toward students majoring in QM and one for non-QM students.

17 On the portion of the questionnaire pertaining to QM programs and training of QM doctoral students, we collected the following data:

- Demographic information, including information on the
number of full-time faculty in the QM program, the
number of lecturers in the QM program, the number of
students in the QM program (full-time or part-time),
and where students took their required statistics
courses (e.g., in the School/College of Education,
statistics department, etc.),
- The extent to which QM doctoral students are trained in
specific statistical topics or methods (assessed using
a checklist),
- Professors' perceptions about the adequacy of QM
doctoral students' computer skills and mathematical
training, and
- Professors' perceptions on whether QM doctoral students could have benefited from additional statistics coursework.

18 The development of the questionnaire addressing the training of non-QM students was more challenging because it needed to be sufficiently flexible to accommodate the various ways in which Schools and Colleges of Education are organized. Specifically, a School/College of Education might consist of a series of schools, divisions, departments, or other organizational units, each with separate programs. Each organizational unit could conceivably have its own statistics requirement and its own statistics course sequence. A non-QM student with one area of specialization at a particular institution could, therefore, take one required statistics course taught in one particular organizational unit, whereas another student with a different area of specialization could take a different required statistics course taught in a different organizational unit.

19 Considering these complexities, it became apparent that completion of the survey of what non-QM students learn in their statistics courses would require input from all of the statistics instructors in each organizational unit at each institution. We thought that such a survey would be too unwieldy, and decided to shift the focus for the non-QM students away from a survey of course content and course requirements and towards a survey of quantitative professors' perceptions of non-QM doctoral students' preparation in educational statistics. We asked one QM professor at each institution to evaluate non-QM doctoral students' training in educational statistics at his or her institution, and to frame this evaluation in terms of doctoral students with whom he or she had direct contact over the past few years -- through committees, research assistantships, classes, and so forth. In this way, we hoped to circumvent potential problems of having professors speculate about what was taught in statistics classes they might not have thorough knowledge of and about the statistical competence of doctoral students whom they might not know well.

20 For the non-QM students, we collected the following data:

- Information on statistics requirements for these
students,
- Whether statistics requirements have changed over the
past five years,
- Professors' perceptions of doctoral students' ability
to critically read and interpret substantive research
articles utilizing specific statistical procedures, and
- Professors' opinions on whether the non-QM doctoral students could have benefited from more statistics coursework.

21 In designing the questionnaire, we were initially concerned
about QM faculty members' ability to provide accurate
information regarding the preparation of non-QM students.
First, we were concerned that some QM professors might not
be familiar with the statistics requirements of non-QM
students. These concerns led us to conduct informal
conversations with some of our colleagues at other
institutions, who did, in fact, appear to be familiar with
the statistics requirements at their universities. Our
concerns were further alleviated by the results of the
survey itself: Of the 23 QM professors we surveyed, only
one person was unfamiliar with the statistics requirements
of the non-QM students at the university. Second, we were
concerned that the QM professors might not have adequate
familiarity with the non-QM students to accurately evaluate
their competence with statistics. To address this problem,
we decided to ask the professors to evaluate only the non-QM
doctoral students with whom they had *direct* contact over
the previous five years -- through dissertation committees,
classes, consulting, and so forth. While this strategy is
still problematic from a design perspective, we believe that
the results provide a solid first step in looking at the
adequacy of non-QM doctoral students' training in
educational statistics.

22 It is important to point out three limitations of the current research. First, sampling the schools listed in the West and Rhee (1995) paper limits the generalizability of the results, as does the use of only one QM faculty member per institution in the survey. Second, because of time and resource constraints we did not collect survey data on doctoral students' preparation in the areas of educational research methods or measurement (see Guo and Nitko 1996), thus limiting the picture we can paint about doctoral student training. Third, no thorough reliability or validity information was obtained for the questionnaire items. On the positive side, we believe that the survey has several advantages, among them, that it is not a time-consuming research method and it is not particularly intrusive.

23 General information for QM programs and students is presented in Table 2.1 and Table 2.2. The median number of full-time tenured and tenure-track faculty in these QM programs is four, with three full-time faculty regularly teaching statistics during the academic year. The distribution of new doctoral students enrolling in QM programs was positively skewed, with a median of four students admitted annually during the previous five years. All of the institutions required students to take at least one statistics course in the School or College of Education; nearly half required students to take one or more courses in the Mathematics or Statistics Departments; and two (9%) required students to take one or more courses in the Psychology Department.

** Table 2.1.** Demographics: Program Size

| Mean | Median | SD | N |
---|---|---|---|---|

Number of full-time tenured or tenure-track faculty in QM program | 5.1 | 4 | 2.2 | 23 |

Number of full-time faculty who regularly teach statistics during the academic year | 3.6 | 3 | 1.7 | 23 |

Number of lecturers and adjunct faculty members who regularly teach statistics in QM program | 1.8 | 1 | 2.8 | 23 |

Approximate number of students admitted to QM program each year over past five years | 7.2 | 4 | 8.8 | 22 |

** Table 2.2.** Location of Required Statistics Courses in QM
Program (N = 22)

School/College of Education | 100% |

Mathematics/Statistics Department | 46% |

Psychology Department | 9% |

Other | 5% |

24 To assess the extent to which doctoral students in QM programs received training in specific procedures, we developed a list that we believe reflects a breadth of statistical procedures, including those considered to be "recent," such as meta-analysis, bootstrapping/jackknifing, multilevel models, and structural equation models. We asked faculty to indicate those procedures and methods in which most or all of their QM students received training. Results are presented in Table 3.

** Table 3.** Preparation of Quantitative Methods Doctoral
Students (Percent of Universities Where Most or All QM
Program Graduates Receive Training in Topic/Method, N = 23)

ANOVA | |
---|---|

Covariance analysis | 100% |

Repeated measures designs | 96% |

Power/sample size calculations | 96% |

Mixed-effects models | 91% |

Random-effects models | 87% |

Nonorthogonal designs | 83% |

Thorough coverage of multiple comparison procedures | 74% |

Cell means models | 57% |

Complex designs (e.g., fractional factorial) | 39% |

Multiple Regression | |

Ordinary least squares estimation | 100% |

Weighted least squares estimation | 70% |

Nonlinear-in-the-predictors models | 70% |

Logistic regression | 65% |

Nonlinear-in-the-parameters models | 39% |

Traditional Multivariate Procedures | |

Canonical correlation | 91% |

MANOVA | 91% |

Discriminant analysis | 91% |

Factor analysis | 87% |

MANCOVA | 83% |

Principal components analysis | 78% |

Log-linear models | 56% |

Nonparametric Procedures | |

Kruskal-Wallis test | 48% |

Exact tests | 44% |

Jackknifing/Bootstrapping | 39% |

Repeated measures tests (e.g., Friedman, Cochran) | 30% |

Asymptotic relative efficiency | 26% |

Rank-transform tests | 22% |

Other Topics and Procedures | |

Matrix algebra | 83% |

Meta-analysis | 74% |

Structural equation models | 74% |

Multilevel models/Hierarchical linear models | 35% |

Times series models | 4% |

25 The data suggest that the vast majority of QM doctoral students receive training in the "old standards" -- ANOVA, multiple regression, and traditional multivariate procedures. The exceptions to this appear to be logistic regression and log-linear models, where more than half of the respondents indicated that their QM students receive training. Nonparametric procedures, on the other hand, appear to be less prevalent. Although a more detailed list of nonparametric procedures was not provided, it seems reasonable to use the Kruskal-Wallis test as a benchmark and to note that only half of the respondents indicated that their QM students received training in this procedure. Finally, in terms of more recent procedures, we see that the majority of programs train their students in meta-analysis and structural equation models (74%), and over one-third of the programs train their students in jackknifing/bootstrapping (39%) and multilevel models (35%).

26 We also asked faculty to rank the mathematical statistics training and computer skills of most of their QM doctoral graduates using a 4-point scale with "1" indicating "weak" skills and "4" indicating "strong" skills. Results are presented in Table 4. Faculty ratings were reasonably symmetrically distributed, with most of the faculty rating their students midway on the scale, either as a "2" (38% of faculty) or as a "3" (48% of faculty). In terms of computer skills, the vast majority (91%) thought their students had "strong" skills in standard data analysis programs such as SAS and SPSS. Faculty perceptions of student competence in other computer skills -- database management and programming -- were more varied.

** Table 4.** Faculty Members' Perceptions of the
Mathematical/Statistical Training and Computer Skills of
Quantitative Methods Program Graduates

Weak | Strong | ||||||
---|---|---|---|---|---|---|---|

(1) | (2) | (3) | (4) | M | SD | N | |

1. Mathematical statistics training of most QM graduates | 1 | 8 | 10 | 2 | 2.6 | .74 | 21 |

(5%) | (38%) | (48%) | (10%) | ||||

2. Computer skills: Standard data analysis programs (e.g., SAS, SPSS, SYSTAT) | 0 | 0 | 2 | 21 | 3.9 | .28 | 23 |

(0%) | (0%) | (9%) | (91%) | ||||

3. Computer skills: Database management | 1 | 6 | 14 | 2 | 2.7 | .69 | 23 |

(4%) | (26%) | (61%) | (9%) | ||||

4: Computer skills: Programming (e.g., FORTRAN, PASCAL, C) | 7 | 5 | 9 | 2 | 2.3 | 1.01 | 23 |

(30%) | (22%) | (39%) | (9%) |

NOTE: For this question, the anchors were 4 = Strong, perhaps took multiple courses in mathematical statistics and probability theory, would probably be well acquainted with topics such as quadratic forms, Likelihood Ratio Test Principle, Gauss-Markoff Theorem, etc.; and 1 = Weak, little or no coursework in mathematical statistics or probability theory.

27 Colleagues have occasionally remarked that they believe earlier graduates of QM programs were better trained than more recent graduates. We investigated these kinds of comments by asking faculty in the survey if they thought that doctoral students who had graduated in the last year or two were better prepared or less well prepared to do publishable statistical work, as compared to students who graduated five to six years earlier. Data from Table 5 do not support the claim that earlier graduates are perceived to be better trained. In fact, none of the faculty indicated that they thought their recent QM students were less well prepared than graduates from five to six years prior. Over half of the faculty (56%) thought that there was no difference between past and present graduates, and the remaining 44% believed that recent graduates are better prepared.

** Table 5.** Faculty Members' Perceptions of QM Program
Graduates' Ability to Do Publishable Statistical Work
(Percent of Faculty Selecting Each Option, N = 23)

Faculty who believe that recent graduates are better prepared than graduates from 5-6 years ago | 44% |

Faculty who believe that recent graduates are less well prepared than graduates from 5-6 years ago | 0% |

Faculty who believe that there is no difference between recent graduates and graduates from 5-6 years ago | 56% |

28 Finally, we asked faculty members to indicate whether they thought their QM students could benefit from additional (probably specialized) statistics coursework. Specifically, they were asked to assess (a) whether more than half or fewer than half would have benefited from one or two additional courses, and (b) whether more than half or fewer than half would have benefited from three or more additional statistics courses. Data are presented in Table 6. Nearly half of the respondents (48%) indicated that they thought more than half of their students could have benefited from one or two additional courses in statistics. Moreover, 20% indicated that they thought more than half of their students could benefit from three or more additional courses.

** Table 6.** Faculty Members' Perceptions of Doctoral Students'
Need for Additional Statistics Coursework for Quantitative
Methods Program (Percent of Faculty Selecting Each Option)

Statistics Coursework | <50% | >50% | Can't Judge |
---|---|---|---|

1-2 additional (probably specialized) statistics courses (N = 21) | 43% | 48% | 10% |

3 or more additional (probably specialized) statistics courses (N = 20) | 65% | 20% | 15% |

NOTE: Data are in response to the following question: In hindsight, of the quantitative methods program doctoral students who graduated from your program in the last 5 years, what percent do you think could have profited from (the following additional coursework)?

29 For the non-QM doctoral students, we began by determining whether each institution had a minimal statistics requirement for all doctoral students in education. Descriptive statistics are presented in Table 7. In nearly half of the institutions (44%), students in some programs can graduate without taking a statistics course, whereas in 39% of the institutions, students in all programs are required to take at least one statistics course. Only 13% of the institutions had a uniform two-course statistics requirement. (Note that this statistic does not estimate how many students actually graduate without a statistics course in a given institution.)

** Table 7.** Coursework Requirements for Doctoral Students in
Education Who Are Not in Quantitative Methods Programs
(Percent of Faculty Selecting Each Option, N = 23)

Students in some programs can graduate without taking a statistics course | 44% |

Students in all programs are required to take at least one statistics course | 39% |

Students in all programs are required to take at least two statistics courses | 13% |

Don't know | 4% |

30 We speculated that students might be taking fewer statistics courses currently than in previous years, due to the increased prevalence of qualitative research methods. The data in Table 8, however, do not indicate a decreasing trend in the number of students taking statistics over the past five years. In approximately one-third of the institutions surveyed, students are taking fewer statistics classes, but in a roughly equal number of institutions, they are taking more statistics classes.

** Table 8.** Changes in Statistics Enrollment over the Past
Five Years for Doctoral Students in Education Who Are Not in
Quantitative Methods Programs (Percent of Faculty Selecting
Each Option, N = 22)

Doctoral students have tended to take fewer statistics courses | 27% |

Doctoral students have tended to take more statistics courses | 36% |

Doctoral students have tended to take about the same number of statistics courses | 27% |

Don't know/No Response | 9% |

31 We asked faculty to rate non-QM doctoral students in terms of their ability to critically read and interpret research articles utilizing specific statistical procedures (see Table 9). In particular, faculty were asked to think about the non-QM doctoral students with whom they had contact over the past five years, and to indicate whether they thought more than half or fewer than half of these students would be competent with a particular statistical method. Notice that this question requires a lower threshold of "competence" than a question that asks about doctoral students' abilities to carry out and analyze research utilizing these procedures. The procedures listed -- including ANOVA, ordinary least squares (OLS) regression, MANOVA, and nonparametric rank tests -- were not meant to be an exhaustive list of procedures, but rather were selected because they appeared to represent a cross-section of "common" and "advanced" statistical methods found in published educational research.

32 Results indicate that the faculty perceived the students to be most competent with ANOVA: 63% of the faculty thought that more than half of their graduates could critically read an article using ANOVA. Only 41% of the faculty thought that more than half of their graduates could critically read an article using OLS regression. Only a few of the 22 QM professors thought that more than half of their students could critically read and interpret substantive articles that utilized more advanced procedures, namely MANOVA, log-linear models, and structural equation models.

** Table 9.** Perception of Students' Competence to Critically
Read and Interpret Research Articles Utilizing Specific
Statistical Procedures for Doctoral Students Who Are Not in
Quantitative Methods Programs (Percent of Faculty Selecting
Each Option, N = 22)

<50% | >50% | |||
---|---|---|---|---|

N | (Pct) | N | (Pct) | |

ANOVA | 8 | (36%) | 14 | (63%) |

OLS regression | 13 | (59%) | 9 | (41%) |

Nonparametric rank tests | 19 | (86%) | 3 | (14%) |

MANOVA | 20 | (91%) | 1 | ( 5%) |

Log-linear models | 21 | (95%) | 1 | ( 5%) |

Causal models | 20 | (91%) | 1 | ( 5%) |

NOTE: Data are based on responses to the following question: Think about the doctoral students in your School/College of Education who are NOT in your quantitative methods program that you have had contact with over the past 5 years or so -- through dissertation committees, classes, consulting, and so forth. Of these non-quantitative students, what percent do you think are competent to critically read and interpret research articles that utilize the following procedures? (1) Less than half, (2) More than half, (3) Can't judge.

33 Faculty were asked to think about non-QM students with whom they had contact over the past five years, and who had graduated, and to assess whether more than half or fewer than half would have benefited from additional statistics coursework. Results are shown in Table 10. Virtually all of the faculty thought that more than half of the graduates could have benefited from additional (probably specialized) statistics coursework: 44% of the faculty thought that over half of the graduates could have benefited from one or two additional courses, and 43% of the faculty thought that at least half of the graduates could have benefited from three or more courses.

** Table 10.** Faculty Members' Perceptions of Graduates' Need
for Additional Statistics Coursework for Doctoral Students
Who Are Not in Quantitative Methods Programs (Percent of
Faculty Selecting Each Option)

Statistics Coursework | <50% | >50% | Can't Judge |
---|---|---|---|

1-2 additional (probably specialized) statistics courses (N = 23) | 52% | 44% | 4% |

3 or more additional (probably specialized) statistics courses (N = 21) | 48% | 43% | 10% |

NOTE: Data are in response to the following question: Think about the doctoral students in your School/College of Education who are NOT in your quantitative methods program that you have had contact with over the past 5 years or so -- through dissertation committees, classes, consulting and so forth. Of these non-quantitative students, what percent do you think could have profited from (the following additional coursework)?

34 Has this survey accomplished anything? Despite some methodological shortcomings, we think so. For the portion of the survey addressing the preparation of QM students, we offer three general findings. First, we now have empirical evidence of the nature of the curriculum offered in a sample of QM programs. We found much of this information encouraging because it indicated widespread professional agreement in the way that doctoral students majoring in quantitative methods are trained in statistics. For example, almost all of the surveyed programs emphasized traditional data-analytic procedures, a result that agrees with the work of Elmore and Woehlke (1988, 1996). There was also evidence that most respondents thought recent graduates were as well or better trained than earlier graduates. The fact that a number of QM programs also offered instruction in more recent or advanced procedures such as meta-analysis, structural equation models, and jackknifing/bootstrapping implies that a number of faculty believe this kind of training is important enough to incorporate into the curriculum. A second implication of our findings is their relationship to those reported by Aiken et al. (1990). In general, the two sets of findings are similar in terms of what is being taught in QM programs in education and psychology, and in terms of the perceptions of QM professors of students. One area of modest disagreement is that our findings suggest that newer statistical techniques are being taught more frequently in Schools and Colleges of Education than in Departments of Psychology. Still, the results suggest that much of the information provided by the two studies is similar and can be combined to draw stronger conclusions and guide future research.

35 Perhaps the most important implication of our research on QM programs is that the findings offer a platform from which we can ask (and in some cases provide tentative answers to) some important questions about the training of doctoral students in quantitative methods. For example, about half of the quantitative methods programs surveyed have incorporated newer techniques into their curricula. In our view, this places our findings in a somewhat less positive light. What accounts for the failure to offer training in these newer techniques? Is it simply the result of constraints on credit hours, too few faculty, or that training students who are not majoring in quantitative methods is now the primary business of many quantitative methods programs, and that this has a severe effect on the curriculum?

36 We believe the questions raised above (and others) are important and thus provide directions for additional work in the area of preparing doctoral students in quantitative methods. One obvious need in future work is to survey at least two faculty per program to allow within- and between-institution variability to be examined; another is to collect information on additional variables that should controlled, such as rank of the school and required statistics training for QM and non-QM students. It is also important to widen the participation of QM programs and to obtain information about the kinds of positions taken by graduates. As the statistical education literature has argued, curricula must be responsive to the needs of the institutions and industries hiring graduates. We believe that the emergence of information about the training of students in quantitative methods programs should encourage a long-overdue dialogue among educational statisticians.

37 For the portion of the survey that addressed the preparation of non-QM students, the research findings suggest that a number of QM faculty think many non-QM doctoral students are under-prepared to be critical consumers of quantitatively-oriented research articles. Specifically, we found that 36% of the faculty thought that fewer than half of the non-QM doctoral students could critically read and interpret research articles utilizing ANOVA, and 59% of the faculty thought that fewer than half of the non-QM doctoral students could critically read and interpret research articles utilizing OLS regression. Moreover, the vast majority of faculty thought that fewer than half of the non-QM doctoral students could critically read articles utilizing more advanced procedures, namely MANOVA, log-linear models, nonparametric rank tests, and structural equation models. These research findings provide preliminary data suggesting that we may have cause for concern about non-QM doctoral students' preparation in statistics.

38 Although these findings on the preparation of non-QM doctoral students are only tentative, they do suggest a direction for future research and discussion. One line of research suggested by this investigation involves studying faculty members' opinions about why some non-QM doctoral students at their universities are under-prepared in educational statistics, and what they think can be done to improve these students' preparation. Another line of research suggested by this investigation involves studying editors of educational research journals. Because these editors review so many manuscripts (both publishable and unpublishable), they are in a good position to evaluate the statistical and methodological training of today's educational researchers. In addition to journal editors, it might prove fruitful to study the opinions of the doctoral students themselves regarding the adequacy of their own training. Lastly, the perceptions of non-QM faculty would help inform a discussion on the statistical preparation of non-QM doctoral students.

This questionnaire is part of a national survey of
statistics training in doctoral programs in education. The
first part of the questionnaire addresses training of
doctoral students in quantitative methods (QM) programs in
education. The second part of the questionnaire addresses
statistical training of doctoral students in education who
are *not* in a quantitative methods program. Your responses
to this questionnaire are anonymous and confidential.

- Does your university operate on the quarter, trimester,
or semester system?

_____ quarter

_____ trimester

_____ semester

- How many full-time faculty
*tenured or tenure-track*are there in your quantitative methods program? ________ - How many
*full-time*faculty members regularly teach*statistics*during the academic year? ________ - How many
*lecturers*and*adjunct*faculty members teach*statistics*in your quantitative methods program during the academic year? ________ - Over the past five years, approximately how many
doctoral students were admitted to your quantitative methods
program each year? ________
- On average, how much time do the full-time tenure/tenure
track faculty in your quantitative methods program spend on
*research*?

_____ >80%

_____ 70%-79%

_____ 60%-69%

_____ 50%-59%

_____ <50%

- Over the last 8-10 years, what types of careers have the
majority of graduates of your doctoral program in
quantitative methods entered into? (Check all that apply).

_____ Positions in the private sector (i.e., small businesses, corporations)

_____ University faculty positions

_____ Positions in the public sector (e.g., school districts; non-profit educational organizations, including testing companies; non-profit research institutions; non-faculty university positions)

_____ Other (please explain):

- Where do the doctoral students in your quantitative
methods program take their
*required*statistic(s) courses? (Check all that apply).

_____ In the School/College of Education

_____ In the Mathematics/Statistics Department

_____ In the Psychology Department

_____ Other (please explain):

- Please check each of the following topics or methods
that
*most*or*all*of the graduates of your doctoral program in the quantitative methods have received training in:### ANOVA

_____ Nonorthogonal designs

_____ Covariance analysis

_____ Repeated measures designs

_____ Cell means models

_____ Random-effects models

_____ Mixed-effects models

_____ Complex designs (e.g., fractional factorial)

_____ Thorough coverage of multiple comparison procedures

_____ Power/sample size calculations

_____ Other: Please describe

### Multiple Regression

_____ Ordinary least squares estimation

_____ Weighted least squares estimation

_____ Nonlinear-in-the-predictors models

_____ Nonlinear-in-the-parameters models

_____ Logistic regression

_____ Other: Please describe

### Traditional Multivariate Procedures

_____ Canonical correlation

_____ MANOVA

_____ MANCOVA

_____ Discriminant analysis

_____ Principle Components Analysis

_____ Factor analysis

_____ Log-linear models

_____ Other: Please describe

### Nonparametric Procedures

_____ Kruskal-Wallis test

_____ Repeated measures tests (e.g., Friedman, Cochran, Hodges-Lehmann block test)

_____ Asymptotic relative efficiency

_____ Exact tests

_____ Rank-transform tests

_____ Jackknifing/Bootstrapping

_____ Other: Please describe

### Other Topics and Procedures

_____ Causal models

_____ Meta-analysis

_____ Time-series models

_____ Multilevel models/hierarchical linear models

_____ Matrix algebra

- How would you rate the
*mathematical/statistical training*of most of your graduates from the quantitative methods program? Use the following scale:**4 = Strong**, perhaps took multiple courses in mathematical statistics, probability theory, would probably be well-acquainted with topics such as quadratic forms, Likelihood Ratio Test Principle, Gauss-Markoff Theorem, etc).**1 = Weak**, little or no coursework in mathematical statistics or probability theory, probably not familiar with Gauss-Markoff Theorem, etc.).Weak Strong 1 2 3 4

- Please rate the
*computer skills*of the graduates from your quantitative methods program.Weak Strong Standard data analysis programs (e.g., SAS; SPSS; SYSTAT; BMDP) 1 2 3 4 Database management 1 2 3 4 Computer programming (e.g., FORTRAN; PASCAL; C) 1 2 3 4

- Do you think that students who graduated from your
quantitative methods program in the last year or two are
better prepared to do publishable statistical work than
students who graduated 5 or 6 years ago?

_____ I think they are*better*prepared now

_____ I think they are*less-well*prepared now

_____ I think they are the*same*

- In hindsight, of the quantitative methods program
doctoral students who graduated from your program in the
last 5 years, what percent do you think could have profited
from:
Less More Can't than Half than Half Judge 1-2 additional (probably <50%>50% CJ specialized) statistics courses 3 or more additional <50%>50% CJ (probably specialized) statistics courses 1-2 additional qualitative <50%>50% CJ methods courses

- Has the increased prevalence of qualitative research
methods had an impact on training doctoral students in your
quantitative methods program?

_____ yes

_____ no

If yes, please explain.

**Directions:** We are now interested in finding out about
methodological training for doctoral students in your School
or College of Education who are NOT in your Quantitative
Methods program.

- Which of the following statements best describes the
*statistics*requirement in your School/College of Education for doctoral students who are NOT in the Quantitative Methods Program?:

_____ Students in some programs can graduate without taking a statistics course

_____ Students in all programs are*required*to take at least*one*statistics course

_____ Students in all programs are*required*to take at least*two*statistics courses

_____ Don't know

- Which of the following statements best describes the
*qualitative research*requirement in your School/College of Education for doctoral students who are NOT in the Quantitative Methods Program?:

_____ Students in some programs can graduate without taking a qualitative methods course

_____ Students in all programs are required to take at least*one*qualitative methods course

_____ Don't know

- Over the past five years, what changes, if any, have
you seen in terms of enrollment in
*statistics*courses for doctoral students in your School/College of Education for doctoral students who are NOT in the Quantitative Methods program?

_____ In the past 5 years, doctoral students have tended to take*fewer*statistics courses

_____ In the past 5 years, doctoral students have tended to take*more*statistics courses

_____ In the past 5 years, doctoral students have tended to take*about the same*number of statistics courses

_____ Don't know

- Think about the doctoral students in your
School/College of Education who are
*NOT*in your quantitative methods program that you have had contact with over the past 5 years or so -- through dissertation committees, classes, consulting, and so forth.*Of these

*non*-quantitative students, what percent do you think are competent to*critically read and interpret*research articles that utilize the following procedures? (circle your answers)Less More Can't than Half than Half Judge ANOVA <50%>50% CJ OLS regression <50%>50% CJ MANOVA <50%>50% CJ Log-linear models <50%>50% CJ Nonparametric rank tests <50%>50% CJ Causal models <50%>50% CJ

*Of these

*non*-quantitative students, what percent do you think could have profited from:Less More Can't than Half than Half Judge 1-2 additional (probably <50%>50% CJ specialized) statistics courses 3 or more additional (probably <50%>50% CJ specialized) statistics course 1-2 additional qualitative <50%>50% CJ methods courses

Aiken, L. S., West, S. G., Sechrest, L., and Reno, R. (1990), "Graduate Training in Statistics, Methodology, and Measurement In Psychology: A Survey Of Ph.D. Programs In North America," American Psychologist, 45, 721-734.

Anonymous (1995), Graduate Study in Educational and Psychological Measurement, Quantitative Psychology, and Related Fields, College of Health and Human Development, Pennsylvania State University.

Brogan, D., and Kutner, M. H. (1986), "Graduate Statistics Service Courses," The American Statistician, 40, 252-254.

Dawson-Sanders, B., Azen, S., Greenberg, R. S., and Reed, A. H. (1987), "The Instruction of Biostatistics in Medical School," The American Statistician, 41, 263-266.

Elmore, P. B., and Woehlke, P. L. (1988), "Statistical Methods Employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research from 1978 To 1987," Educational Researcher, 19-20.

Guo, F., and Nitko, A. J. (1996), "Graduate Programs that Prepare Educational Measurement Specialists," Educational Measurement: Issues and Practice, 15, 28-31.

Noether, G. E. (1980), "The Role Of Nonparametrics in Introductory Statistics Courses," The American Statistician, 34, 22-23.

Tukey, J. W. (1980), "We Need Both Exploratory and Confirmatory," The American Statistician, 34, 23-25.

West, C. K. and Rhee, Y. (1995), "Ranking Departments or Sites within Colleges of Education Using Multiple Standards: Departmental and Individual Productivity," Contemporary Educational Psychology, 20, 151-171.

Deborah A. Curtis

DAIS/College of Education

San Francisco State University

1600 Holloway Avenue

San Francisco, CA 94132

Michael Harwell

Department of Psychology in Education

5B27 Forbes Quad

University of Pittsburgh

Pittsburgh, PA 15260

Return to Table of Contents | Return to the JSE Home Page