Over the last 20 years, US schools have widely adopted annual student testing, a move which many researchers believe may have been of little benefit to students’ educational outcomes. Julian R. Betts, Youjin Hahn and Andrew C. Zau examine the impact of a different type of mathematics testing – one which is aimed at determining students’ strengths and weaknesses in math. They find that since 2000 the Mathematics Diagnostic Testing Project in California’s second largest school district has increased student’s state test score by up to 4 percentage points. They attribute these increases to the detailed information and insights into student performance the tests provide compared to more traditional examinations.
In the last two decades, the American public education system has widely adopted annual testing of students in subjects such as math and English Language Arts. Part of this change originated with the passage of the federal No Child Left Behind (NCLB) law in 2001, which tied federal subsidies of public school systems to annual testing and interventions and sanctions for schools with low test scores.
So, has this widespread move towards annual standardized state testing actually benefited students? Researchers have reason to believe that testing alone brings few benefits. State tests typically provide overall scores for a student and perhaps subscores in a few subcomponents of the test, but not enough to direct either the student or the teacher.
In new research, we examine the impact of a different type of test which is intended to diagnose specific student strengths and weaknesses. Our findings suggest that diagnostic testing can both change the way that teachers teach their students, and boost test scores.
Multiple choice math testing
The Mathematics Diagnostic Testing Project (MDTP) has provided free math testing to California schools for several decades. These multiple choice tests, carefully designed by university professors and secondary school math teachers, offer course-readiness assessments that provide teachers feedback on their students’ strengths and weaknesses in mathematics.
The first advantage of MDTP tests is rapid turn-around. If teachers administer tests on paper, they will receive feedback on individual students and the entire class within a week of testing. More recently, MDTP testing has largely moved online, giving teachers instant access to results. This contrasts strongly with many states’ annual tests, which often take several months to process. The second advantage of the MDTP tests is that they provide more detailed feedback about each student’s areas of strength and weakness than do the former or current California state mathematics tests. MDTP attempts to “diagnose”, not just “report”.
Figure 1 shows an example, where a teacher hovers the cursor over the most frequently chosen wrong response to an individual question, and receives feedback on likely student misunderstanding.
Figure 1 – Example of How a Teacher Can Quickly Check Student Responses and Areas of Misunderstanding
In our study, we evaluate the effect of MDTP testing on students’ mathematics achievement, using detailed student-level data from the San Diego Unified School District (SDUSD), the second largest district in California. Beginning in 1999-2000, the district began mandating the end-of-year testing of all students in certain grades. The goal of this mandatory testing was to help teachers place students in mathematics classes of appropriate difficulty in the following academic year. Schools also used MDTP test results to help identify students who should attend summer school.
We examine the impact of mandatory use of the MDTP test on student math achievement on statewide tests. We take advantage of a “natural experiment” in which mandatory testing in San Diego was phased in over years and grades. We find that MDTP testing boosts scores on the state math test by enough to move students up several percentile points during the course of a year.
Our focus covered earlier secondary school grades in which virtually all California students take the same math test. We find gains of 4 percentage points in a single year for a student initially at the 50th percentile, and 3 percentage points for a student at the 25th percentile.
Mandated MDTP testing does help students
Why did students gain from mandated testing? We examined two channels, tracking and summer schooling. The MDTP was used to more accurately track students into appropriate math classes the following year. The district also used the tests to recommend which students should attend summer school. However, we did not find that these factors explained the gains.
“Exam” by Alberto G. is licensed under CC BY 2.0
Intrigued, two of us, in a team led by Dr. Karen Bachofer, later conducted both a survey of math teachers in San Diego and a more detailed set of interviews with a smaller number of math teachers. Our findings suggest that teachers value this diagnostic form of testing. When we asked about the period when San Diego Unified mandated that all students in certain grades be tested, teachers reported that because they knew these tests were available they would examine the results from the prior spring for their incoming classes, identify weaknesses, and adjust their lesson plans accordingly. For example, fully 93 percent of teachers who administered MDTP tests under the district mandate reported making use of the results during that same school year, and 75 percent of teachers reported making use of mandated test results to assist students in the following school year.
While the MDTP is widely used in California, do our results hold implications for testing in other states? Recently, many states have either revised their own tests or signed on to one of two multi-state testing consortia. Both the Smarter Balanced Assessment Consortium (SBAC) and the Partnership for Assessment of Readiness for College and Careers (PARCC) are designing computer-adaptive tests for use in multiple states, which will provide speedier feedback on student performance than the earlier wave of state tests. In spite of the improvements in these tests, the MDTP approach still offers lessons for how to improve the value of testing. Teachers and students receive very detailed information on performance on very specific math subcategories, while the SBAC test that California now uses offers much broader information on overall subcategories. A second distinction is that with the MDTP a teacher can look at results for each question. If many students picked one of the incorrect answers, teacher can obtain information on the student misunderstanding that probably led to that answer being chosen. The MDTP does this through the clever use of “distractors”, that is, wrong answers that a student could arrive at through a certain incorrect way of thinking about the problem.
Although far from a magic bullet, the diagnostic nature of the MDTP does offer a valuable approach to testing that moves testing from merely summarizing overall achievement, to carefully diagnosing student strengths and weaknesses. It also offers teachers insights into topics that they can use to enhance teaching.
- The authors thank Dr. Kimberly Samaniego, State Director of the MDTP, for providing the example in Figure 1.
- This article is based on the paper, “Can testing improve student learning? An evaluation of the mathematics diagnostic testing project”, in the Journal of Urban Economics.
Please read our comments policy before commenting.
Note: This article gives the views of the author, and not the position of USAPP– American Politics and Policy, nor of the London School of Economics.
Shortened URL for this post: http://bit.ly/2Dpzjkd
_________________________________
Julian Betts – University of California San Diego
Julian Betts is a professor, and former chair, in the Department of Economics at the University of California San Diego (UCSD). He is the Executive Director of the San Diego Education Research Alliance at UCSD (sandera.ucsd.edu) which undertakes studies of many different education policy issues in the San Diego Unified School District (SDUSD). Betts writes on topics ranging from school choice to mathematics interventions, the academic progress of English Learners, and the impact of school resources on short- and long-term outcomes of students. Betts has advised the U.S. Department of Education on several major research projects and regularly serves on its research proposal review panels. He is an Adjunct Fellow at the Public Policy Institute of California and a Research Associate of the National Bureau of Economic Research. He holds a Ph.D. from Queen’s University, Kingston, Ontario, Canada, and a M.Phil. in Economics from Oxford University.
Youjin Hahn – Yonsei University
Youjin Hahn is an assistant professor in School of Economics at Yonsei University in South Korea. She holds a Ph.D. in Economics from UCSD. Before joining Yonsei, she was an assistant professor in the Department of Economics at Monash University in Australia. Her research focuses on topics in Economics of Education, Health economics and Development Economics. Examples of her recent work include long-term evaluation of female stipend program in Bangladesh and identifying the effect of health insurance expansion on young adults’ labor market outcomes.
Andrew Zau – University of California San Diego
Andrew Zau is Senior Statistician at the San Diego Education Research Alliance at UCSD (sandera.ucsd.edu). His research focuses on the effects of policy on student achievement in K–12 education. He holds an MPH in epidemiology from San Diego State University and a BS in bioengineering from UCSD. He has 17 years of experience analyzing SDUSD student-level longitudinal data and has played key roles in evaluations of the impact of reading reforms, diagnostic math testing, and school choice in San Diego, using experimental and quasi-experimental approaches.
