NBER Reporter: High-stakes testing, like school choice, has become an increasingly prominent feature of the educational landscape. Every state in the country, except Iowa, currently administers state-wide assessment tests to students in elementary and secondary school. Federal legislation requires states to test students annually in third through eighth grade and to judge the performance of schools based on student achievement scores.

The debate over high-stakes testing traditionally has pitted proponents arguing that such tests increase incentives for learning and hold schools accountable for their students' performance against opponents who argue that the emphasis on testing will lead teachers to substitute away from teaching other skills or topics not directly tested on the exam. Along with Brian Jacob, I have written two papers that explore a very different concern regarding high-stakes testing -- cheating on the part of teachers and administrators. As incentives for high test scores increase, unscrupulous teachers may be more likely to engage in a range of illicit activities, such as changing student responses on answer sheets, or filling in the blanks when a student fails to complete a section. Our work in this area represents the first systematic attempt to identify empirically the overall prevalence of teacher cheating and to analyze the factors that predict cheating.

To address these questions, we once again turn to data from the Chicago Public Schools, for which we have the question-by-question answers given by every student in grades 3-7 taking the Iowa Test of Basic Skills (ITBS) over an eight year period. In the first paper,(4) we develop and test an algorithm for detecting cheating. Our approach uses two types of cheating indicators: unexpected test score fluctuations and unusual patterns of answers for students within a classroom. Teacher cheating increases the likelihood that students in a classroom will experience large, unexpected increases in test scores one year, followed by very small test score gains (or even declines) the following year. Teacher cheating, especially if done in an unsophisticated manner, is also likely to leave tell-tale signs in the form of blocks of identical answers, unusual patterns of correlations across student answers within the classroom, or unusual response patterns within a student's exam (for example, a student who answers a number of very difficult questions correctly while missing many simple questions).

Empirically, we find evidence of cheating in approximately 4 to 5 percent of the classes in our sample. For two reasons, this estimate is likely to be a lower bound on the true incidence of cheating. First, we focus only on the most egregious type of cheating, where teachers systematically alter student test forms. There are other more subtle ways in which teachers can cheat, such as providing extra time to students, that our algorithm is unlikely to detect. Second, even when test forms are altered, our approach is only partially successful in detecting illicit behavior. We then demonstrate that the prevalence of cheating responds to relatively minor changes in teacher incentives. The importance of standardized tests in the ChiPS increased substantially with a change in leadership in 1996. Schools that scored low on reading tests were placed on probation and faced the threat of reconstitution. Following the introduction of this policy, the prevalence of cheating rose sharply in classrooms with large numbers of low-achieving students. In contrast, schools with average or higher-achieving students, which were at low risk for probation, showed no increase in cheating.

Our second paper on this topic(5) reports on the results of an unusual policy implementation of our cheating detection tools. We were invited by ChiPS to design and implement auditing and retesting procedures implementing our methods. Using that cheating detection algorithm, we selected roughly 120 classrooms to be retested on the Spring 2002 ITBS. The classrooms retested include not only cases suspected of cheating, but also classrooms that had achieved large gains but were not suspected of cheating, as well as a randomly selected control group. As a consequence, the implementation also allowed a prospective test of the validity of the tools we developed in our first paper on the subject.

The results of the retesting provided strong support for the effectiveness of the cheating detection algorithm. Classrooms suspected of cheating experienced large declines in test scores (on average about one grade equivalent, although in some cases the fall in mean classroom test scores was over three grade equivalents) when retested under controlled conditions. In contrast, classrooms not suspected of cheating a priori maintained virtually all of their gains on the retest. As a consequence of these audits and subsequent investigations, disciplinary action was brought against a substantial number of teachers, test administrators, and principals.

The Economics of Education, S. Levitt, Research Summary