Policy Research in Education: Some Limitations to Think About
These days education research is increasingly becoming coincident with policy research. In fact, a reviewer’s comments on one of the papers I co-authored went like this: It did not deserve to be published because it was not policy relevant. Upshot: It was bad research if it was not immediately about policy. In this brief note, I suggest some questions about exclusive privileging of policy research in education, its consequences for what counts as data and knowledge under this development, and why this is hazardous. While it could sound like one to some, this post is not meant to be an indictment of policy relevant education research. I do value policy research in education, like most of my colleagues. But I increasingly feel that we do not talk about its limitations. This post is an attempt to just do that.
All research that can potentially inform education policy can be characterized as policy-relevant research. Some scholars have further sub-divided policy-relevant research into policy analysis and policy research while distinguishing both from disciplinary research in general (see, for example, Weimer 2009). Policy analysis is about systematic assessment of the alternative policy options that governments can adopt to address problems of public concern. Policy research seeks to systematically assess the impacts of particular interventions. Thus, the analysis precedes and informs the policy decisions, while policy research primarily concerns the review and evaluation of particular policies. Notwithstanding the difference in timing and purposes of policy analysis and policy research, there are methodological similarities between the two.
First, there is an increasing trend in this kind of education research to take the student scores on the achievement tests of sorts as the de facto regressands. The research is designed to see how [whether or not] they go up and down or stay the same as a result of a change in some of the many independent variables [or regressors] in controlled settings. Henceforth, if we are policy researchers, we begin to operate in a research arena occupied by regressors and regressands only.
Secondly, the Randomized Controlled Trials (RCTs), which were the gold standard for clinical trials earlier, are increasingly becoming the preferred approach to education research. Needless to say, RCTs and other quantitative design methodologies are highly effective due to their promise to neatly test the hypothetical and actual effectiveness of education policies and programs in terms of their effect on student achievement. Yet, as I will briefly tell below, they may be equally highly restricting despite their scientific appearance and methodological elegance. As researchers with interest in education, we need to be cognizant of their promises as well as perils.
First, a sheer focus on outcomes hides from our view what happens inside the classrooms. I was recently engaged in examining transcripts of classroom conversation of about 43 odd classrooms in Punjab as part of a study. The more I read them, the more I realized that these classrooms were not the sites of meaningful education. Here is a small vignette from a math lesson to illustrate the point:
In this vignette the teacher was telling the students exactly what to do. Read this carefully, and if you were a reader equipped with some basic arithmetic knowledge, you’d figure that these students were not likely to develop a conceptual understanding of division of fractions, much less to use this concept meaningfully in real life situations. There is a full range of concepts that the students must already know before understanding division of fractions. Did these students know them? The teacher never asked them any questions to assess their existing knowledge so had no way of ascertaining what they knew [or not]. Yet, these students would successfully attempt the problems involving division of fractions if they were presented in the above form by simply remembering a few simple rules.
Under the circumstances, what information the student test score would contain is anyone’s guess. This teacher could also be judged as a high performer if his students scored high on the test. He might also get an award or two if there was a performance reward policy in place. Upshot: research that only looked at the achievement in terms of scores would be blinded to the highly undesirable nature of classroom interactions. The conclusions reached about high [or low] performance of teachers would reward undesirable instructional traits. Regardless of their intentions to do the opposite, the score-driven research, as well as the policy based on it, could lead to undesirable patterns of instruction. Now some of you may call this point of view naive and argue that better tests and well-designed research can overcome these problems. They will also refer me to advancements in Item Response Theory (IRT) and other such developments. Well that may be true, but my argument is that if you focus your sights on the end points, you do it at the cost of understanding the process. We need to always be aware of what our limitations are and the consequences of such limitations for the policy.
Secondly, when test scores assume high stakes, the incentives to game the system also increase. I have documented elsewhere the instances of cheating in the US public school system as a possible consequence of a policy emphasis on test scores. What has that got to do with policy research? A lot! The policy research strand, with which I am concerned in this post, fosters a system of accountability and teacher performance management based on student test scores.
Thirdly, and most importantly, a focus on student achievement should not be allowed to blind us from asking and responding to other important questions about purposes of education, its overall philosophy, history of education, and those that dug deeper into the dilemmas and challenges of teaching regularly encountered by professional teachers.
Before closing the post, I would like to reemphasize that education policy is not just about increasing test scores. It should also be about improving the processes. Not every aspect of a process, and not every element of an education system, will have a direct bearing on student achievement. We should keep this in mind while defining the scope of policy research in education.
References
Weimer, D. L. (2009). Making Education Research more Policy Analytic. In G. Sykes, B. Schneider, D. N. Plank,and T. G. Ford (Eds.), Handbook of Education Policy Research. Routledge Publishers, New York, NY.
The author would like to thank Campaign for Quality Education (http://www.cqe.net.pk) for providing the classroom vignette used in this post.
