Measuring long-term effects of a school improvement initiative

Sammanfattning: There is a growing demand for studies applying quantitative methods to large-scale data sets for the purpose of evaluating the effects of educational reforms (UVK, 2010). In this thesis the statistical method, Propensity Score Analysis (PSA), is presented and explored in the evaluating context of an extensive educational initiative within science and technology education; the Science and Technology for All-program (NTA). The research question put forward reads; under what conditions are PSA-analyses a useful method when measuring the effects from a school improvement initiative in S & T?The study considers the use of PSA when looking for long-term effects that could be measured, what to take into consideration to be able to measure this, and how this could be done. The baseline references (outcome variables) used in order to measure/evaluate the long-term effects from the studied program is students’ achievements in the national test (score and grades) and their grades in year 9. Some findings revealed regarding the object of study (long-term effects from using NTA) are also presented.The PSA method is found to be a useful tool that makes it possible to create artificial control groups when experimental studies are impossible or inappropriate; which is often the case in school education research. The method opens up for making use of the rich source of registry data gathered by authorities. PSA proves reliable and relatively insensitive to the effects of covariates and heterogeneous effecter if the number of samples is large enough. The use of PSA (or other statistical methods) also makes it possible to measure outcomes several years after treatment. There are issues of concern when using PSA. One is the obvious demand for organized collection of measurement data. Another issue of concern is the choice of outcome variables. In this study the chosen outcome variables (pupils’ score and grading in national tests and grades in year 9) open up for discussions regarding aspects that might not be reflected/measured in national tests and/or teachers’ grading. Findings regarding the long-term effects from using NTA) show significantly positive effects in physics on test scores (average increase 16.5%) and test grades, but not in biology and chemistry. In this study no significant effects are found for course grades. PSA approach has proved to be a reliable method. There is however a limitation in terms of the method's ability to capture more subtle aspects of learning. A combination of quantitative and qualitative approach when studying long-term effects from educational intervention is therefore suggested.