Issue 4
Vol. 43
2001
|
|
|
|
I.
Stelzl The before-after-control-group-design (experimental group: measurement - treatment - measurement; control group: measurement ------------- measurement) is one of the most common research designs in evaluation research. The present study compares four statistical strategies to answer the question whether individual differences in the treatment effect (some people may gain more from the treatment than others) can be predicted from the first measurement: (1) One may compute the correlation between the first measurement (X) and the difference score (Y - X) from the data of the EG. (2) One may compute the correlation between the first measurement and the residual-gain-scores in the EG (The data of the CG are used to determine a regression equation to predict the second measurement Y from the first X, if no treatment is applied. This equation is used to compute a predicted score without treatment for each subject of the EG. The residual-gain-score is the difference between the scores Y which the subjects of the EG obtained after treatment and the predicted score without treatment). (3) One may split both EG and CG by the median of the first measurement in two subgroups (high vs. low) and handle the design as a repeated measurement-design by ANOVA. (4) One may split EG and CG by the median of the first measurement, as before, but may use only the second measurement as dependent variable in the ANOVA-design. Procedure (1) is not recommended, as it does not answer precisely the question posed: The difference score Y - X confounds the treatment effect with many other effects which may cause change, e.g. effects of practice, memory, regression to the mean etc. Procedure (2) takes this argument into account and corrects for other sources of change. However, a Monte-Carlo-study leads to the result, that significance testing of the correlation between the first measurement and the residual-gain-score leads to actual alpha values considerably above the nominal value. For procedures (3) and (4) one may expect problems from non-normality and heterogeneous variances. Monte-Carlo-studies lead to the result, that ANOVA is robust, if the number of subjects is equal in all groups. With unequal group sizes the Welch-James modified F-test leads to a good approximation to the nominal alpha. Key words: evaluation research, before-after-design, measurement of change, change-score, difference score, residual-gain-score, analysis of variance, robustness of ANOVA, Welch-James-statistic, Monte-Carlo-study Prof.
Dr. Ingeborg Stelzl
|
