How Measurement Reliability Affects Research

Reliability and Research

1. Reliability and sampling error. The average is always more reliable (constant, consistent) than the individual observations. The greater the number of observations, the more reliable the average.

1. Effect sizes in AVOVA.

2. Correlation and regression coefficients

3. Power in statistical tests. As N increases, the expected value of a point estimate is constant; the confidence interval shrinks.

2. Reliability and measurement error. Techniques like regression, correlation and ANOVA tacitly assume perfect reliability of measurement. When these statistical techniques were developed, measurement error wasn't much of a problem. ANOVA came from agronomy, where typical measure might be bushels of wheat, weight of grain, etc. Early psychologists looked at height, weight, reaction time, and so forth. Spearman felt that error of measurement reduced the size of observed relations (e.g., correlations). He decided to try to analyze such effects. This was the beginning of the now classical notion that reliability sets the ceiling on validity, thus:

Effects of increasing measurement error.

1. ANOVA Effect sizes

reduce effect size
reduce power
increase effect size variance

2. Correlations

reduce absolute value of r
increase variance of sampling distribution
decrease power

3. Regression: slope, intercept, and standard error.

Can have the same slope and intercept (that is, the same regression line) but different standard errors in regression. The standard error of prediction is important for R-square and confidence about b-weights.

A. Effects of unreliability in Y

1. Increases standard error

2. decreases R-square

3. Slope and intercept are NOT affected (in the expected value).

Illustration of increasing the unreliability in Y. Note that the line stays the same, but the scatter of points about the line increases. Such an increase results in a decrease in R-square and an increase in the standard error of the b-weight.

B. Unreliability of X.

1. Regression line always passes through , that is, through the means of both X and Y. Increasing random error to X and Y will not change the expected value of the means of X and Y, but it will increase the variance of both X and Y. As X becomes unreliable, the regression line swivels or rotates about the sample means to become more and more horizontal. This means that the absolute value of the slope will decrease. The intercept will increase or decrease depending on the slope of the original line. Adding error will increase the standard error of prediction and decrease R-square.

An illustration of adding random error only to X. Note how the regression line swivels.