Empirical Bayes Local Validation Estimates with Artifact Corrections

Michael T. Brannick

University of South Florida

Steven M. Hall

Embry-Riddle Aeronautical University

Yufan Liu

University of South Florida

(2002, April). Paper presented in S. Morris (Chair) Rethinking artifact corrections in Meta-analysis: Innovations and Extensions. Symposium presented at the 17th Annual Conference of the Society for Industrial and Organizational Psychology, Toronto.

Abstract

We present a method that allows the user to compute a ‘best’ estimate of local validity by combining two sources of information. The first source is a local validation study that includes corrections for criterion reliability and direct range restriction on the predictor; the second source is a previously completed meta-analysis. The method is evaluated using Monte Carlo data.

[Slide 1 – Title (Slides are in a separate powerpoint file.)]

In the l977, Schmidt and Hunter developed a method of meta-analysis that they labeled ‘Bayesian.’ They showed how the method could be used to estimate a mean and variance of infinite sample size validation studies. They also noted how the method could be used to compute empirical Bayes estimates for local validation studies by combining such studies with a previously completed meta-analysis. They said that the proper thing to do in such a case was to use a local study that had been corrected for range restriction and criterion reliability. But they didn’t show how to do so. That’s our job for today. We’ll show you how to combine a Hunter and Schmidt (1990) meta-analysis with a local validation correlation corrected for criterion reliability and range restriction. You can download a paper and slides that describe all this by going to my web site: http://luna.cas.usf.edu/~mbrannic and clicking on ‘conference papers.’

Although Schmidt and Hunter dropped the Bayesian label, other people (particularly Raudenbush and colleagues, e.g., Kalaian & Raudenbush, 1996; Raudenbush & Bryk, 1985) used hierarchical linear modeling programs for meta-analysis. Such programs produce empirical Bayes estimates of effect sizes for individual studies. In 2001, I (Brannick) presented a Bayesian analysis of test validation that showed how to compute empirical Bayes estimates of local validities for both fixed- and random-effects meta-analyses, but I didn’t deal with correlations corrected for reliability and range restriction.

I’m going to show you how to do that in a minute, but first consider how meta-analyses of test validation data are typically completed. When we think of the results section, we typically think of the mean and standard deviation of a distribution of effects sizes, that is, the overall summary. The typical study is reported something like what you see in the next slide. The point of this slide is not the particular numbers, but the column labels. We typically see the total N, the number of studies, the mean observed correlation, the observed variance, and then the disattenuated mean and standard deviation of infinite sample size studies. Usually there is an overall analysis and then subgroup analyses for moderators.

[Slide 2 – The Usual Stuff]

The big difference between the conventional meta-analysis and the Bayesian meta-analysis is that in the Bayesian meta-analysis, we use information from the other studies to adjust individual or local studies to be more like the grand mean effect size. So in a Bayesian analysis, we get a revised estimate for every local validation study. We can also take a previously completed meta-analysis and use that information to adjust a new local validation study. Bayesian statistics provide a means to say how much the local validation studies should be adjusted. The bottom line is, essentially, that the amount of adjustment should be made relative to the precision of the estimate in the local validation study and the precision of the estimate in the meta-analysis. If the local validation study is not very precise and the meta-analysis is very precise, we should give more weight to the meta-analysis than to the local validation study. If the local validation study is very precise and the meta-analysis is not very precise, we should give more weight to the local validation study than to the meta-analysis.

[Slide 3 – Empirical Bayes Estimates]

Brannick (2001) showed how to do this for both fixed- and random-effects scenarios using correlations that are not adjusted for reliability or range restriction. Today our goal is to show an approximate method to do the same job where we correct for criterion reliability and range restriction. After we show you how to do it, we will present some Monte Carlo data and comment on the quality of the approach.

First, let’s do an example without corrections for range restriction and reliability. Today I’m only going to talk about random-effects scenarios; the fixed effects work is simpler but probably does not apply well to real data (Hunter & Schmidt, 2000). In the random-effects scenario, there is some true variance in population correlations. That is, if we could measure correlation across situations using infinite sample sizes, there would still be some variance in the correlations. We want to estimate that variance. Hedges and Vevea (1998) have presented one way to do so; Hunter and Schmidt (1990) have presented another method.

The basic idea is simple. We want to combine information from a meta-analysis with information from a local validation study. The Bayesian notion is to find a weighted average where we weight the two pieces of information by their respective precisions. In the case of the local study, we have N, the sample size. The sampling variance of z, our local validation statistic, will serve as the precision of this bit of information. (Bayesians consider the variance of z to be 1/N but Fisherians consider it to be 1/(N-3). It will hardly matter in practice.) The variance of rho in the meta-analysis will serve as the precision of the meta-analysis. The precision of both the local validation and the meta-analysis can be expressed either in terms of a variance or a sample size.

[Slide 4 – Inverse Variance Weights]

Suppose we conducted a local validation study with an N of 150 and found an observed correlation of .245, so that after transforming to z, we have .25. According to the Bayesians, the expected sampling variance is 1/150 = .007 (you also get .007 if you use N-3). Then suppose we found a relevant meta-analysis that was computed in z. Suppose our mean z is .50, and our estimated SD is .10, that is, the SD of distribution of true effect sizes, or in other words, the infinite sample size SD.. Our estimated random effects variance component is thus .01 (=SD² = .10²). This is the variance or uncertainty about the meta-analytic mean as it applies to the local validation study. We are treating the meta-analytic estimates essentially as if they were the results of another single study. Such a practice will be reasonable so long as the meta-analytic estimates are accurate (based on large N and k).

The empirical Bayes estimate is just a weighted average of the two values. We multiple each value by the inverse of the variance of each estimate. This can be thought of as weighting by sample size equivalents. In our example, the inverse variance weight for the meta-analysis is 100 (because 1/.01 = 100), and for the local study is 150. The resulting empirical Bayes estimate is .35. The variance of that estimate is the reciprocal of the sum of the weights (=1/250).

It can be shown that if both constituent elements (likelihood and prior or local study and meta-analysis) have normal distributions, then the weighted average will also have a normal distribution, and one can compute appropriate confidence intervals (Box & Tiao, 1973; Lee, 1989). I took the square root of that variance (1/250), multiplied by 2 and added and subtracted to get the confidence interval in z. I back translated from z to get the same interval in r. This is the method you should use if you follow the random-effects method of meta-analysis developed by Hedges & Vevea (1998) or the method shown in Lipsey and Wilson (2001) and this is what I presented in Brannick (2001).

Schmidt and Hunter (1977) argued that if you were going to find empirical Bayes estimates, that such estimates should be based on local corrected correlations, that is, on local estimates that have been adjusted for reliability and range restriction. This is a problem because in order to combine the estimates from the meta-analysis and the local validation study, we have to have the variance of the corrected local coefficient. Fortunately, Raju, Burke, Normand, Langlois (1991) provided formulas for correcting local estimates and for estimating their sampling variance.

[Slide 5 - Bayes Estimate in r (1)]

Suppose we did a local validation study on 75 people with direct range restriction and we found an observed correlation of .15. Our estimate of reliability of the criterion in our validation sample was .75, and the ratio of the standard deviations for range restriction turned out to be .75, so the reciprocal of that is 1.33. We now have all the data we need to compute the disattenuated correlation. We find a value of .227 as our disattenuated correlation.

[Slide 6 – Bayes Estimate in r (2)]

Now we need to find the sampling variance of this estimate. According to the formulas in Raju et al. (1991), the estimated sampling variance for our example is .028. It takes a couple of minutes to calculate, but you can see how to do it from the slide. In the third step, we have to find a relevant meta-analysis. We are looking for one that employs the method described by Hunter and Schmidt (1990). Both the mean and variance expressed in such an analysis will be in the metric of r rather than z, and (we assume) will be corrected for range restriction and criterion reliability. Suppose we find one where rho-bar is .50 and Sdrho is .15. Then the variance of rho would be .0225.

[Slide 7 – Bayes Estimate in r (3)]

For the final step, we have to combine the local and meta-analytic estimates. We do that in just the same manner we did when we were working in z. When we do that, we find that our best estimate of the local (disattenuated) rho is .38, and the confidence interval ranges from .15 to .60. I have not asked them, but so far as I can tell, this is what Schmidt and Hunter had in mind in 1977 for the Bayesian aspect of the analysis.

When I tell some people about this, they tell me something is funny about it. Why should we use the variance of rho to weight the meta-analysis but the sampling variance of r to weight the local?. The short answer is that the variance of rho is the estimate of precision for a random-effects meta-analysis and the sampling variance is the estimate of precision for a local; therefore, precision to precision is being compared. But some do not find this argument compelling.

For a slightly longer answer, let’s do a little thought experiment. Let’s assume that in any given local context that the validity is constant or fixed over a reasonable time period for a given job and test. Let’s further assume that local contexts vary in lots of independent ways that either push the correlation up or down a little bit. Therefore, over contexts, it’s reasonable to expect a normal distribution of validities and thus the random-effects case applies to the meta-analysis. Now suppose that we had incredible funding and a wonderful improbability drive so that we could actually collect a whole bunch of infinite sample size validity studies so that we could actually compute the mean and variance of infinite sample sizes directly in a meta-analysis. Now we know the distribution of rho, including rhobar (which let’s say for sake of argument turns out to be .4) and Sdrho (which say turns out to be .10). But we don’t know the value of the validity in any randomly selected context, so we have to estimate that. Say we grab a context at random, sample 100 people from that context, and find the local r is .30. How should we combine that information with what we already know? Before we did the local validation study, our best guess of the local validity would be .4. And if we ignore the meta analysis, our best guess is .3. Based on what we know, our best guess should be somewhere between .3 and .4. In deciding how close our best guess is to either .3 or .4, two things matter. Those things are (1) the local validation sample size and (2) the variance of the distribution of rho. Therefore those are the quantities we use in the empirical Bayes estimate.

[Slide 8 – Monte Carlo Distribution of disattenuated r]

Strictly speaking, the confidence interval for the disattenuated correlation should depend upon the sampling distribution of the disattenuated correlation. The intervals shown here are based on the assumption that the sampling distribution is normal. We used Monte Carlo Methods to generate some correlations, attenuate them for criterion reliability and range restriction, and then disattenuate them using the formulas from Raju et al. (1991). When we did that, we found the distribution of the disattenuated correlation to be slightly biased and skewed, much like the distribution of correlations that have never been attenuated. A distribution of disattenuated correlations is shown in Slide 8. There are 1000 correlations where N=100 for each sample. Not too bad.

[Slide 9 – Monte Carlo Distributions for never attenuated, attenuated, and disattenuated r]

Slide 9 shows raw correlations from a population with rho =.40 in the first boxplot. The second boxplot shows what happens when we have direct range restriction (take the top half of applicants; SR = .5) and criterion reliability of .75. The third boxplot shows the correlations disattenuated by the Raju et al. (1991) formulas. As you can see the distribution looks pretty good. The center of the distribution goes back where it should. The price we pay for disattenuation is that the distribution of corrected correlations has a bigger variance than the distribution of uncorrected correlations.

Summary. We showed how to compute empirical Bayes estimates of local validity while correcting for range restriction and criterion unreliability. Go forth and compute to your heart’s content. A couple of caveats. We treat the meta-analytic estimates as known population values. If these values are not well estimated, the confidence intervals for the empirical Bayes estimates will be too small, kind of like the difference between the z and t distributions. Avoid using this procedure unless your local r is based on at least 100 people because the distribution of r tends to be naughty unless sample size is large.

References

Box, G. E. P., & Tiao, G. C. (1973). Bayesian inference in statistical analysis. New York: Wiley.

Hedges, L. V. & Vevea, J. L. (1998). Fixed- and random-effects mo2els in meta-analysis. Psychological Methods, 3, 486-504.

Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage

Hunter, J. E., & Schmidt, F. L. (2000). Fixed effects vs. random effects mata-analysis models: Implications for cumulative research knowledge. International Journal of Selection and Assessment, 8, 275-292.

Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed linear model for meta-anlaysis. Psychological Methods, 1, 227-235.

Lee, P. M. (1989). Bayesian statistics: An introduction. New York: Halsted Press.

Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.

Raju, N. S., Burke, M. J., Normand, J., & Langlois, G. M. (1991). A new meta-analytic approach. Journal of Applied Psychology, 76, 432-446.

Raudenbush, S. W., Bryk, A. S. (1985). Empirical Bayes meta-analysis. Journal of Educational Statistics, 10, 75-98.