Email from Frank Schmidt in May 2002:

…Attached is an in-press book chapter on VG and meta-analysis that has two pages (pp. 21-22) that discuss your Bayesian approach to meta-analysis. This discussion, however, is incomplete. I think a consideration not discussed in this chapter is that the Bayesian approach implicitly accepts the hypothesis of situational specificity. That is, it assumes that if the result in the local study is different from the mean of the prior, this difference is due to something specific about the local setting. But there are many reasons why any particular study might produce different results (whether it is a local study or not)--nonsubstantive reasons beyond the artifacts of range restriction, measurement error, and sampling error, which are taken into account. For example, there may be something peculiar about the criterion measure used, some bias on the part of the researcher that affected the result, etc., etc. There are many such possibilities. So it seems inappropriate to assume that if the local study is different, that difference indicates situational specificity. So my objection is not mathematical--it is conceptual.

Hope this helps.

Frank

The following text represents Frank Schmidt’s (May 2002) view on Bayesian VG. This is the material he mentions in the paragraph above. It is an excerpt from:

Schmidt, F. & Hunter, J. (in press). History, Development, Evolution, and Impact of Validity Generalization and Meta-Analysis Methods, 1975 – 2001. In K. R. Murphy (Ed.) Validity Generalization: A Critical Review. Mahwah, NJ: Erlbaum.

Bayesian Models in VG

After a VG analysis is completed for a particular predictor-job combination, it is likely that within a short period of time a new primary validity study will appear on that same predictor-job combination. How should that that new study be combined with the pre-existing VG meta-analysis? After the first few years of work in this area, the answer we adopted is that the VG analysis should be updated (rerun) to include this new study and any others that have appeared since completion of the original VG analysis. This ensures that each new study will be treated and weighted in the same way as other studies. This procedure also has the advantage of being general: it is the procedure of choice for handling new studies in all other research areas (non-VG areas) inside and outside of I/O psychology.

However, there is another possibility, one that we considered in our early publications (e.g., Schmidt and Hunter, 1977): the new study can be combined in a Bayesian way, using Bayes Theorem, with the distribution of operational validities from the original meta-analysis. That is, and from the VG study is taken as defining the Bayesian prior distribution and this distribution is multiplied by the results of the new study (the likelihood function) to produce a Bayesian posterior validity estimate. This estimate can be taken as applying to the setting in which the new study was conducted. Brannick (2001) has recently re-visited this possibility and has recommended this approach. We discontinued our advocacy of this approach when it became apparent that it leads to overweighting of the new primary study. That is, this approach typically gives much heavier weight to the single new study than it would get if the VG study were rerun including this study. In view of findings casting doubt on situational specificity of validities, it is highly doubtful that such heavy weights are justified. In addition, we believe that any procedure used in VG should have the quality of generality: it should be equally appropriate if applied to non-VG research literatures. It is hard to think of a research literature in which it would be appropriate to give much larger weight to one study simply because that study happened to be conducted after the relevant meta-analysis was completed. Hence we believe that all VG studies (and all meta-analyses in all research areas) should be updated and rerun periodically by incorporating new studies that have become available in the interim. This is the best way to incorporate new data into the cumulative research literature.

It should be noted that even if Bayes Theorem is used as described above, it can only be used in the heterogeneous case (i.e., where > 0). In the heterogeneous case, only a random effects meta-analysis model can be used; random effects models allow > 0, whereas fixed effects models assume a priori that = 0. (See discussion below). Brannick (2001) argues that the Bayesian procedure can be used with fixed effects models as well as with random effects models. In fact, it is a logical contradiction to apply Bayes Theorem using the fixed effects meta-analysis model. If the fixed effects model is used, then of the prior Bayesian distribution is by definition zero, and the final (posterior) validity estimate is always the same as the from the prior meta-analysis. That is, the new study will always get an effective weight of zero and will not have any effect on the final estimate. Hence the Bayesian model cannot be used with fixed effects meta-analysis models.