MULTITRAIT-MULTIMETHOD MATRIX
Read the article by Campbell and Fiske. They somehow saw this structure in the data and saw what it meant. Their article is perhaps the most widely cited article in psychology, and is certainly the most widely cited in the psychological methods literature. Their basic idea is this:
When we measure a construct, we use a method. Some examples:
Construct |
Method |
Team Coordination |
Instructor Ratings |
Team Coordination |
Team Member Ratings |
Depression |
Clinical Rating |
Depression |
Beck Depression Inventory |
Mechanical Aptitude |
Bennett Mechanical Aptitutde Test |
Mechanical Aptitude |
Apprenticeship Grades |
The variance in observed scores can be thought of as arising from both the trait (e.g., team coordination) and the method (e.g., instructor ratings). Each measure can be thought of as a trait-method unit. With a single trait method unit, we cannot separate variance due to traits and methods; with multiple trait-method units, we can begin to do so. With multiple trait-method units we can construct a correlation matrix that shows the relations among all the units. The name for such a matrix is (surprise) the multitrait-multimethod matrix, or MTMM for short. There are two basic ideas related to validity associated with the MTMM that are so important that they have their own names: convergent and discriminant validity.
Convergent validity is evaluated when we have one trait and two methods. For example, we could check convergent validity with team coordination (trait), measured by instructor ratings (method 1) and team member ratings (method 2). The idea is that if most of the variance in the observed measures is due to the trait, then the measures will converge on a common number, that is, measures of the SAME TRAIT should be in agreement even if they are taken with DIFFERENT METHODS You would expect good agreement, for example, from weight measured by a spring type bathroom scale (the kind you stand on and peer down at a circular dial on the scale) and weight measured by the balance type scale (where you adjust weights with your arms until the lever balances). We will ignore for the moment specialized knowledge that allows you to fudge the scale to make it read the way you want.
Discriminant validity can be evaluated when we have two traits and one method. For example, we could measure team coordination (trait 1) and team resources (trait 2) with a single method (team member ratings). The idea is that DIFFERENT TRAITS should be distinguished from one another, even if they are measured with the SAME METHOD. We would expect to see small correlations between traits (assuming that the traits are distinct). On the other hand, if we have very different traits measured by the same method
and we still find that the traits are highly correlated, we can bet that much of the variance in the observed measures is due to the method rather than the traits.
Let's take a look at the multitrait-multimethod matrix. Suppose we want to measure three traits: anger, guilt and depression (some of my favorite constructs). Further suppose that we have developed three different ways to measure each of these: a paper and pencil test, a clinical rating and a self rating. Because we have three traits and three different ways of measuring each of them, we will have nine observed variables or trait-method units. For example, we will have a test score for anger, a clinical rating of guilt, and a self rating of depression. Figure 1 shows a schematic diagram of such a situation (never mind the confirmatory factor analysis bit or the little deltas and their associated arrows, i.e., ¬ d ).
The circles stand for traits and methods. The squares stand for observed measures. The straight arrows from the circles to the squares show that the traits and methods both contribute some variance to the observed measures. The curved arrows indicate that the traits and methods may have some relations among themselves, that is, they may be correlated to some degree with one another.
Jargon
If we take the nine observed variables shown in Figure 1 and compute a correlation matrix, we might see something like what is shown in Table 1. Note that each trait meausre by each method is paired with all others, for a 9 by 9 correlation matrix. First, we will talk nomenclature (jargon) to label the parts of the matrix. This will come in handy when we want to communicate with others in the scientific community. It is also indispensable at cocktail parties when you want to baffle people from other disciplines. The first part of the mattrix to notice is the main diagonal of the matrix (see Table 2). The entries on the main diagonal of any correlation matrix are, of course, equal to 1.0. It is often the case in MTMM matrices that the main diagonal is replaced with reliability estimates.
There are two types of blocks of entries in the matrix, monomethod and heteromethod. The monomethod blocks occur where all of the correlations come from traits measured by the SAME METHOD; that's why they are called MONOMETHOD blocks (see Table 3). There are three monomethod blocks. Each of them is composed of all the elements measured by the same method. For example, in the lower right corner, we have self ratings of anger, guilt and depression. The same goes for the center block except that we are looking at clinical ratings.
The other blocks are the heteromethod blocks (see Table 4). The heteromethod blocks are formed where the traits are measured by DIFFERENT METHODS; that's why they are called HETEROMETHOD blocks. There are three unique heteromethod blocks in Table 4, one for each combination of two different methods. The bottom center block is where clinical
ratings are crossed with self ratings. There are two other heteromethod blocks, one for crossing tests and clinical ratings, and the other for crossing tests and self ratings.
Both the monomethod blocks and heteromethod blocks have two types of information in them. In the monomethod blocks, we have the diagonal elements. These are, of course, the main diagonal elements on a correlation matrix, and are equal to 1.0 if the calculations to get the matrix are correct. However, it is customary to replace the main diagonal (1.0) elements of the matrix with estimates of reliability for each variable. Imagine for the moment that all the entries on the main diagonal in Table 1 were replaced by .9s. Then we would be talking about a reliability diagonal. The other entries in the monomethod block are the monomethod triangles (I call these "monos" for short). They show the relations among different traits measured by the same method (see Table 5).
In the heteromethod blocks we also have two types of information. First, we have the same trait measured by different methods. For example, we have anger measured by self ratings and tests. Such entries are called the validity diagonal. In Table 6, the validity diagonal entries are all set equal to .8 and are shown in bold face. The last set of entries in the heteromethod block are called heterotrait heteromethod triangles (I call them "hets" for short). These entries show the correlations of different traits measured by different methods( see Table 7).
Recap:
1. Monomethod Block
a. reliability diagonal
b. (heterotrait) monomethod triangle
2. Heteromethod Block
a. validity diagonal
b. (heterotrait) heteromethod triangles