Understanding Validity

What Is Validity?

Validity refers to the extent to which something does what it is intended to do. A measure intended to predict future success is said to have predictive validity, to the extent that it can predict that success.

How Is Validity Established?

There is no magic formula that you can apply to say that one measure is valid and another is not. Instead, we talk about the degree of prediction—not "Is it valid?" but rather, "How valid is it?" This question can be answered by determining the relationship between the measure used for prediction and a measure of later success.

What Factors Affect the Amount of Validity?

There is a saying that the only things that are certain are death and taxes. Prediction is never perfect. Imagine if one could predict perfectly who would succeed in business school—you would never need to give grades, because you would already know how everyone would perform! Every person has a story of someone they know who succeeded "against all odds" or someone who did not succeed even though "he had everything going for him." There will always be some error associated with prediction that cannot be accounted for by the variables being measured; however, there are also some errors that are expected because of the way the predictive validity is established, such as restriction of range.

What Is "Restriction of Range"?

Restriction of range is commonly explained as a reason why the amount of prediction may be lower than expected. Let's say there are 1,000 people who apply to a particular business school program. Because only 200 can be admitted, the school uses several different criteria to predict who would be the best candidates to be enrolled. The prediction was done for all 1,000 students, but when it comes time to measure the accuracy of that prediction, it can only be done for the 200 students who enrolled, because only those students have a measure of success—grades in the program. Thus, the amount of prediction can only be measured on a restricted-range, select group of students.

Why Would Restriction of Range Make Prediction Appear to Be Less?

Because the group analyzed is a select group of students, chosen for the same characteristics, then they are very much alike with regard to those characteristics. Few differences among them regarding those characteristics mean that those characteristics cannot be used to explain why they have different levels of success; hence, there will be less prediction. As an example, if two students have the same GMAT® score but different first-year grades, then the GMAT® score would appear to have no predictive ability, even if the grades are very close.

What about the GMAT?

The GMAT® exam is intended to predict academic success in a graduate management program. Therefore, a logical measure of academic success, namely average grades halfway through the program, is used in determining the test's predictive ability.

How Valid Is the GMAT?

The amount of prediction observed for the GMAT® exam depends on the program being measured and how the scores are combined. Over decades of studying predictions at different schools, the GMAT® has proven to be one of the best predictors of academic success in numerous types of programs. However, the best prediction almost always occurs when several factors are combined, supporting the advice of the Graduate Management Admission Council® to always consider GMAT® scores in combination with other pertinent information when making admissions decisions.

What Do the Numbers Mean?

The report that comes with your validity study results contains all the statistics from the analysis of your data. Included are summaries of your data, such as average performance, as well as the validity information. The Prediction section gives validity coefficients, which represent the strength of the relationship between the predictors and first-year grades.

For ease in interpretation, we provide a presentation containing the highlights of the report. In the presentation, the validity coefficients are converted to the percent of variance explained, which means that of the differences in grades among students, a portion of those differences is attributable to the predictors. More specifically, if one student has higher grades than another student, then part of the reason owes to the abilities measured by the GMAT® and by undergraduate grades.

What Would Be a Typical Result?

Based on the last few years of available data, GMAT® scores combined with undergraduate GPA explain about 20% of the variance in first-year GPA. This does not mean that predictions made on the basis of this information will only be 20% accurate. What it does mean is that if you are trying to decide how one student may perform compared with another student, the GMAT® and undergraduate GPA data will provide you with information specifically related to that performance to help you make that decision.

If Only 20% Is Explained, What about the Other 80%?

There will be some variance that can never be explained, like the "against all odds" cases. More accurately, you might say that there are many differences that cannot be explained by things that are measurable. What other criteria do you look at for admissions decisions? It is likely that these factors make up at least part of the 80%, though often the difference is not much. Limitations of restriction of range and other measurement issues often make it unlikely that validity will be very high.

What is important to remember is that although nothing is 100% accurate, if a measure gives you at least some information that you wouldn't have otherwise, then it is valuable in making an informed decision. The GMAT® exam, for instance, provides schools with standard information about ability that is different from what you could learn from previous grades or from interviews and essays. Even if it adds only a small amount to prediction, admissions decisions are better made with it than without it.

Why Do Programs Have Different Predictive Validity?

Even though the GMAT® exam is the same, regardless of the program it is used for, predictive validity may differ because of specific factors of a particular program or school. Because schools differ in their degree of selectivity, grading policies, program requirements, and other characteristics, then naturally, the predictions would differ, because each of these differences will contribute to differences in validity.

Why Should My School Conduct a Validity Study?

Because programs are all different, it is important for each one to know how the selection criteria uniquely interact to result in the best possible prediction. Each validity study can be customized to answer the questions that are most important for the programs at your school.

Where Can I Get More Information?

You can get more information on GMAT® validity and the Validity Study Service (VSS) by contacting GMAC® at vss@gmac.com.