Home | Letters to the Editor | www.gmac.com
To print this story, please select “Print” from the File menu of your browser.
Selections Interview with Gayle Garrett, Project Director, U.S. News & World Report’s Best Graduate Schools Guide
Selections: When did U.S. News begin ranking business schools, and why did your publication embark on that endeavor?
Garrett: I’ll let Bob [Morse, director of data research at U.S. News] answer the “when” and fill you in on why. I know that one of the things that he will say is that, especially for graduate schools, the intent was to consider those areas where the largest numbers of degrees are awarded. The largest categories are education and business. That has been fairly constant over the past 10 years. Around here, we talk about the “big five”: business, engineering, education, law, and medicine.
Selections: Why did you think it was important to begin offering rankings for these areas?
Garrett: Again, Bob can offer specifics on why we did this. But those are the areas with the largest numbers of consumers of graduate school education. It just makes sense to try to concentrate on giving them information that would be helpful to them in looking at schools and making comparisons and evaluations.
Selections: Speaking specifically about the business school rankings, whom do you believe are your main stakeholders? To whom do you target your work?
Garrett: There are obviously two audiences: the prospective students and the schools. Certainly, the thrust of the guide is for prospective students. These are people who are going to be making decisions, and the choices that are made editorially, and I should say that kind of editorial choice . . . for example, when we look at the guide itself, and we are putting columns of figures on a page and that page is limited. It is the book editor, and that is Anne McGrath, who makes that kind of decision. I might say, “We should include such and such information,” and she would say, “We can put that on the Web, because we have more room on the Web, but I don’t think that is one of the first things that students want to know.” She is the person who makes the decision about what goes on the page.
Selections: There is more expanded information on the Web?
Garrett: Yes.
Selections: So your main audience is the prospective students, correct?
Garrett: Right. Obviously, if you sit in this position, it doesn’t take very long before you realize a lot of the calls you are getting are from schools. And, clearly, schools watch these figures very carefully.
Selections: How should your survey be used and viewed by students and the schools? How should they be using this information?
Garrett: The students don’t see the survey, per se. They see the results. They are presented in the guide, in the magazine, and also on the Web site. The hope is always that students will use . . . particularly the properties of the Web site. I was just cleaning out my office and putting away some information from a focus group that was held last fall, and one of the clear outcomes of that focus group was that mostly people are getting their information from the Web site. And some said that while they will get the guide and thumb through it to look at the tables, that when they get down to the actual ways that we hope they are going to use it—and that is for doing their own comparisons—they will go to the Web site. Part of the reason is that the Web site makes it easy. It does allow you to bring up records of various schools and have the information displayed side by side for up to three schools at any one time.
Selections: So you’re hoping that the prospective students will use this information to make comparisons among the schools that they are interested in, correct?
Garrett: Yes.
Selections: How should schools be using the information that is published in the rankings?
Garrett: What I hope the schools are using the information for is . . .well, I hope they are being very, very careful about the information that they provide us, so that when it comes out on the Web site, they will be taking ownership of it at the beginning of the process rather than at the end. There is usually not a lot of difficulty with the MBA data, in terms of the survey coming out. But one of the things that I think is a matter of concern, and I know that this is also a matter of concern of the AACSB and the [MBA] Career Services Council, and that is . . . all the time we are trying to get clearer and clearer salary data that we publish, and precisely what it means and to which program it applies and for how many graduates it is applicable. As a matter of fact, I just got through revising our survey for this coming fall, and we have always used—in terms of the admissions data—we have always tried to follow the GMAC criteria. And there have been new criteria this year, and I have incorporated those into the new survey. There are also standards for reporting salary data, and those are put out by the MBA Career Services Council. I understand that GMAC is also going to be looking at those, and that there is a new revision coming out. I just made the decision in doing our new survey that we would not ask for any percentages to be provided to us by the schools, rather only exact counts. We will compute our own percents. There has been some misinterpretation in providing percents, in terms of what is the denominator. I have simplified our survey this year by not including all of the forms that the Career Services Council has developed. I have selected from them so that we can concentrate on following the path, first to make very clear that when we publish numbers, these refer to graduates of the full-time program only. That is number one. And number two, that they refer only to the graduates who are job seeking. And then following it all the way down the line so that they have to provide us the number of graduates who provided base salary data . . . the number of graduates who provided signing bonus data. And then for every one of the functional categories, such as how many graduates went into manufacturing, I’m not asking for the percent. They are asked to provide mean, median, high, and low, and I want to know the actual numbers that those statistics are computed on. Eventually, we will be redesigning the Web site, and every time you see a list of those statistics, you will see exactly how many students are involved there. That is very important. Right now, the instructions say, “At least three students or 1 percent of the graduates.” Well, if you have a high, low, and median for three students, that is very different [from] if you have it for 30 students. And so by having that right alongside, it will immediately say to whoever looks at it, “Oh, that is interesting information, but it is not based on a very big group.”
Selections: That gets to one of my questions about the improvements you’re making to your survey methodology so that it is more accurate and transparent.
Garrett: These changes will make the survey more transparent, because we can’t really . . . the way things were formulated, it wasn’t absolutely clear what we were getting.
Selections: You want to make sure you’re getting apples from all of the schools and not a hodgepodge of apples, oranges, and bananas, correct?
Garrett: Exactly.
Selections: How did U.S. News develop the methodology—including the weighting systems—for its business school rankings?
Garrett: I will leave that up to Bob. But I will say this: Both law and business are areas in which we have been able to incorporate outcome measures, and the outcome measures are in terms of employment. How quickly graduates get jobs. And what salaries they are able to command. Because everybody wants to be able to answer the question, Is it going to be worth it to me financially to make this big investment to get my MBA or to get my law degree? And one of the ways that you can indicate that is by showing what other people are able to achieve by going through that particular program.
Selections: Are there other changes that you’ve made or are in the process of making to ensure transparency and accuracy of your rankings?
Garrett: One of the things that we will be doing, and this comes out of a meeting we had with GMAC about a month ago. And that has to do with reporting admissions data, specifically, providing more information about undergraduate GPA. The point being made that not all applicants and not all applicants that get accepted do or can provide undergraduate GPA information. Some schools do not look at it, particularly in regard to the case of foreign students. One of the things that we are going to be asking on the survey this year is, again, the number of students for which that data is provided. So, I don’t yet know how we will factor that in, in terms of the ranking model. But, certainly, on the Web site we will show the number again. They all have to take the GMAT, but they don’t necessarily have to provide that GPA. When students look at that information in the table, they should bear that in mind. They should be aware of that.
Selections: Speaking of the GMAT scores, I know that U.S. News looks at GMAT scores, but Business Week does not. How do you defend using the GMAT scores as a way to evaluate a school and then rank it?
Garrett: You’ve asked a couple of questions. One, you’ve made a comparison between what we do and what Business Week does. If you look at our ranking model, and look at all of the indicators in it, we ask for the opinion of peer deans and program directors about school quality. We ask recruiters their opinion of program quality. We gather data on the GPA and GMAT scores of entrants, and we look at the acceptance rate. And then we look at the outcome indicators of how soon graduates get jobs and what they get paid. That is a cluster of indicators that all point of the quality of the program, and specifically, the level of the people who go in; the peer opinion as to what is going on, in terms of the program; and then the peer opinion of what comes out of the program, including hard numbers about the jobs they get. It is really looking at before, during, and after, with regard to the program.
I think I will let Bob speak about Business Week, because I haven’t really looked at their ranking. I know that they do include a lot of descriptive work in their particular rankings. And I’m not certain exactly how they make comparisons. It is always nice to have these little capsule reviews, but if you’re then going to have a way of ordering them, I’m not familiar enough with how they do that particular ordering. I do know, based on some of the calls that I field, that people speak to me as if people here at U.S. News sat around and gave their own opinions about programs. And actually the U.S. News rankings involve no opinions of anybody who works for U.S. News. The U.S. News knowledge and input comes in making a decision as to what indicators we look for and how we balance them or weigh them. But nobody sits around here and says, “You know, I’ve heard that such and such a program is on the skids, or such and such a program is up and coming.” That just never happens.
So in a way, the model is a product of what U.S. News does, rather than U.S. News as a group of people taking a look at individual schools and making a judgment.
Selections: The opinions come in the form of asking the peer deans and the program directors and the recruiters. Why not ask the students? Why not factor in their opinions, as Business Week does?
Garrett: The major reason for that is because of cost. Were we to try to do that, it would bust our budget. Besides being a really daunting task, we would have to hire a lot of people. I think in a way—and Bob may not agree with this—when you look at the surveys that are put out, rather than saying, “Well, Business Week, The Wall Street Journal, and U.S. News don’t come up the same results,” people should look at it that these are actually enterprises that ask different questions And they have a different perspective. And while I’m personally not averse to people looking at the results of other approaches, I think it is very, very important for the people who look at the results from our work and from other people’s work to keep that in mind. I, for example, thought the Wall Street Journal ranking was fascinating, because it was really asking recruiters, “What schools are good for you to work with?” And that was very clear in the analysis. I think certain schools would be well advised to take good care of their recruiters.
Selections: That was an interesting survey, because Wall Street Journal took a different tack with how they approached their survey. It was interesting that Stanford, which was ranked number 1 in the latest U.S. News survey, was ranked number 45 by Wall Street Journal.
Garrett: So many of their recruiters said, “We don’t go out to California, because those graduates just expect really high salaries and really plush jobs.” What they are saying is, “They are not really operating in a market where we want to go and try to get the cream of the crop.”
Selections: But then the media reports on how drastically different the results of the major business school rankings are, without really going into why those differences in results exists. How should people be looking at these surveys in comparison? Should the results be compared?
Garrett: I would say that whenever you look at the results of a survey, try to get some information on how the results were constructed, what the model was. That is not always easy. For anyone who sees any statistics whatsoever. . . I know constantly I’m seeing graphs on the front page of really respected newspapers, and I say, “well that is really interesting, but I wonder how big the sample was and exactly how it was chosen, etc.”
Consumers are only going to get more information like that if they ask for it. Because there is constantly going to be the editor who says, “That information doesn’t belong on the front page.” And sometimes it goes on Page 13, and sometimes it gets knocked off Page 13. That is a dynamic process. The consumers have to keep asking for that information.
Selections: How would you convince someone that your rankings are an accurate reflection of a school’s quality?
Garrett: I would say that, first of all, that every single one of the indicators we use is associated with quality. I think people are saying they want to see information at a more specific level, and “your results don’t give that to me.” They are really saying things like, “I want to know how much group work is required in a program, because I consider that to be really important.” That is really not what we are trying to do. We are not answering questions at that level. So, what maybe . . . again, getting back to the consumer and what kind of questions that person wants to ask. Our Web site, besides giving information on our indicators, it gives overall quality, in terms of what peers think about a school. And what these numbers indicate. Every time you look up a school on our Web site, you will see a link to a school’s own Web site. The link is right there, and all you have to do is click on it. So a person can go to a school’s own Web site when looking for the specific information.
Part of the reason that we don’t go into that level of detail is because it then becomes really, really subjective. People say what we are doing is subjective because we are going to deans and recruiters. And they always raise the argument that “you didn’t tell me about X.” If you deconstruct that question, the questioner is actually saying, “I, the questioner, subjectively think that X is the most important piece of information.” If U.S. News were to say that, we would immediately get the criticism of, “Why do you think that is the most important?” That is where the subjectivity comes in. Everyone who is looking for a program should be very clear about his or her own goals and his or her own values, and that is perfectly fine. They should be satisfied. But that particular component of overall quality is not the specific thing that U.S. News should be targeting.
Selections: How much difference really exists between a school that is ranked number 11 and one that is ranked number 9 or number 13?
Garrett: Not a lot. One thing that we did do last year was that wherever there was a tie, we only listed the tied number once. For example, for MBA, there wasn’t a tie until we got to school number 12. But three schools were ranked number 12: NYU, UCLA, and Yale. If you look at the list, there is a difference between a school’s rank and its place level. By which I mean, the rank of a school tells you how many schools are ahead of it. So, the first ranked school has no schools ahead of it. The 26th ranked school has 25 schools ahead of it, but there is no school that is rank number 25. That is because there were three schools tied for rank number 23. I am looking at our top 50 schools. If I start with rank 26, there are nine groupings. Of the schools that are listed there, the 28 schools listed there—of those 28 schools, there are nine separate ranks. Specifically, at rank number 35, you have six schools. If you look over at the next column, they all have scores of 65. If you look at the next three schools that all have scores of 64. . . If one of those three schools with a score of 64 had scores that were slightly different, they would be catapulted from the 41st rank to the 35th. What that is saying is that when people look at the rank number, they should also look at the score, and they should be looking at the full table. Particularly with business schools, there are not wide jumps if you look at the overall scores. Harvard and Stanford were actually tied last year. And this year, Harvard is only one point below Stanford. It is very different if you look at medical schools, where there is a gap between Harvard and Hopkins, and that is because Harvard has such an enormous research budget. It dwarfs everyone else’s. But you don’t have that with regard to business schools. So if business schools are very close in rank, there is not that much difference. So I would say, if any student made a decision between Ann Arbor [University of Michigan] and Dartmouth solely on the basis of that one point, that should not be the basis for making a decision. If the student had gone through all of the various aspects, really looked into the programs, really thought about his or her own goals, and then it was tied. Maybe in that case, yes. But just to say, “Oh, I would never ever go to Dartmouth because it is ranked 11th and Ann Arbor is ranked 10th.” That is really a silly way to make a decision.
Selections: I’m sure you know that the business schools themselves take these rankings very seriously. There are even stories that people will lose jobs because of a slip in the rankings. How do you respond to that?
Garrett: Is it really as bad as some of the anecdotes would make it appear? I would certainly hope it isn’t. If it is, I really would hope that the people who were forcing decisions on that basis can be educated to look a little more closely and clearly as to what is going on. And perhaps to look at school’s performance this year as compared to last year. Not necessarily in terms of where the school is ranked, but in terms of the particular scores it is getting on its indicators. Just as I would say it is silly for a student to base an entire decision solely on one point in the rankings or one point in the score, I think even more if administrations make major decisions only based on that, there is a lot of information in the tables. I have seen schools that have made statements about . . . they point out that the entrance scores for their students have gone up, for example. They should be looking at that kind of thing. I know it is hard for schools, especially if they are just slightly off the page. But any school that provides enough data so that we are able to calculate a ranking has access to that information. We only publish the top 50, but the school itself will be provided its ranking as long as we got enough information so that we could calculate it. And once the school has it, the school is free to publish it. So if a school is just a little bit below the Top 50, they are certainly entitled to publish that information.
Selections: How many schools do you survey?
Garrett: We survey 341 accredited MBA programs. But a large number of those are basically part-time programs. The model really only evaluates full-time programs. Last year, we surveyed 341 schools, but there were only 105 that were full-time programs that gave us all the data—including all of the salary data—that we needed.
We don’t go through and say, “These schools didn’t evaluate other schools, so we are going to knock them out of the rankings.” We don’t do that. What we do in calculating the score for peer assessments is do a trimmed mean. This means that we eliminate the two highest and the two lowest values. And one of the reasons we do that is to take out the effect of people who vote for themselves as a perfect 5.
Selections: In terms of taking the starting salaries of students into account, does your rating system account for variation in salaries of people with different amounts of work experience or for people who accept jobs in various parts of the United States or in other countries, where salaries can be much different?
Garrett: Not yet. I think we would consider adding that if the Career Services Council gave us some guidance on that. What we do publish for any school is the geographic distribution for where the graduates are working. That information doesn’t factor into the ranking model, but if someone wants to look at a specific school that has very high salaries, they can look at the detailed information for a specific school on our Web site and they can see where the geographic concentrations are for people earning those salaries.
Selections: Do you rank international business schools?
Garrett: No. Why not? I know AACSB has now changed its name to highlight the fact that they are now accrediting international schools. So far we haven’t done that, and partly it is a resources issue. I don’t think if we got more resources we would put it toward that area, because we don’t really gather any. . . We have not tried to construct a model for part-time programs. And given the number of students who are enrolled in part-time programs, which really swamps the number who are going into international programs, we might try to work in that area if we were able to get more resources. Right now, the only kind of ranking that we can get comes from the deans and the program directors.
Selections: Where do you publish the results of your rankings?
Garrett: The rankings are published in the hardcover book America’s Best Graduate Schools. It costs $7.95, and it goes on sale at the beginning of April and stays on until the beginning of December. The results are published in the magazine at about the same time the guide comes out, so that would be at the beginning of April. The information is also published on our Web site, which is free right now.
Selections: Have you noticed any increase in magazine sales for your rankings issue?
Garrett: That is not information that we get as a matter of course. Bob may have more specific information on that. Certainly, we know the Web site gets a lot of hits the day that the rankings are released. Bob would have more information on those particular statistics.
Selections: How do you respond to the allegations out there that the rankings are, above all, just a way to sell more magazines?
Garrett: Well, I don’t know how many magazines we sell for the rankings issues. I do know that in the year 2000, the magazine cover that had the highest newsstand sales was the one that said, “Is the Bible True?” It was not a rankings issue. The other thing you have to realize is that U.S. News is not really a newsstand-driven publication. It is a subscription-based publication. But I really can’t say any more than that, because the newsstand sales figures are not circulated to us. I do know that point was made during one meeting last year. The rankings are something that U.S. News is identified with. It differentiates us from other news weeklies. However, I think the perception that the rankings are bringing enormous amounts of money into the U.S. News coffers is probably not that accurate.
Selections: How do you see rankings improving the business school admissions and selections process as well as improving management education overall?
Garrett: I don’t really know about the admissions and selections process, per se. I was speaking with the dean of an MBA program that is in our top 50. It is ranked between 26 and 53. The dean is very much involved with the AACSB and the Career Services Council. He spoke about how really important it is to get clearer salary data out there. And, specifically, the changes that I outlined to you about giving the numbers of students that go with the salary information. He really applauded that effort to make those changes this year and to get that information out in our publication and on our Web site. Because what it does is, if we put it out on our Web site, we can only put out what schools give us, and if it becomes apparent to a prospective student doing comparative work, if they see that school A gave us lots of detailed information and school B didn’t, then that person is apt to act on that information, either by going back to that school that didn’t give much information and asking why not, or saying, “Well, the school that did give more information is maybe to be more trusted.” So, what it does is provide a forum that reinforces the standards that the deans themselves, acting in the best interest of the profession, want to foster. Any dean always has to balance the effort to make his school come first with the effort to increase the standards that everyone in the group holds. At one point, I was trying to track down a mythical directory of MBA recruiters, because we go to each school and ask for data on names and contact information for their recruiters. I was told that everyone wants to see this kind of directory but no school wants to give away its information before another school does. So the kind of things that we do when we start publishing information on our Web site. . . It sort of ups the ante for all schools.
Selections: Are you concerned about the proliferation of rankings by other publications? Do you consider these other rankings as competition, and do you think this proliferation helps or hurts management education?
Garrett: My personal opinion is that what we have not . . . I want to emphasize that I’m really not speaking for U.S. News . . . but my personal opinion is that the three kinds of rankings and surveys that are now being published are really so different that—and I may be the kind of person, because I am involved in the construction of these rankings, I have an interest in how things are put together and, therefore, what the results are. I would suspect that there are some people who would not like to see multiple rankings. Bob may have a different answer. And the U.S. News marketing people might not like to see other rankings that they then have to address. I don’t think there is anybody out there who is trying to do what we do, in the manner that we do it, and doing it better. I do think that other people are presenting information from a different perspective. And it is really important that people understand that.
Selections: To whom do you find yourself accountable in your work of publishing business school rankings?
Garrett: Obviously, I am accountable to my editor, because the editor makes decisions about what gets included in the table. I am accountable to U.S. News to make certain that whatever we publish is accurate, that when data comes to us, we have all kinds of ways of comparing it against data that was given to us the previous year to catch errors. There is a minimum of two separate processes by which we send information back to the schools to be verified. I am constantly trying to make what we do more and more transparent. That is not always easy, because we really have a brutal publication schedule. There is not a lot of time between the time the data comes in and the time that it has to be put out for publication. And then during the year, I don’t have access to our Web people as much as I would like. I would like to publish more information on the Web. But every year, we keep getting better at it. There is an information feedback loop with the schools. As deans come in—and particularly as they come in and are willing to work their requests through the appropriate bodies in the discipline—the AACSB or the Career Services Council specifically, or the GMAC people. U.S. News tries very hard and I try very hard to look at those standards and implement them as carefully as we can, so that we’re lifting the best standards from the profession. Standards of accountability.
Selections: What do you see for the future of the rankings?
Garrett: Well, I don’t think they are going to go away. It’s amazing how much correspondence we get via e-mail and telephone. People who say, “Why don’t you do this, too?” And Bob may know more about this, but I think if we had more resources, we would probably do more. There certainly are requests for us to do more. And again, following this feedback loop with the organizations, as organizations adopt more standards and as more schools buy into those standards, the GMAC people are adopting standards now that say, “if you want to be one of the schools listed on our Web site, you have to submit to an auditing process.” As that is put in place, that is going to make things more and more transparent. As that is enforced, it is a two-way street. Our publishing things causes the data to be clearer. And GMAC requiring audits is also going to cause the data to be clearer. I think there is a dynamic there to get clearer and clearer information and more reliable information out there. I think people have just gotten used to having this information available.
© Selections: Fall 2001