Home | Letters to the Editor | www.gmac.com
To print this story, please select “Print” from the File menu of your browser.
Selections Interview with Joy Sever, Senior Vice President and Director of the Reputation Practice, Harris Interactive (on behalf of Wall Street Journal)
Selections: What were the goals behind the creation of the Wall Street Journal/Harris Interactive business school survey?
Sever: We talked about it in terms of a five-part mission. Most important, it was a study to take a look at business schools from the perspective of recruiters and only from recruiters. We talk about this five-part mission in great detail in the e-book. The e-book only costs $15 and it provides a very, very detailed explanation of our methodology.
Part of our five-part mission was to include as many schools as possible—provided they met the criterion of having at least 50 full-time students. And then to rank them [the schools] on as many criteria as possible, based on the experience of recruiters, and to include as many recruiters as possible. We also wanted to provide information that wasn’t just about ranking schools but actually provide information that schools could use to learn something. We didn’t want to produce a ranking for ranking’s sake but actually get into what is important to recruiters. The fifth part was to be as open as we could on the methodology, which is why there is such a detailed methodology in the e-book.
Selections: I did notice that a lot of information was published on the CareerJournal.com Web site about the methodology, including a response to criticisms.
Sever: When we did our two-year preparation for this, we learned that the schools’ biggest frustration was that they would get ranked and they couldn’t figure out why they were ranked the way they were. They knew it was important, but they felt very powerless to really do anything about it in a real, legitimate way. How could they make themselves a better school if they don’t really know how they got there?
Selections: There are already a number of MBA rankings out there and there are a lot of criticisms of the rankings that do exist now, so why did the Wall Street Journal even want to get into the business of ranking the business schools?
Sever: I’m not sure if you’ll get the same answer, but I’ve heard them answer this question. Being a business publication and based on their readership, they felt this was information that was going to be important to the people who read their paper.
Selections: What was your role with the survey?
Sever: We [Harris Interactive] designed and conducted the survey. We are very much partners in that we copresented at many conferences, and we shared in meetings as we were developing the survey. The Wall Street Journal and Harris Interactive pretty much did things side by side.
Selections: Whom do you consider to be the main stakeholders in this endeavor?
Sever: Recruiters. They were the only stakeholder.
Selections: I know you go into this quite a bit on the CareerJournal.com Web site, as to the many differences between the Wall Street Journal/Harris Interactive ranking and the Business Week, U.S. News, and Financial Times rankings, but I wanted to ask you what you consider the most significant differences?
Sever: There are three main differences, and I did write about this for CareerJournal. I wrote about the three questions each of us approach in our surveys in a different way. One is the universe of schools eligible to be rated. We have a broader universe of schools, and what we heard schools talk about in terms of their frustrations was that the same schools [show up on the other surveys] almost year after year. I think their methodologies lend themselves to that, because it’s sort of continuing to go back to schools that had been listed in the previous rankings. When we set out to do this, we cast the net as wide as possible. We said we are going to consider all schools, and then those schools that don’t qualify—because they don’t have enough students—we will remove from the list, as opposed to starting in a more restricted way. So that would be number one.
The second would be the respondents rating the schools. That is very different across all four surveys. We have just recruiters, and you will know from talking to the others that they use other respondents, as well.
The third is the rating criteria. We talked to recruiters to identify what guided them when they go to the schools and decide which type of students to recruit. That was our main focus, and again, that is not what you will find in the other surveys. The other surveys have different components and different percentages attached to those components.
We surveyed more than 1,600 recruiters, and Business Week and U.S. News each survey under 300 for that same component. And Financial Times doesn’t have recruiters surveyed at all.
Selections: Why did Harris Interactive decide to participate in this, and how does Harris Interactive benefit from having partnered with the Wall Street Journal to conduct this extensive survey of recruiters?
Sever: They approached us and presented the idea to us. In deciding whether or not this was something that we wanted to do, we took a look at what was already being done, and I think we were surprised after reading and learning about the surveys that did exist. We knew about the impact that these surveys have, but I had never looked in depth at the methodology of the others. I think the approach that the Journal wanted to take, which was the recruiter perspective, we saw that as a challenge and something that would be a very visible research project. We thought it was something that would be a challenge, and it’s probably one of the most interesting projects I’ve ever worked on.
Selections: That’s interesting that you talk about wanting the challenge, because I know in one of his stories, Ron Alsop [of the Wall Street Journal] talked about how it turned out to be one of the most challenging projects he has worked on.
Sever: I would put it as one of the most challenging and most interesting.
Selections: What were some of the most significant challenges?
Sever: We wanted to do it very rigorously. We knew that this would really matter. And we knew that we had both of our reputations—the Journal’s and ours—to protect. We knew that a lot of attention would be brought to this. We knew that business schools greatly value their reputations. When we originally started, we had a project schedule of six months. That ended up being extended to two years. The challenge was, each time we looked at a particular way of doing something, we learned that in order to do it right, we would have to extend the schedule or try something else. Just to give you an example: Schools all operate in different ways, so finding the right contact people, making sure that every school was included, making sure that if a school came up with a particular way of doing things we shared that, so that all schools were given the same information and were working with the same material. Not all schools keep track of their recruiters in the same way. So, some schools gave us many names, and others didn’t. So, there were variations there. Recruiters move around a lot, and there isn’t this database of [recruiter] information out there, so we had to construct our own sample of recruiters. We were doing this internationally, so that when we would think we were ready to close, we would find out that there were holidays in Spain and would have to keep the survey open. We learned that how recruiters make their decisions is somewhat complex. When we thought that maybe we would be rating the schools on 10 dimensions, it turned out to be more than 20.
Those were some of the main challenges, off the top of my head.
We wanted to meet with as many people as we could. We wanted to hear what their concerns were, and we would learn about other biases that we might not have been aware of. So, we would meet with someone and they would say, “Well, are you paying attention to this?” And we would say, “Oh, okay.” So, we would build that into the survey. The analysis took time. All of those challenges were driven by the fact that we knew this would matter.
Selections: You talked about wanting to create a very rigorous methodology for your ranking. In what ways was the Wall Street Journal/Harris Interactive survey methodology more rigorous than other MBA ranking models?
Sever: I think it goes back to those three differences between our ranking methodology and the others. Broader universe. More recruiters. Trying to have as many criteria as possible so that we weren’t ranking schools based on one or two measurements. It was really across 27 different elements.
Selections: Why base your survey solely on recruiter input? Why not factor in input from students or business school deans or administrators?
Sever: First, we felt the recruiters were a significant enough stakeholder group to devote an entire survey to them. Second, that was challenging enough. It took us two years to do that. Three, those other groups that you mentioned, I don’t know that we can figure out a way to include their input without introducing bias. So, those two groups are ones that are most frequently felt to introduce responses that definitely have a kind of self-serving aspect to them.
Selections: I did read the article published on the CareerJournal.com site by John Lynch, Jr., a faculty member at the Fuqua School at Duke, about the bias he felt was present in your survey. I also read your response to his claims. Given his points, is there anything you would have done differently with the survey if you were to start over right now?
Sever: We welcomed all of the feedback we got, and we discussed every decision that we made very carefully. There were many things that we couldn’t do in the first year because of the nature of it being the first year, because of the nature of what we were trying to do, because of all of the challenges that I mentioned. We had to make decisions that were the best possible decisions for the survey that we were doing. There were definitely people who didn’t like the ranking. But we heard so many responses and comments from people who said and reinforced that, yes, we got it right. We got enough feedback and we knew how carefully we worked on this—that told us we did the right thing. In an ideal world or if the world of recruiting were structured differently, there might be things that we would have liked to have done differently. But we would never have published results if we didn’t think that we could stand behind them.
Selections: I saw that you did survey more than 5,200 students but didn’t include their responses in the final ranking. How did you use those responses?
Sever: We used them in a couple of different ways. And to clarify on why we didn’t use them, it was because different schools chose to handle their requests in different ways. We gave them an invitation to share with their students. It was an invitation to participate in an online survey. So, some schools had a few hundred students participate, and others only had two. We never intended those responses to be part of the ratings and rankings. That information, we provided in a summary report that is available to the schools. And there is some of the most interesting information in there on recruiting tactics. So, we may do something with it, but we haven’t done anything with it yet. But it’s fantastic information.
Selections: So, that information is not provided in the e-book?
Sever: No.
Selections: How should prospective MBA students use the information published in the Wall Street Journal/Harris Interactive ranking?
Sever: They should be looking at a number of different pieces of information when making a decision, and I am sure they do. It’s a big decision. In addition to the surveys, there is so much information out there that can help them make a decision. I think all others things being equal, I think perhaps they may come down to two or three schools that may be of interest to them, and they are trying to make a decision between them. I think what our survey provides is information on that bond between the recruiters and the schools. It would provide an indication of the strength of the relationship, and I think a school that has a very good relationship with its recruiters and is rated highly by recruiters is one that is probably going to work well for the students when it comes time to getting a job.
Selections: How should the schools, and then also the recruiters, use this information?
Sever: Well, the schools can learn so much from this. We learned that what recruiters really value are strong interpersonal communication skills. They really are turned off by arrogance. They really want smart students. But I think there are a lot of students in MBA programs who are smart. I think there are varying degrees, and there are different aspects that recruiters look for, but the qualities that were most important really had to do with the personality and culture and fit and really fitting well. I think schools can learn that, so that those that may not have done as well as they thought they should have or that they have done on other surveys—and often that was the case—they can learn from that. They can also learn from the details. Different recruiters from different industries look for different things when they are going to different schools. That also could be beneficial.
Selections: I know there were differences between the Wall Street Journal/Harris Interactive findings and those of other surveys. One example is Stanford, which was ranked number 45 in your survey and number 1 in that last U.S. News survey. Why is there so much disparity in the ranking results?
Sever: Stanford is an excellent example and one that helps to prove our point that we’ve got really good information here. Stanford didn’t do as well as it does on other surveys partially because it was a very unique year for Stanford and other schools on the West Coast because of the demand. I think some schools had a harder time meeting that demand and there were some recruiters who couldn’t be at the top of the list in terms of being heard or met with. They were probably upset. That could have been behind it. But in the same survey, when we asked recruiters where they would go if they were going back to do a MBA, Stanford was at the top of the list [ranked second behind Harvard]. I think that shows they still appreciate it as a place they would want to go, but in terms of fulfilling their recruiting needs for the company they work for, it didn’t satisfy them. It fell short. But do they still see it as a good school? Of course. I think that helps to prove our point, as opposed to people thinking this doesn’t make sense. It makes perfect sense.
Selections: What impact do you see the Wall Street Journal ranking in particular, and business school rankings in general, having on business schools and management education?
Sever: Well, we’re not in this to impact . . . we didn’t touch on the educational aspect itself. So, in terms of the quality of the school and its professors and its academic contributions, this survey is not meant to look at that. This is a survey purely about recruiters’ perceptions. But where I think it will have an impact and where we would hope it would have an impact is that the schools will learn what their recruiters value, and they will learn that perhaps the recruiters that recruit there don’t recognize their strengths. They can say, “Well, recruiters actually value one of our particular qualities, but they rated us poorly on that. We need to do a better job of communicating that.” Or they may learn that recruiters value something and they have neglected it. So, it can help them address something that they have neglected, because it actually is important to their recruiters. I think schools can actually do something about how they interface with their customers and actually improve. And that is what this should be about. This shouldn’t be about some arbitrary exercise that the schools have to do just so that they can do well on a survey. That is really the wrong thing for us to be doing. So, if they learn from this and they do something, it will actually be done with all of the right intentions. If that happens, I think the recruiters will benefit, the students will benefit, and the schools will benefit. I don’t think it is going to happen overnight, but if anyone were to change their behavior, we hope that is the direction it goes in.
Selections: To whom do you consider yourself accountable in the work of producing the rankings?
Sever: Probably multiple audiences. I sort of put the schools first. I think we owe it to the schools to make sure we represent them in a correct way and don’t misrepresent them. But I think we are accountable to the students, the recruiters, the readers. It’s pretty broad.
Selections: The results of this ranking were reported in the Wall Street Journal, published on the Web site, and published in this very lengthy e-book, correct?
Sever: Yes.
Selections: Do you know how many e-books have been sold thus far?
Sever: No. The Wall Street Journal may know that. I don’t know.
Selections: Harris Interactive will do customized reports based on this information?
Sever: Yes.
Selections: How much do those reports cost?
Sever: There is a range in prices. It costs $2,000 for a summary report. These contain information that cover the whole recruiting world in general. It’s not specific to any one school. And then there is the $6,000 package, which includes a profile report on that particular school and the summary report. It’s a package of two reports, and it’s everything we have about that school. So if a school were rated by 40 recruiters, other than the identity of the recruiters, the school pretty much finds out what industries they come from and what is important to them, what they said about the schools. We’ve sold reports, and in every case when we take a look at these findings, I really have come to learn something about the schools that really comes out of this. It is very rich. They also get any student responses that pertain to their school. And it’s questions like, What can the school do to improve its reputation? Then there is a package of reports called the “comparison set,” where a school is able to pick a minimum of five schools as a comparison set and we aggregate the findings and present that aggregate as a comparison to the schools. So, they can see what recruiters who are rating the comparison set had to say. This way, they can see where there are gaps in perceptions, and that can tell them where maybe there is some gap that they need to close. That is $10,000.
Selections: How many of those reports have you sold?
Sever: I don’t want to give a specific number, but it’s enough that we know we are providing something useful. And we didn’t attempt to try to sell a lot of them. This is a very sensitive area. Schools are not used to paying this amount of money for this information. We priced these much less than we would for an equivalent report for a corporation. We know what we need to do in terms of the amount of work. We also prepared the reports at the requests of the schools. We didn’t go into this intending to sell reports. But there is so much information, and when we would meet with the schools, they would ask to get their data. So, we met with schools and worked our way to a price we felt schools might be comfortable with. We’ve had schools buy from the top [of the rankings] all the way to bottom. So, it’s not only schools at the top and not only schools at the bottom. It ranges from schools in the top 10 to schools ranked close to 50.
Selections: Do you expect to conduct this ranking again?
Sever: Yes, we are conducting it again. We haven’t released our intended released date. We have contacted all of the schools and sent them a business school help kit. We’ve heard that schools get asked to participate in around 20 different surveys. We have given them a very detailed schedule of what we will need from them and when we will be contacting them. The interviewing will begin later this year and go into February. We will analyze the data, and the results will be published sometime next year.
Selections: When you first approached the schools with the news that the Wall Street Journal was doing a different kind of ranking, what kind of reaction did you get?
Sever: Mixed. It was like, “Oh, no, another ranking.” Positive because we were addressing something that had been neglected. And very positive when they saw our approach to this. And schools came away—at least they told us—very impressed with how detailed we were and how careful we were in conducting this. We wouldn’t have done this if we hadn’t received their support. If we had received negative reactions, we wouldn’t have done it.
Selections: Do you consider the other rankings out there to be competition?
Sever: I think other people put them in that category, but they are very, very different surveys.
Selections: What do you think the future holds for business school rankings in general and the Wall Street Journal/Harris Interactive ranking in particular?
Sever: We have made the commitment to do it for a second year. We want to enhance areas that we can enhance. So, as certain possibilities are now made real, we can now do things that we couldn’t do in the first year, because either we’re more known or the technology allows us to do it. We want to make it better and provide more information. And we hope schools will learn from this. We want to provide something that is meaningful.
Selections: Do you anticipate trying to find a way to include input from the students or the schools themselves?
Sever: We’ve talked about extending this, but I don’t know if that is the first place we would turn in enhancing this. The amount of work and the amount of time that we devote to this . . . It took us two years to do this one. It will be less than two years for the next one. But I think we want to make sure we’ve got the routine and procedures and everything figured out here before we were to introduce another one. I think we’ll stick with this one for a while. But we have talked about ways to enhance it or extend it, add other elements.
Whatever may not be perfect about our survey, at least we are very open, so that the schools can evaluate what we’ve done. And we are very “out there.”
© Selections: Fall 2001