Broadening the Impact of Management Research
Article
“Assessing scholarly impact, what it is, how we should measure it, and how our metrics shape society are the existential questions facing business schools today,” says Usha Haley, the W. Frank Barton Distinguished Chair in International Business and professor of management at Wichita State University’s Barton School of Business in Kansas. “The debate about rigor versus relevance has morphed into one on broader impacts.”
To address some of those pressing questions, Haley recently wrote Impact and the Management Researcher, which is based on results from a survey the Academy of Management (AOM) recently sent to its 20,000 members worldwide. The survey asked respondents what their views were on the current state of faculty research, the way it is assessed for relevance, and the way it is used in tenure and promotion decisions. Haley recently spoke with AACSB Insights to discuss what the survey revealed about the state of business school research today—and how business scholarship could achieve even greater impact in the future.
In the debate between rigor and relevance, a key question revolves around who is reading and using the research produced by management faculty. According to the AOM survey, what are the primary audiences for business school research?
The top five are management academics, corporate decision makers, governmental policymakers, social science academics, and students.
Those results reflect the fact that the historical focus for academic research has always been on internal audiences and that institutions have always given the highest support to faculty who publish in A-level journals. In fact, respondents say the top five activities supported by their institutions are publishing in top-tier journals, receiving scholarly citations for their research, obtaining research grants, publishing books, and publishing in practitioner journals.
Where do faculty think their research generally has the greatest impact—inside or outside the business school?
Inside. They rank their research as having the most influence on management theorizing, teaching, future research practice, management policy and practice in large enterprises, and students’ career decisions. Most appear to believe that their research exercises little influence on broader social issues.
Even so, large numbers of respondents recognize the value of practical research. About 54 percent consider impact on practice as strongly or intensely important, and about 46 percent say the same about impact on government policy. Additionally, about 59 percent view interdisciplinary research as more impactful than research that draws on one discipline, though it can be more difficult to get published.
Because impact is such an important component of tenure and promotion decisions, how do business schools measure it?
The top five indicators of scholarly impact tend to be the same as the activities supported by the school: articles in top-tier journals, citations, scholarly books, research grants, and articles in practitioner publications. Faculty say that some of the other indicators of research impact are media coverage, keynote speech opportunities, practitioner books, regulatory influence, and awards.
But all of these metrics have their flaws, none are objective, and no one metric can stand by itself. For instance, citations are problematic because of self-citations, forced citations, and citations made by researchers who haven’t even read the articles. Additionally, citation counts vary by source—for instance, those in Google Scholar are different from those in Clarivate. I can go down the list and identify issues with each metric.
The top five indicators of scholarly impact are articles in top-tier journals, citations, scholarly books, research grants, and articles in practitioner publications. But all of these metrics have their flaws, none are objective, and no one metric can stand by itself.
Another concern is that several elements of the present system of evaluation reinforce each other: For example, publications in elite A* journals bolster business school rankings. A higher ranking helps a school attract more students. And when more students want to attend an institution, professors at those schools command higher salaries.
Yet, culturally and psychologically, we do need some way to assess performance.
It’s not surprising to learn that schools rely heavily on journal publication and article citations to judge impact. But did any other results of the survey surprise you?
Despite the fact that journal rankings and lists are ubiquitous metrics, about 60 percent of the respondents say that these measures probably do not, definitely do not, or might not reflect scholarly impact. I found this ambivalence striking, as journal rankings and the journal impact factor serve as the gold standards for evaluating research performance at business schools.
It was also eye-opening to see that only about 38 percent of respondents say that their institutions support their pursuit of impact all the time or most of the time. This is a sobering statistic, because institutional rhetoric emphasizes how research serves society, and institutional support has great repercussions for career and life satisfaction.
What must change in the current system for faculty to be able to produce more relevant research—and have that research supported by their administrations?
In business schools, most scholars conduct rigorous research that speaks to just a few people. We should start incentivizing the pursuit of broader impact. For example, business school administrators could use promotions or perquisites to reward researchers who identify how their projects have generated community outreach or benefited certain audiences.
In the U.S., researchers who have applied for National Science Foundation (NSF) grants know that they have to identify the broader impacts of their research or show that it has the potential to benefit society and contribute to the achievement of desired societal outcomes. The impact statement is so important that a poor one can sink a proposal, even if it displays good science. Good science is simply not enough anymore. Neither is just talking about impact without doing anything to actually achieve it.
Ironically, talking about impact, especially in the major journals, is incentivized; having actual impact externally is at best ignored, almost never rewarded, and sometimes punished.
In the major academic journals for the field of management, we have seen an exponential rise in discussions and mentions of impact. Simultaneously, we have seen an exponential decrease in the influence our research has on regulation and on our ability to catch the attention of external constituencies. Ironically, talking about impact, especially in the major journals, is incentivized; having actual impact externally is at best ignored, almost never rewarded, and sometimes punished.
The incentive structure has to change. We need to ask: Who does this system of research benefit, and how do we throw a wider net?
Once administrators change the incentive system, what else can they do to ensure that academic research has a broader impact?
We posed that question to some prominent and experienced scholars in the AOM, and their answers coalesced around four points.
First, they argued for expanding measures of scholarly impact to correspond with the missions of universities and academic associations. They also advocated for increased awareness of the strengths and weaknesses of alternate metrics.
Second, several brought up the need to achieve a broader participation in ideas. They suggested that researchers partner with outside technical specialists in academic fields, as well as people working in businesses and government. They cautioned against excessive balkanization, noting that we have become so specialized that we resemble angels dancing on a pinhead.
Third, many of these experienced scholars proposed investing resources in disseminating the knowledge we create to make it less esoteric and more accessible. When our vocabularies change to include external stakeholders, when we acknowledge their aspirations and needs, we extend the reach of our ideas.
Finally, the AOM experts recommended that business schools pursue alliances with other academic associations to create interdisciplinary collaborations and achieve wider impact. This is a concern that echoes across many other disciplines, including political science and psychology. Business schools are not alone, and we cannot go it alone. To solve the issue of social and external impact, we need to rise above the level of individual academic disciplines.
When it comes to creating more impact, what is the role of government? For instance, in a number of countries, federal agencies require schools to show a return on investment before funding research. Have these requirements affected what research gets conducted?
Well, the ROIs for governments differ from those for corporations. Corporate goals generally concentrate on monetary returns to stockholders, as well as prestige, status, and high salaries for managers. Governments, on the other hand, have broader social goals that may include increased stability, enhanced positions for underprivileged groups, and national security. Many observers have accused our universities, and especially our business schools, of behaving more like corporations—focusing on ROI that benefits us and ignoring the ecosystems in which we operate.
Many observers have accused our universities, and especially our business schools, of focusing on ROI that benefits us and ignoring the ecosystems in which we operate.
The U.K.’s Research Excellence Framework (REF) stands out as one of the most influential governmental efforts to provide accountability for public investment in research and to establish reputational yardsticks. In REF 2021, 25 percent of a university’s evaluation depends on the impact, socioeconomic effects, and benefits of its research beyond academia. For the most part, such a demand for external accountability does not exist for universities in other parts of the world.
You’ve noted that elite institutions have a higher success rate for governmental grant applications. Can you explain why this is problematic? How could changes in measures of scholarly impact lead to more diverse research?
Some research indicates that scientific output approximates a curvilinear function indicating diminishing marginal returns: As funding per researcher increases beyond a certain point, productivity decreases.
The research I cited in my book focused on National Institutes of Health (NIH) grants. Prestigious institutions had on average 65 percent higher success rates for grant applications and were granted awards that were 50 percent larger; however, less-prestigious institutions produced 65 percent more publications and had a 35 percent higher citation-impact per dollar of funding.
These findings suggest that implicit biases and social prestige mechanisms may influence where federal dollars go, which ultimately affects net returns to taxpayers’ investments. These results also provide some support for changing funding policy to ensure more equitable, diverse, and productive measures of scholarly impact and prestige.
Even so, I remain a big fan of government grants and funding for research. Government agencies in the U.S. such as NSF and DARPA have highly successful models that fund industry-altering research on relatively modest budgets. (Full disclosure, I currently hold an NSF grant as sole principal investigator and I review for the NSF.) These federal agencies have taken calculated risks on basic research and on as-yet-undeveloped applications.
For example, the shale gas revolution can be credited to government funding of basic research in new technology when the potential global market did not even exist as a gleam in anyone’s eyes. This technology originated from massive hydraulic fracturing that the Department of Energy first demonstrated in 1977. Oil entrepreneur George Mitchell learned of the technology and its potential applications from the Eastern Gas Shales Project, a partnership between the Department of Energy’s Morgantown Energy Research Center and dozens of companies and universities. From 1981, Mitchell’s geologists drew heavily on that research to guide their explorations.
What do you expect to happen next, in both business and business education, as the industry moves toward more relevant research?
Niels Bohr, Nobel laureate in physics, reportedly stated, “Prediction is very difficult, especially if it’s about the future!” My hope is that business schools will start rewarding researchers who address the wicked problems facing society, rather than continuing to incentivize methodological slivers and incremental knowledge development. If we do that, we will have more impact in a truer sense on broader society. We are so well situated to tackle some of these problems—we just need the push to do so.
Sharon Shinn is an editor with AACSB Insights.
Article
Article
Article