Dear Colleagues and Fellow SIMians:
I would like to recommend to everyone, a series of articles that recently appeared in Organizational Research Methods. Art Bedeian and co-authors did a study examining the level of “scientific achievement” attained by members of the editorial boards of major management journals. Using a very objective approach, they found the recent levels of "scientific achievement" of members of editorial boards significantly lacking. This generated a series of commentaries by some high profile members of the Academy.
From my perspective, I believe an interesting question is why is this the case? What is driving this phenomenon? I certainly have my own theory. I also think this is very much related to the editorial process for the AOM conference.
With Warm Regards,
Wayne Grossman
Associate Professor
Department of Business 3N219
College of Staten Island/CUNY
2800 Victory Blvd.
Staten Island, NY 10314
Phone: 718-982-2973
Fax: 718-982-2965
-----Original Message-----
From: Linda Trevino
LTrevino@psu.edu
Sent 3/23/2009 3:54:12 PM
To:
SIM@AOMLISTS.PACE.EDU
Subject: Re: [SIM] Is the Academy review process broken? What to do?
Linda,
Just a couple of thoughts.
When you receive a submission that does not meet submission guidelines, you should quickly notify the Program Chair and return the paper unreviewed. Such a paper should not be entered in the review process.
Also, if your assessment is that you are not competent to review a paper (without having to read massive amounts) you should again notify the Program Chair and return the paper.
Linda K. Trevino
Distinguished Professor of Organizational Behavior and Ethics
Smeal College of Business
402 Business Building
Smeal College of Business
The Pennsylvania State University
University Park, PA 16802
Phone: 814-865-2194 Fax: 814-863-7261
Email:
ltrevino@psu.edu
From: Social Issues in Management Listserv [
SIM@AOMLISTS.PACE.EDU] On Behalf Of Linda C. Rodriguez [
blondie@pobox.com]
Sent: Monday, March 23, 2009 9:51 AM
To:
SIM@AOMLISTS.PACE.EDU
Subject: Re: [SIM] Is the Academy review process broken? What to do?
I just quickly refreshed over the first three replies below. I have a question and its very nave. Im a first year professor and I know that my comments for reviews are not ever good enough, but then Im faced with two issues to overcome: 1) being a neophyte in the business and sometimes, I can barely understand some papers because of my lack of experience for which Im supposedly qualified (review mismatch) and 2) the theory is way over my head. So in order for me to make sense of a 40 page paper I have to read a lot of the lit review articles to get up to speed. Then, because its way past my level of expertise, how does one comment on improvements? On some of the reviews that I did early on, I got paper that did not meet the style guidelines. Those I kicked to the curb as automatically unreadable. 45 pages of 10 point font is inappropriate. But beyond basic formatting and grammar and style, some of the papers that Ive read for the Academy leave me cold because they are so long that I cant invest. I prefer to submit a five to ten page abstract or prospectus and then take comments for improvement of the final paper that will be submitted to a journal. In fact, the process is onerous to me that I prefer not to participate in the AoM for a while longer. In the tenure review process, getting a paper accepted at AoM is the same as getting a paper accepted at other smaller conferences as long as they are peer reviewed we arent too picky! To submit elsewhere does not mean that a paper lacks quality or contribution, and I think too many people feel that AoM is the only publication in town that has quality and contribution.
No need to go through the headache of an AoM review unless I just want it for free and for fun to prepare for publication elsewhere. The process is just onerous either as an author or reviewer. Im in favor of 20 page papers that get to the point, but there is always one nay-sayer that asks why a particular topic wasnt asked or covered when, indeed, in the longer paper it was. Then that same visionless reviewer might not recommend a paper based on the fact that things arent coverednever asking if it could have been covered in the space allotted. Sigh, win-lose, all the way around.
I have inflation-itis-
My $.10
Linda
--
From: Social Issues in Management Listserv [mailto:
SIM@AOMLISTS.PACE.EDU]On Behalf Of Lee, Min Dong
Sent: Monday, March 23, 2009 9:14 AM
To:
SIM@AOMLISTS.PACE.EDU
Subject: Re: [SIM] Is the Academy review process broken? What to do?
I am not an economist, but I do think Dons pricing model makes a sense if we can come up with pricing level that would not become a barrier for unfunded Ph.D. students (or maybe create an alternative system for students).
Another suggestion I have is to reduce the length of papers to present only the essence of the paper. I was speaking with a friend in Physics recently, and he was astonished that our papers are often 40-50 pages long. His words: how can you possibly read all of them and reflect on them? Meaning if we spend too much time reading, we dont have enough time/energy/motivation to think. Or, if we are pressed against time, then we simply do not read the papers very carefully and come up with just perfunctory comments. As all of you know, good review takes time and good review on 40 pages takes very long time.
There is another advantage of limiting paper length. I believe writing a good short paper that just contains the essence would be more difficult than writing a longer paper. And reviewing a very short paper for quality is probably easier, because authors cannot hide behind long quotes, lit. reviews and rhetoric etc. I tested this hypothesis with my class (student essays), and it seems to be trueJ So, the burden of proof falls on authors, not reviewers.
I also think that since we will hear the presentations, we really do not need the full paper. Papers can be submitted to journals, not AOM conference.
So, my suggestion is to limit the length of papers to something like 2000 words and allow reviewers more time to think.
Thats my 2 cents.
Paul
Min-Dong Paul Lee, Assistant Professor
Department of Management Organization
College of Business
University of South Florida
4202 E. Flowler Ave., BSN 3403
Tampa, FL 33620
(813) 974-1721
From: Social Issues in Management Listserv [mailto:
SIM@AOMLISTS.PACE.EDU]On Behalf Of Donald S Siegel
Sent: Sunday, March 22, 2009 10:49 AM
To:
SIM@AOMLISTS.PACE.EDU
Subject: Re: [SIM] Is the Academy review process broken? What to do?
HI Mike:
Its always a pleasure to read your lucid and insightful comments on this list-serve. I understand your frustration. At the risk of sounding like a stereotypical economist, I view this as a pricing problem. With the proliferation of journals, increasing submissions to the AOM meetings, and greater pressure to publish (tenure committees are often focused more on quantity than quality), it is difficult to find reviewers. One solution is to raise the cost of submission, which is now zero, while simultaneously compensating reviewers (i.e., the revenue generated from submissions could be used to pay reviewers, with additional funds coming from the division). In other fields, especially in economics and finance, journals pay reviewers for on-time reviews or offer to donate a certain sum of money to charity. The latter option might be especially appealing to SIM members.
Best regards,
Don
Dr. Donald S. Siegel
Dean and Professor
School of Business
University at Albany, SUNY
1400 Washington Avenue
Albany, NY 12222
Tel: (518) 442-4910
DSiegel@uamail.albany.edu
http://www.albany.edu/business/
http://www.albany.edu/business/news_and_events/DonSiegel-CV.pdf
http://econpapers.repec.org/RAS/psi32.htm
http://ssrn.com/author=33607
From: Social Issues in Management Listserv [mailto:
SIM@AOMLISTS.PACE.EDU]On Behalf Of Barnett, Michael
Sent: Sunday, March 22, 2009 10:29 AM
To:
SIM@AOMLISTS.PACE.EDU
Subject: [SIM] Is the Academy review process broken? What to do?
Dear fellow SIMians:
I think the review process for the Academy of Management conference might be broken, within SIM and perhaps beyond. Id like to see if others share my concern, and if its not just me, Id like to push the division, or the Academy in general, to do something to fix it.
This morning, I received notification that my paper was accepted at SIM. Thats good. Then I read my three reviews not good. The first reviewer offered a reasonable analysis and a couple good suggestions. It wasnt a journal quality review, but it was a solid review basically, of the caliber we should probably expect from a conference review, given time constraints, etc.
Then theres the other two. Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), heres the entirety of the sage feedback I received from the other two reviewers:
R#2:
Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.
R#3:
I have an issue with your title. I think it is inappproriate for an academic conference. I would leave off Taking a Chit.
Include some up-to-date references (at least 2007 and above).
So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have some post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).
Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof. Seriously its not that Im a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but thats not the point), but its that these reviews dont even address what I wrote.
Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, Im concerned about introducing this level of noise into the review process. How can the program chair make selections when the majority of data are at best useless, and at worst, wrong? Does this selection process lead to a program that is most useful to Academy members?
This is not a slam at Barry Mitnick, the SIM division program chair. If anything, its a testament to his hard work and dedication, despite the troubled process. I cant imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time. Rather, its a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.
So I think we first need to decide if the current model is broken in a way that cant be fixed. I think it is. The problem is that we cant obtain an adequate pool of competent and engaged reviewers. Therefore, we rely on the incompetent and disengaged, and to an increasing degree. This is a problem for our major journals (see the recent pleas for reviewers), and so its going to be even worse for SIM at Academy (cant even imagine the challenge at conferences with lower profiles). I dont think theres a way to get up to adequate quality while retaining or even increasing quantity. We could better screen potential reviewers (must have some basic qualifications dont think we do this now), and we could disallow bad reviewers from reviewing again (dont think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.
If we accept that the current system is broken and cant be fixed, then that frees us up to search for a new system. I hope the executive committee will do just this. What are some options? Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat 40% get spots. I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews. If the result is the same, while the efficiency is much greater, we should do this.
Another option is to decrease the number of reviews required for each paper. Two quality reviews would be better than three reviews where two are useless or wrong. This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged.
A third option is to unblind the review process and allow selection, perhaps by a committee, based on record. This has many obvious chicken-egg problems how does one break in to the system? but it would increase the likelihood of securing a quality program. And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).
A fourth option is to make all papers theme dependent. The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldnt be accepted. Reviewers would need only determine fit.
Im sure there are more options, or even combinations. The things Ive suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than were doing now.
Best,
Mike
********************
Michael L. Barnett, PhD
University of South Florida
College of Business Administration
Department of Management Organization
4202 E. Fowler Avenue, BSN 3527
Tampa, FL 33620-5500
Phone: 813-974-1727
Fax: 813-974-1734
Webpage:http://www.coba.usf.edu/barnett
View my research on my SSRN Author page:
http://ssrn.com/author=414796
_______________________________________________________________________
To send a message to the list, send your email to
SIM@aomlists.pace.edu
_______________________________________________________________________
Visit the SIM Division website at:
http://sim.aomonline.org _______________________________________________________________________
If you wish to unsubscribe from this list or change your delivery options, you can do so online at:
http://aomlists.pace.edu/scripts/wa.exe?SUBED1=simA=1 _______________________________________________________________________
_______________________________________________________________________
To send a message to the list, send your email to
SIM@aomlists.pace.edu
_______________________________________________________________________
Visit the SIM Division website at:
http://sim.aomonline.org _______________________________________________________________________
If you wish to unsubscribe from this list or change your delivery options, you can do so online at:
http://aomlists.pace.edu/scripts/wa.exe?SUBED1=simA=1 _______________________________________________________________________
_______________________________________________________________________
To send a message to the list, send your email to
SIM@aomlists.pace.edu
_______________________________________________________________________
Visit the SIM Division website at:
http://sim.aomonline.org _______________________________________________________________________
If you wish to unsubscribe from this list or change your delivery options, you can do so online at:
http://aomlists.pace.edu/scripts/wa.exe?SUBED1=simA=1 ______________________________________________________________________________________________________________________________________________
To send a message to the list, send your email to
SIM@aomlists.pace.edu
_______________________________________________________________________
Visit the SIM Division website at:
http://sim.aomonline.org _______________________________________________________________________
If you wish to unsubscribe from this list or change your delivery options, you can do so online at:
http://aomlists.pace.edu/scripts/wa.exe?SUBED1=simA=1 _______________________________________________________________________
_______________________________________________________________________
To send a message to the list, send your email to
SIM@aomlists.pace.edu
_______________________________________________________________________
Visit the SIM Division website at:
http://sim.aomonline.org_______________________________________________________________________
If you wish to unsubscribe from this list or change your deliveryoptions, you can do so online at:
http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1_______________________________________________________________________
_______________________________________________________________________
To send a message to the list, send your email to
SIM@aomlists.pace.edu
_______________________________________________________________________
Visit the SIM Division website at:
http://sim.aomonline.org
_______________________________________________________________________
If you wish to unsubscribe from this list or change your delivery
options, you can do so online at:
http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1