Discussion: View Thread

Expand all | Collapse all

Is the Academy review process broken? What to do?

  • 1.  Is the Academy review process broken? What to do?

    Posted 03-22-2009 10:29

    Dear fellow SIMians:

     

    I think the review process for the <st1:place w:st="on"><st1:placetype w:st="on">Academy</st1:placetype> of <st1:placename w:st="on">Management</st1:placename></st1:place> conference might be broken, within SIM and perhaps beyond.  I'd like to see if others share my concern, and if it's not just me, I'd like to push the division, or the Academy in general, to do something to fix it. 

     

    This morning, I received notification that my paper was accepted at SIM.  That's good.  Then I read my three reviews – not good.  The first reviewer offered a reasonable analysis and a couple good suggestions.  It wasn't a journal quality review, but it was a solid review – basically, of the caliber we should probably expect from a conference review, given time constraints, etc. 

     

    Then there's the other two.  Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), here's the entirety of the sage feedback I received from the other two reviewers:

     

    R#2:

    Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.

     

    R#3:

    I have an issue with your title.  I think it is inappproriate for an academic conference.  I would leave off "Taking a Chit".

     

    Include some up-to-date references (at least 2007 and above).

     

    So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine – that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have "some" post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).

     

    Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof.  Seriously – it's not that I'm a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but that's not the point), but it's that these reviews don't even address what I wrote. 

     

    Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, I'm concerned about introducing this level of noise into the review process.  How can the program chair make selections when the majority of data are at best useless, and at worst, wrong?  Does this selection process lead to a program that is most useful to Academy members? 

     

    This is not a slam at Barry Mitnick, the SIM division program chair.  If anything, it's a testament to his hard work and dedication, despite the troubled process.  I can't imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time.  Rather, it's a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.

     

    So I think we first need to decide if the current model is broken in a way that can't be fixed.  I think it is.  The problem is that we can't obtain an adequate pool of competent and engaged reviewers.  Therefore, we rely on the incompetent and disengaged, and to an increasing degree.  This is a problem for our major journals (see the recent pleas for reviewers), and so it's going to be even worse for SIM at Academy (can't even imagine the challenge at conferences with lower profiles).  I don't think there's a way to get up to adequate quality while retaining or even increasing quantity.  We could better screen potential reviewers (must have some basic qualifications – don't think we do this now), and we could disallow bad reviewers from reviewing again (don't think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.

     

    If we accept that the current system is broken and can't be fixed, then that frees us up to search for a new system.  I hope the executive committee will do just this.  What are some options?  Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat – 40% get spots.  I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews.  If the result is the same, while the efficiency is much greater, we should do this.

     

    Another option is to decrease the number of reviews required for each paper.  Two quality reviews would be better than three reviews where two are useless or wrong.  This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged. 

     

    A third option is to unblind the review process and allow selection, perhaps by a committee, based on record.  This has many obvious chicken-egg problems – how does one break in to the system? – but it would increase the likelihood of securing a quality program.  And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).

     

    A fourth option is to make all papers theme dependent.  The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldn't be accepted.  Reviewers would need only determine fit.

     

    I'm sure there are more options, or even combinations.  The things I've suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than we're doing now.

     

    Best,

    Mike 

     
    ********************
    Michael L. Barnett, PhD
    University of South Florida
    College of Business Administration
    Department of Management & Organization
    4202 E. Fowler Avenue, BSN 3527
    Tampa, FL 33620-5500
    Phone: 813-974-1727
    Fax: 813-974-1734
     
    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>
     
    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 2.  Is the Academy review process broken? What to do?

    Posted 03-22-2009 10:49

    HI Mike:

    It's always a pleasure to read your lucid and insightful comments on this list-serve.  I understand your frustration.  At the risk of sounding like a stereotypical economist,  I view this as a pricing problem.  With the proliferation of journals, increasing submissions to the AOM meetings,  and greater pressure to publish (tenure committees are often focused more on quantity than quality), it is difficult to find reviewers.  One solution is to raise the cost of submission, which is now zero, while simultaneously compensating reviewers (i.e., the revenue generated from submissions could be used to pay reviewers, with additional funds coming from the division).  In other fields, especially in economics and finance, journals pay reviewers for on-time reviews or offer to donate a certain sum of money to charity.  The latter option might be especially appealing to SIM members.

    Best regards,

    Don

     

    Dr. Donald S. Siegel

    Dean and Professor

    School of Business

    University at Albany, SUNY

    1400 Washington Avenue

    Albany, NY 12222

    Tel: (518) 442-4910

    DSiegel@uamail.albany.edu

    http://www.albany.edu/business/

    http://www.albany.edu/business/news_and_events/DonSiegel-CV.pdf

    http://econpapers.repec.org/RAS/psi32.htm

    http://ssrn.com/author=33607

     

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Barnett, Michael
    Sent: Sunday, March 22, 2009 10:29 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: [SIM] Is the Academy review process broken? What to do?

     

    Dear fellow SIMians:

     

    I think the review process for the Academy of Management conference might be broken, within SIM and perhaps beyond.  I'd like to see if others share my concern, and if it's not just me, I'd like to push the division, or the Academy in general, to do something to fix it. 

     

    This morning, I received notification that my paper was accepted at SIM.  That's good.  Then I read my three reviews – not good.  The first reviewer offered a reasonable analysis and a couple good suggestions.  It wasn't a journal quality review, but it was a solid review – basically, of the caliber we should probably expect from a conference review, given time constraints, etc. 

     

    Then there's the other two.  Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), here's the entirety of the sage feedback I received from the other two reviewers:

     

    R#2:

    Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.

     

    R#3:

    I have an issue with your title.  I think it is inappproriate for an academic conference.  I would leave off "Taking a Chit".

     

    Include some up-to-date references (at least 2007 and above).

     

    So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine – that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have "some" post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).

     

    Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof.  Seriously – it's not that I'm a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but that's not the point), but it's that these reviews don't even address what I wrote. 

     

    Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, I'm concerned about introducing this level of noise into the review process.  How can the program chair make selections when the majority of data are at best useless, and at worst, wrong?  Does this selection process lead to a program that is most useful to Academy members? 

     

    This is not a slam at Barry Mitnick, the SIM division program chair.  If anything, it's a testament to his hard work and dedication, despite the troubled process.  I can't imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time.  Rather, it's a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.

     

    So I think we first need to decide if the current model is broken in a way that can't be fixed.  I think it is.  The problem is that we can't obtain an adequate pool of competent and engaged reviewers.  Therefore, we rely on the incompetent and disengaged, and to an increasing degree.  This is a problem for our major journals (see the recent pleas for reviewers), and so it's going to be even worse for SIM at Academy (can't even imagine the challenge at conferences with lower profiles).  I don't think there's a way to get up to adequate quality while retaining or even increasing quantity.  We could better screen potential reviewers (must have some basic qualifications – don't think we do this now), and we could disallow bad reviewers from reviewing again (don't think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.

     

    If we accept that the current system is broken and can't be fixed, then that frees us up to search for a new system.  I hope the executive committee will do just this.  What are some options?  Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat – 40% get spots.  I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews.  If the result is the same, while the efficiency is much greater, we should do this.

     

    Another option is to decrease the number of reviews required for each paper.  Two quality reviews would be better than three reviews where two are useless or wrong.  This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged. 

     

    A third option is to unblind the review process and allow selection, perhaps by a committee, based on record.  This has many obvious chicken-egg problems – how does one break in to the system? – but it would increase the likelihood of securing a quality program.  And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).

     

    A fourth option is to make all papers theme dependent.  The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldn't be accepted.  Reviewers would need only determine fit.

     

    I'm sure there are more options, or even combinations.  The things I've suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than we're doing now.

     

    Best,

    Mike 

     

    ********************

    Michael L. Barnett, PhD

    University of South Florida

    College of Business Administration

    Department of Management & Organization

    4202 E. Fowler Avenue, BSN 3527

    Tampa, FL 33620-5500

    Phone: 813-974-1727

    Fax: 813-974-1734

     

    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>

     

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 3.  Is the Academy review process broken? What to do?

    Posted 03-22-2009 10:56
    Thanks for the always prompt reply, Don.  It's an interesting idea, but I'm not sure how well it deals with the underlying problem of quality.  I think it could even exacerbate the problem, or at least complicate the process.  People do respond to incentives, but often by providing the minimal level of goods or services necessary to get the incentive.  My R#2 & R#3 might have turned in their "reviews" on time and so received the pay.  So you'd have to set up a more complex system to judge quality of reviews and determine if they're worth the payout (or variable payouts?).  This wheel has been invented elsewhere, as you note, so maybe these challenges have been dealt with, but I'm ignorant of how effective they are.  The charity angle is a neat adaptation, though.
     
    Best,
    Mike
     
    ********************
    Michael L. Barnett, PhD
    University of South Florida
    College of Business Administration
    Department of Management & Organization
    4202 E. Fowler Avenue, BSN 3527
    Tampa, FL 33620-5500
    Phone: 813-974-1727
    Fax: 813-974-1734
     
    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>
     

    From: Donald S Siegel [mailto:DSiegel@uamail.albany.edu]
    Sent: Sun 3/22/2009 10:48 AM
    To: Barnett, Michael; SIM@AOMLISTS.PACE.EDU
    Subject: RE: [SIM] Is the Academy review process broken? What to do?

    HI Mike:

    It's always a pleasure to read your lucid and insightful comments on this list-serve.  I understand your frustration.  At the risk of sounding like a stereotypical economist,  I view this as a pricing problem.  With the proliferation of journals, increasing submissions to the AOM meetings,  and greater pressure to publish (tenure committees are often focused more on quantity than quality), it is difficult to find reviewers.  One solution is to raise the cost of submission, which is now zero, while simultaneously compensating reviewers (i.e., the revenue generated from submissions could be used to pay reviewers, with additional funds coming from the division).  In other fields, especially in economics and finance, journals pay reviewers for on-time reviews or offer to donate a certain sum of money to charity.  The latter option might be especially appealing to SIM members.

    Best regards,

    Don

     

    Dr. Donald S. Siegel

    Dean and Professor

    School of Business

    University at Albany, SUNY

    1400 Washington Avenue

    Albany, NY 12222

    Tel: (518) 442-4910

    DSiegel@uamail.albany.edu

    http://www.albany.edu/business/

    http://www.albany.edu/business/news_and_events/DonSiegel-CV.pdf

    http://econpapers.repec.org/RAS/psi32.htm

    http://ssrn.com/author=33607

     

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Barnett, Michael
    Sent: Sunday, March 22, 2009 10:29 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: [SIM] Is the Academy review process broken? What to do?

     

    Dear fellow SIMians:

     

    I think the review process for the Academy of Management conference might be broken, within SIM and perhaps beyond.  I'd like to see if others share my concern, and if it's not just me, I'd like to push the division, or the Academy in general, to do something to fix it. 

     

    This morning, I received notification that my paper was accepted at SIM.  That's good.  Then I read my three reviews – not good.  The first reviewer offered a reasonable analysis and a couple good suggestions.  It wasn't a journal quality review, but it was a solid review – basically, of the caliber we should probably expect from a conference review, given time constraints, etc. 

     

    Then there's the other two.  Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), here's the entirety of the sage feedback I received from the other two reviewers:

     

    R#2:

    Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.

     

    R#3:

    I have an issue with your title.  I think it is inappproriate for an academic conference.  I would leave off "Taking a Chit".

     

    Include some up-to-date references (at least 2007 and above).

     

    So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine – that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have "some" post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).

     

    Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof.  Seriously – it's not that I'm a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but that's not the point), but it's that these reviews don't even address what I wrote. 

     

    Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, I'm concerned about introducing this level of noise into the review process.  How can the program chair make selections when the majority of data are at best useless, and at worst, wrong?  Does this selection process lead to a program that is most useful to Academy members? 

     

    This is not a slam at Barry Mitnick, the SIM division program chair.  If anything, it's a testament to his hard work and dedication, despite the troubled process.  I can't imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time.  Rather, it's a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.

     

    So I think we first need to decide if the current model is broken in a way that can't be fixed.  I think it is.  The problem is that we can't obtain an adequate pool of competent and engaged reviewers.  Therefore, we rely on the incompetent and disengaged, and to an increasing degree.  This is a problem for our major journals (see the recent pleas for reviewers), and so it's going to be even worse for SIM at Academy (can't even imagine the challenge at conferences with lower profiles).  I don't think there's a way to get up to adequate quality while retaining or even increasing quantity.  We could better screen potential reviewers (must have some basic qualifications – don't think we do this now), and we could disallow bad reviewers from reviewing again (don't think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.

     

    If we accept that the current system is broken and can't be fixed, then that frees us up to search for a new system.  I hope the executive committee will do just this.  What are some options?  Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat – 40% get spots.  I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews.  If the result is the same, while the efficiency is much greater, we should do this.

     

    Another option is to decrease the number of reviews required for each paper.  Two quality reviews would be better than three reviews where two are useless or wrong.  This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged. 

     

    A third option is to unblind the review process and allow selection, perhaps by a committee, based on record.  This has many obvious chicken-egg problems – how does one break in to the system? – but it would increase the likelihood of securing a quality program.  And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).

     

    A fourth option is to make all papers theme dependent.  The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldn't be accepted.  Reviewers would need only determine fit.

     

    I'm sure there are more options, or even combinations.  The things I've suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than we're doing now.

     

    Best,

    Mike 

     

    ********************

    Michael L. Barnett, PhD

    University of South Florida

    College of Business Administration

    Department of Management & Organization

    4202 E. Fowler Avenue, BSN 3527

    Tampa, FL 33620-5500

    Phone: 813-974-1727

    Fax: 813-974-1734

     

    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>

     

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 4.  Is the Academy review process broken? What to do?

    Posted 03-22-2009 11:07
    Dear fellow SIMians:

    I respectfully disagree with Mike about (1) what is wrong with the review process; and (2) solutions to the review process. Ultimately, the goal of the review process is to accept quality papers for the program. Mike's paper was accepted. The process failed if it should not have been accepted, and succeeded if it was accepted.

    Imperfections in the process:
    Need to assign X no. of papers to Y no. of reviewers. This means that there will be imperfect matches between the expertise of a reviewer and the paper that they are assigned to review. As we all have our own niches, there is a need to review papers that are not a perfect match. However, I find that even in that instance, it is pretty easy to differentiate good and bad theory, research design, and writing.

    Assessing the quality of a review only by comments:
    Reviewers rate papers. They don't all comment. Just as our student evaluations are based on nos. but we tend to consider comments. And, remember, reviewers can say confidential things to the session chair. Don't condemn the entire system based on one aspect.

    Feedback isn't a process that comes naturally. Critical feedback is even more difficult. My mother used to say: "If you can't say anything nice, don't say anything at all." Obviously not a good way to review. But, even with blind reviews, it is a problem. What to do? Training!

    Training of reviewers is spotty at best. Brief reviews may be made by inexperienced doctoral students who we URGE to participate by reviewing. Should we stop? I think not. Encouraging early participation is a GOOD thing.

    How else to learn to review than by doing? I have sought out opportunities to learn to be a better reviewer. Provide PDWs--EVERY YEAR. Set up an online training for reviewers at the SIM website. Improve quality through better training!

    Hope this is helpful to the discussion.

    Amy

    -
    Amy Klemm Verbos, J.D.
    PhD Candidate & Adjunct Professor
    Lubar School of Business
    University of Wisconsin-Milwaukee

    -

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org
    _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery
    options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1


  • 5.  Is the Academy review process broken? What to do?

    Posted 03-22-2009 11:14
    Mike and others:

    First, congratulations Mike for having a paper in the Academy -- I still think
    it's an accomplishment. I do, however, agree with you about the quality of the
    reviews ,,, and as you push the theoretical envelope they tend to get worse. I
    place the blame squarely on us. As a scholarly body we just don't (can't) bring
    ourselves to up the quality. Maybe our respective institutions don't give
    credit for reviewing, maybe we're so specialized we can't see the merits of
    other apporaches, maybe we don't train doctoral candidates well enough for
    effective reviews (a potential annual PDW?). I don't think it's anything more
    sinister than institutional obstacles -- I don't think we are purposely passing
    along papers that parrott entrenched views -- but that's what happens when the
    scholarly imagination is crimped by a small pool of reviewers -- I wonder how
    many were doctoral students? I didn't do any reviewing this year. I should
    have made it point to...I won't burden anyone with excuses. I help when I
    think I can. That's not OK. I submitted papers that were accepted that
    probably weren't clear enough, well enough linked to the literature, precise
    enough in describing the methods, well enough organized to assist the reviewer
    to know where to help. Interestingly for SIM, the SBE due date is right behind
    the Academy -- I wonder what people think of SBE reviews? Anyway -- I hope we
    can improve the system -- some of your ideas are interesting (although I
    wouldn't identify the reviewers -- it's not all their fault!). When I look
    around the Academy, I see lot of interest groups or divisions that were born to
    SIM -- maybe that's the way it's destined to go -- more and more
    specialization. g



    Quoting "Barnett, Michael" <mbarnett@coba.usf.edu>:

    > Dear fellow SIMians:
    >
    >
    >
    > I think the review process for the Academy of Management conference might be
    > broken, within SIM and perhaps beyond. I'd like to see if others share my
    > concern, and if it's not just me, I'd like to push the division, or the
    > Academy in general, to do something to fix it.
    >
    >
    >
    > This morning, I received notification that my paper was accepted at SIM.
    > That's good. Then I read my three reviews - not good. The first reviewer
    > offered a reasonable analysis and a couple good suggestions. It wasn't a
    > journal quality review, but it was a solid review - basically, of the caliber
    > we should probably expect from a conference review, given time constraints,
    > etc.
    >
    >
    >
    > Then there's the other two. Bearing in mind that the paper I submitted is a
    > theory paper and has an intentionally edgy title designed to play off the
    > existing literature and catch attention (Take a chit: Cognitive constraints
    > on stakeholder response to corporate misconduct), here's the entirety of the
    > sage feedback I received from the other two reviewers:
    >
    >
    >
    > R#2:
    >
    > Authors provide an interesting perspective on why CSR efforts may/often fall
    > short. Paper would make an interesting conference discussion, and I would be
    > interested to see the authors developing testable hypotheses from the
    > propositions and collecting data.
    >
    >
    >
    > R#3:
    >
    > I have an issue with your title. I think it is inappproriate for an academic
    > conference. I would leave off "Taking a Chit".
    >
    >
    >
    > Include some up-to-date references (at least 2007 and above).
    >
    >
    >
    > So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it
    > for a research proposal (not that uncommon outside our world, but we should
    > be getting reviews by people in our world), and R#3 focuses on the title
    > (fine - that was expected) and takes issue with references being from earlier
    > than the prior year (this was submitted at the end of 2008; obviously written
    > prior to that; also, I suspect I did have "some" post 2006 cites; and
    > interestingly, R#1 rightfully suggests I add some cites from several decades
    > prior).
    >
    >
    >
    > Two-thirds of my reviewers were disinterested, incompetent, or some
    > combination thereof. Seriously - it's not that I'm a prima donna author who
    > is angered that others have not fully appreciated my written genius (maybe I
    > am but that's not the point), but it's that these reviews don't even address
    > what I wrote.
    >
    >
    >
    > Beyond my anger that these two reviewers, for whatever it is worth to them,
    > will be allowed to claim to have engaged in service to the profession, I'm
    > concerned about introducing this level of noise into the review process. How
    > can the program chair make selections when the majority of data are at best
    > useless, and at worst, wrong? Does this selection process lead to a program
    > that is most useful to Academy members?
    >
    >
    >
    > This is not a slam at Barry Mitnick, the SIM division program chair. If
    > anything, it's a testament to his hard work and dedication, despite the
    > troubled process. I can't imagine the headaches he has had, trying to sort
    > information from noise, incompetence from earnestness, and so on, all on a
    > grand scale and with limited time. Rather, it's a call to change the process
    > so that it becomes more useful, and perhaps efficient, if at all possible.
    >
    >
    >
    > So I think we first need to decide if the current model is broken in a way
    > that can't be fixed. I think it is. The problem is that we can't obtain an
    > adequate pool of competent and engaged reviewers. Therefore, we rely on the
    > incompetent and disengaged, and to an increasing degree. This is a problem
    > for our major journals (see the recent pleas for reviewers), and so it's
    > going to be even worse for SIM at Academy (can't even imagine the challenge
    > at conferences with lower profiles). I don't think there's a way to get up
    > to adequate quality while retaining or even increasing quantity. We could
    > better screen potential reviewers (must have some basic qualifications -
    > don't think we do this now), and we could disallow bad reviewers from
    > reviewing again (don't think we do this either), but we seem to need any
    > semi-warm body we can find if we are to have the numbers.
    >
    >
    >
    > If we accept that the current system is broken and can't be fixed, then that
    > frees us up to search for a new system. I hope the executive committee will
    > do just this. What are some options? Most cynically, since the current
    > system introduces a great deal of randomness, we might as well draw names
    > from a hat - 40% get spots. I seriously believe the outcome will approximate
    > the current process in a few short years, as the reviewer pool further
    > degrades under increased demand for reviews. If the result is the same,
    > while the efficiency is much greater, we should do this.
    >
    >
    >
    > Another option is to decrease the number of reviews required for each paper.
    > Two quality reviews would be better than three reviews where two are useless
    > or wrong. This would allow for more stringent screening of reviewers,
    > enabling us to leave out more of the incompetent and disengaged.
    >
    >
    >
    > A third option is to unblind the review process and allow selection, perhaps
    > by a committee, based on record. This has many obvious chicken-egg problems
    > - how does one break in to the system? - but it would increase the likelihood
    > of securing a quality program. And if high status folks come to abuse their
    > ability to get in by offering up bad papers, they could be barred from future
    > seating based on record and instead have to go through a more intensive
    > selection process (so this process could also degrade in time).
    >
    >
    >
    > A fourth option is to make all papers theme dependent. The committee could
    > establish a set of topics the prior year, and then papers would need to fit
    > with one of the themes, else it wouldn't be accepted. Reviewers would need
    > only determine fit.
    >
    >
    >
    > I'm sure there are more options, or even combinations. The things I've
    > suggested are off the top of my head. I hope we can put more of our heads
    > together, so that we can do better than we're doing now.
    >
    >
    >
    > Best,
    >
    > Mike
    >
    >
    > ********************
    > Michael L. Barnett, PhD
    > University of South Florida
    > College of Business Administration
    > Department of Management & Organization
    > 4202 E. Fowler Avenue, BSN 3527
    > Tampa, FL 33620-5500
    > Phone: 813-974-1727
    > Fax: 813-974-1734
    > Webpage: http://www.coba.usf.edu/barnett
    >
    > View my research on my SSRN Author page:
    > <http://ssrn.com/author=414796
    > <https://email.usf.edu/exchweb/bin/redir.asp?
    URL=http://ssrn.com/author=414796>
    > >
    >
    >
    > _______________________________________________________________________
    >
    > To send a message to the list, send your email to SIM@aomlists.pace.edu
    >
    > _______________________________________________________________________
    >
    > Visit the SIM Division website at: http://sim.aomonline.org
    > _______________________________________________________________________
    >
    > If you wish to unsubscribe from this list or change your delivery
    > options, you can do so online at:
    > http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1
    > _______________________________________________________________________
    >
    >
    >



    -------------------------------------------------
    SIUE Web Mail

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org
    _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery
    options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1


  • 6.  Is the Academy review process broken? What to do?

    Posted 03-22-2009 11:34
    Thanks for the interesting points, Amy.  I surely shouldn't make a habit of responding to what I hope are many replies, but I'll respond to yours, and then step back and, despite my urges otherwise, just observe responses for a while. 
     
    That's an important distinction you made about the purpose of the review system.  I concentrated on selection in my note.  Perhaps the numerical comments and notes to the editor items in the current process are of sufficient caliber to achieve a good selection outcome.  I doubt it -- I don't see how they'd be independent from comments (there are certain "reviewer effects" here).  Also, comparison with the student rating process, in my mind, only further highlights the flaws of the system.  A further complication, though, is that there are other selection issues, such as best paper proceedings, that may require qualitative insights.  But beyond this, I think we should also view this as a development process, and we should expect the reviews to provide meaningful and constructive insights to the authors.  Maybe I'm wrong here -- perhaps this is something we can sacrifice.  We've certainly come not to expect it. 
     
    On the reviewer training angle, that's a good idea.  If you don't have sufficient experience, perhaps you can "test in" to the reviewer pool by attending a certain class (maybe online).  This may help somewhat, though I think a good chunk of this remains lack of effort.  This might be where Don's ideas on incentives would come in.
     
    Interesting discussion!
     
    Best,
    Mike
     
    ********************
    Michael L. Barnett, PhD
    University of South Florida
    College of Business Administration
    Department of Management & Organization
    4202 E. Fowler Avenue, BSN 3527
    Tampa, FL 33620-5500
    Phone: 813-974-1727
    Fax: 813-974-1734
     
    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>
     

    From: Social Issues in Management Listserv on behalf of Amy Klemm Verbos
    Sent: Sun 3/22/2009 11:06 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

    Dear fellow SIMians:

    I respectfully disagree with Mike about (1) what is wrong with the review process; and (2) solutions to the review process. Ultimately, the goal of the review process is to accept quality papers for the program. Mike's paper was accepted. The process failed if it should not have been accepted, and succeeded if it was accepted.

    Imperfections in the process:
    Need to assign X no. of papers to Y no. of reviewers. This means that there will be imperfect matches between the expertise of a reviewer and the paper that they are assigned to review. As we all have our own niches, there is a need to review papers that are not a perfect match. However, I find that even in that instance, it is pretty easy to differentiate good and bad theory, research design, and writing.

    Assessing the quality of a review only by comments:
    Reviewers rate papers. They don't all comment. Just as our student evaluations are based on nos. but we tend to consider comments. And, remember, reviewers can say confidential things to the session chair. Don't condemn the entire system based on one aspect.

    Feedback isn't a process that comes naturally. Critical feedback is even more difficult. My mother used to say: "If you can't say anything nice, don't say anything at all." Obviously not a good way to review. But, even with blind reviews, it is a problem. What to do? Training!

    Training of reviewers is spotty at best.  Brief reviews may be made by inexperienced doctoral students who we URGE to participate by reviewing. Should we stop? I think not. Encouraging early participation is a GOOD thing.

    How else to learn to review than by doing? I have sought out opportunities to learn to be a better reviewer. Provide PDWs--EVERY YEAR. Set up an online training for reviewers at the SIM website. Improve quality through better training!

    Hope this is helpful to the discussion.

    Amy

    -
    Amy Klemm Verbos, J.D.
    PhD Candidate & Adjunct Professor
    Lubar School of Business
    University of Wisconsin-Milwaukee

    -

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org
    _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery
    options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1
    _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 7.  Is the Academy review process broken? What to do?

    Posted 03-22-2009 11:54

    It's hard to know if the conference process is fully "broken" but it is worth worrying about because careers depend on it.   Doctoral students are under pressure to have papers on the program as they develop market-ready cv's, and untenured professors are under pressure to show that they are presenting their work at the right meetings.  At many institutions accepted papers at refereed conferences are significant elements of a tenure package.  Accordingly, a process that is flawed or arbitrary has real and significant consequences for real humans. 

     

    There are a few things that program chairs could do without too much commitment of extra time that might help:  (1) Examine reviews for papers that are marginal around the accept cut-off point (perhaps the first 10% or 20% that didn't make it), and eliminate nakedly bad reviews to see if it yields a change in outcome.  (2) Scan all reviews quickly after the process is ended, and drop a polite but firm note to particularly slack reviewers telling them that they wrote reviews of substandard quality.  (3) As someone else suggested, experiment with fewer reviewers per paper.  (4) Consider having more structured evaluation items, which compel a reviewer to weigh in with a judgment of more numerous and more specific elements of the paper – although that's a dodgy one since it risks inviting reviewers to write less in their unstructured comments.

     

    bb

     

    -----

    Bruce Barry

    Brownlee O. Currey, Jr. Professor of Management

    Professor of Sociology

    Vanderbilt University

     

     

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 8.  Is the Academy review process broken? What to do?

    Posted 03-22-2009 11:58
    Mike,
     
    I have been in the same situation, with some reviewers giving very good comments, some giving "general" comments.  The problem also rests with the limited time (resources) necessary to provide comments when most of our semesters are commencing which is usually a busy time for most of us.  We also give priorities to reviewing levels, a top journal would probably get more of our time than a middle level journal which would get more of our time than a conference.  So, there are many reasons for these initial reviews not meeting the standards of a hiqh quality journal or our higher quality expectations.
     
    But, I'm not so sure the system is broken.  Especially when you consider the whole picture and purpose of the conference.  You received at least one good set of comments.  That is better than none.  Also, a conference paper is usually deemed a work in progress, so you can expect to receive additional comments later.  Some divisions will have discussants.  Other people will download your paper from the site and maybe read it and even contact you with comments (rare, but it is done).  Hopefully, you will get comments back from people who attend your session (or at the poster session or at the roundtable, or whatever they have).  Altogether the more complete picture is that you will typically get many comments over the whole conference process.  Thus, I view the whole process as rather successful for a volunteer situation.  Given the types of conferences I have seen over the years, the AOM conference provides some of the best feedback I have received on my work.
     
    Can it be improved? Yes.  Is it completely useless? I don't think so.
     
    I think it really goes back to expectations.  I am happy to receive any comments back from conferences. 
     
    -Joe
    From: Social Issues in Management Listserv [SIM@AOMLISTS.PACE.EDU] On Behalf Of Barnett, Michael [mbarnett@coba.usf.edu]
    Sent: Sunday, March 22, 2009 11:34 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

    Thanks for the interesting points, Amy.  I surely shouldn't make a habit of responding to what I hope are many replies, but I'll respond to yours, and then step back and, despite my urges otherwise, just observe responses for a while. 
     
    That's an important distinction you made about the purpose of the review system.  I concentrated on selection in my note.  Perhaps the numerical comments and notes to the editor items in the current process are of sufficient caliber to achieve a good selection outcome.  I doubt it -- I don't see how they'd be independent from comments (there are certain "reviewer effects" here).  Also, comparison with the student rating process, in my mind, only further highlights the flaws of the system.  A further complication, though, is that there are other selection issues, such as best paper proceedings, that may require qualitative insights.  But beyond this, I think we should also view this as a development process, and we should expect the reviews to provide meaningful and constructive insights to the authors.  Maybe I'm wrong here -- perhaps this is something we can sacrifice.  We've certainly come not to expect it. 
     
    On the reviewer training angle, that's a good idea.  If you don't have sufficient experience, perhaps you can "test in" to the reviewer pool by attending a certain class (maybe online).  This may help somewhat, though I think a good chunk of this remains lack of effort.  This might be where Don's ideas on incentives would come in.
     
    Interesting discussion!
     
    Best,
    Mike
     
    ********************
    Michael L. Barnett, PhD
    University of South Florida
    College of Business Administration
    Department of Management & Organization
    4202 E. Fowler Avenue, BSN 3527
    Tampa, FL 33620-5500
    Phone: 813-974-1727
    Fax: 813-974-1734
     
    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>
     

    From: Social Issues in Management Listserv on behalf of Amy Klemm Verbos
    Sent: Sun 3/22/2009 11:06 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

    Dear fellow SIMians:

    I respectfully disagree with Mike about (1) what is wrong with the review process; and (2) solutions to the review process. Ultimately, the goal of the review process is to accept quality papers for the program. Mike's paper was accepted. The process failed if it should not have been accepted, and succeeded if it was accepted.

    Imperfections in the process:
    Need to assign X no. of papers to Y no. of reviewers. This means that there will be imperfect matches between the expertise of a reviewer and the paper that they are assigned to review. As we all have our own niches, there is a need to review papers that are not a perfect match. However, I find that even in that instance, it is pretty easy to differentiate good and bad theory, research design, and writing.

    Assessing the quality of a review only by comments:
    Reviewers rate papers. They don't all comment. Just as our student evaluations are based on nos. but we tend to consider comments. And, remember, reviewers can say confidential things to the session chair. Don't condemn the entire system based on one aspect.

    Feedback isn't a process that comes naturally. Critical feedback is even more difficult. My mother used to say: "If you can't say anything nice, don't say anything at all." Obviously not a good way to review. But, even with blind reviews, it is a problem. What to do? Training!

    Training of reviewers is spotty at best.  Brief reviews may be made by inexperienced doctoral students who we URGE to participate by reviewing. Should we stop? I think not. Encouraging early participation is a GOOD thing.

    How else to learn to review than by doing? I have sought out opportunities to learn to be a better reviewer. Provide PDWs--EVERY YEAR. Set up an online training for reviewers at the SIM website. Improve quality through better training!

    Hope this is helpful to the discussion.

    Amy

    -
    Amy Klemm Verbos, J.D.
    PhD Candidate & Adjunct Professor
    Lubar School of Business
    University of Wisconsin-Milwaukee

    -

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org
    _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery
    options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1
    _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 9.  Is the Academy review process broken? What to do?

    Posted 03-22-2009 12:35

    Mike,

    Having served as Program Chair, I delve into this discussion hesitantly.

    But, I may have some information that you'll find useful.

     

    Over the past couple of years, we have engaged in efforts to improve the quality of reviewers.  At least, we've attempted to collect information about expertise and experience.

    I'll leave it to Barry to address whether those are working.

     

    When I was Program Chair, I assigned papers to multiple reviewers with the understanding that some reviewers were clearly more capable and experienced than others.  For example, I might assign the third review to a doctoral student who had never reviewed before, understanding that I could not count on the quality of that review as much as I could a clearly experienced and qualified reviewer.  But, I felt that I was helping a doctoral student gain experience with reviewing.  My ultimate assessments of the paper took that quality into account and I hope that I made the right decisions most of the time.

     

    We should be careful not to think of AOM conference reviewing like we do the journal review process.  Many reviewers have agreed to do 3 reviews from SIM and some reviewers also review for other divisions.  The reviews are due back pretty quickly, making it difficult to write lengthy reviews at a time when most of us are also teaching.  I didn't expect lengthy reviews.  I was happy to get timely reviews with reasonable consensus about the quality of the paper and whether to include it on the program.

     

    Having said all of that, the year I did this, I was surprised that some very capable SIMiams did not sign up to review.  They may have had good reasons.  But, it certainly made my job more difficult.  I encourage SIMians to view this contribution to the profession as your professional responsibility.  This is also a topic I wrote about last year when I was still Associate Editor of AMR.

     

     

    Linda Treviño

    Chair, SIM Division

     

     

     

     

     

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Barnett, Michael
    Sent: Sunday, March 22, 2009 10:29 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: [SIM] Is the Academy review process broken? What to do?

     

    Dear fellow SIMians:

     

    I think the review process for the Academy of Management conference might be broken, within SIM and perhaps beyond.  I'd like to see if others share my concern, and if it's not just me, I'd like to push the division, or the Academy in general, to do something to fix it. 

     

    This morning, I received notification that my paper was accepted at SIM.  That's good.  Then I read my three reviews – not good.  The first reviewer offered a reasonable analysis and a couple good suggestions.  It wasn't a journal quality review, but it was a solid review – basically, of the caliber we should probably expect from a conference review, given time constraints, etc. 

     

    Then there's the other two.  Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), here's the entirety of the sage feedback I received from the other two reviewers:

     

    R#2:

    Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.

     

    R#3:

    I have an issue with your title.  I think it is inappproriate for an academic conference.  I would leave off "Taking a Chit".

     

    Include some up-to-date references (at least 2007 and above).

     

    So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine – that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have "some" post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).

     

    Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof.  Seriously – it's not that I'm a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but that's not the point), but it's that these reviews don't even address what I wrote. 

     

    Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, I'm concerned about introducing this level of noise into the review process.  How can the program chair make selections when the majority of data are at best useless, and at worst, wrong?  Does this selection process lead to a program that is most useful to Academy members? 

     

    This is not a slam at Barry Mitnick, the SIM division program chair.  If anything, it's a testament to his hard work and dedication, despite the troubled process.  I can't imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time.  Rather, it's a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.

     

    So I think we first need to decide if the current model is broken in a way that can't be fixed.  I think it is.  The problem is that we can't obtain an adequate pool of competent and engaged reviewers.  Therefore, we rely on the incompetent and disengaged, and to an increasing degree.  This is a problem for our major journals (see the recent pleas for reviewers), and so it's going to be even worse for SIM at Academy (can't even imagine the challenge at conferences with lower profiles).  I don't think there's a way to get up to adequate quality while retaining or even increasing quantity.  We could better screen potential reviewers (must have some basic qualifications – don't think we do this now), and we could disallow bad reviewers from reviewing again (don't think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.

     

    If we accept that the current system is broken and can't be fixed, then that frees us up to search for a new system.  I hope the executive committee will do just this.  What are some options?  Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat – 40% get spots.  I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews.  If the result is the same, while the efficiency is much greater, we should do this.

     

    Another option is to decrease the number of reviews required for each paper.  Two quality reviews would be better than three reviews where two are useless or wrong.  This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged. 

     

    A third option is to unblind the review process and allow selection, perhaps by a committee, based on record.  This has many obvious chicken-egg problems – how does one break in to the system? – but it would increase the likelihood of securing a quality program.  And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).

     

    A fourth option is to make all papers theme dependent.  The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldn't be accepted.  Reviewers would need only determine fit.

     

    I'm sure there are more options, or even combinations.  The things I've suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than we're doing now.

     

    Best,

    Mike 

     

    ********************

    Michael L. Barnett, PhD

    University of South Florida

    College of Business Administration

    Department of Management & Organization

    4202 E. Fowler Avenue, BSN 3527

    Tampa, FL 33620-5500

    Phone: 813-974-1727

    Fax: 813-974-1734

     

    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>

     

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 10.  Is the Academy review process broken? What to do?

    Posted 03-22-2009 18:13

    All ... I like Linda was a program chair and concur with her.  With a few caveats and reinforcements.

    First on the conference:

    1.  Good program chairs will distribute papers out to reviewers of varying experience.  This is just the necessity of having to get 100, 200, 300 papers reviewed in six weeks.  In reading teh reviews you know what is a good review and what is not and how to make a decision based on that.
    2.  The relationship between experience and a good review is not clear.  It is not uncommon to find more senior PhD students and Jnr faculty being the most engaged and those most likely to provide excellent reviews.  Indeed, the IM Division gives out both best reviewer awards and outstanding reviewer awards (for those doing well over several years) and these are quite prized.  The correlations between 'status' and reviewer quality is an empirical issue but I would say that any relationship found would be weak.
    3.  The key in the conference is that it is a way station for ideas.  All the reviews really tell you is that idea is worthy of inclusion.  It is not a journal review and if you look at the link between papers in the best paper proceedings and ultimate paper publications you will find the correlation woefully weak!
    4.  The AOM gives you the opportunity to rate reviewers.  This is more than many journals and other conferences.  It is a good idea and should be used.  I know that we use it to make decisions about best reviewers and who we use in the future.

    Now to the other issues:

    As Don noted so clearly it is a pricing issue.  The number of papers coming into conferences and journals is expanding dramatically and this puts strains on the system.  Even getting the right people to review in any given area is difficult.  When I was an Assoc Editor of Mgt Science 20 years ago it was a lot easier than what I have to do today.  Indeed, it takes sometimes 10 people to get 2 reviews.  My own ed board activity means that I end up doing 10 papers a month on average.  Others do even more.  So the problem is that a lot more goes into the system than can be handled adequately by 'trained' reviewers.

    That said, the quality of the journal reviews, has gone down dramatically.  What this means is that you end up with either a lot of prefunctory reviews or huge variance.  This puts pressure on good editors to make margin calls based on their views and experience.  Bad editors just listen to the reviewers, which leads to real science being drowned out by mediocrity and marginal incrementalism (how's that for a value judgement).  Just to hint at this, on one journal where I had ed responsibilities, I did not have a single paper I accepted that had all reviewers on side -- If I waited for the reviewers to converge I would either still be waiting or would have virtually no papers to publish (and certainly no interesting ones as the reviewers would have sanitized them).

    So Michael  ... it is a terrible system ... but give me an alternative that would work! .... I like many actually pay more attention to working papers these days than journal papers.  With things like SSRN you can get the news early and unfettered by opinion! ... it may not look good, but as scientists we can make our own judgements.

    Tim


    Prof. Timothy Devinney

    Global Numbers: +61 2 8006 0048 or +1 412 5677440

    Other Numbers:
    Australia (Mob): +61 412 276 467
    USA (Mob): +1 (847) 207 3202
    Germany (Mob): +49 (151) 5316 7974
    China (Mob): +86 (136) 5116 1087
    Email: T.Devinney@unsw.edu.au

    Web 1 (Academic):     http://www.agsm.edu.au/tdevinney
    Web 2 (China Related):     http://vincitveritas.spaces.live.com/
    Flickr:     http://Flickr.com ... Search for the "tag" Timothy_Devinney or use http://flickr.com/search/people/?q=t.devinney
    Linda Trevino <LTrevino@psu.edu>
    Sent by: Social Issues in Management Listserv <SIM@AOMLISTS.PACE.EDU>

    23/03/2009 03:35 AM

    Please respond to
    Linda Trevino <LTrevino@psu.edu>
    To
    		SIM@AOMLISTS.PACE.EDU
    cc
    Subject
    		Re: [SIM] Is the Academy review process broken?  What to do?





    Mike,
    Having served as Program Chair, I delve into this discussion hesitantly.
    But, I may have some information that you'll find useful.
     
    Over the past couple of years, we have engaged in efforts to improve the quality of reviewers.  At least, we've attempted to collect information about expertise and experience.
    I'll leave it to Barry to address whether those are working.
     
    When I was Program Chair, I assigned papers to multiple reviewers with the understanding that some reviewers were clearly more capable and experienced than others.  For example, I might assign the third review to a doctoral student who had never reviewed before, understanding that I could not count on the quality of that review as much as I could a clearly experienced and qualified reviewer.  But, I felt that I was helping a doctoral student gain experience with reviewing.  My ultimate assessments of the paper took that quality into account and I hope that I made the right decisions most of the time.
     
    We should be careful not to think of AOM conference reviewing like we do the journal review process.  Many reviewers have agreed to do 3 reviews from SIM and some reviewers also review for other divisions.  The reviews are due back pretty quickly, making it difficult to write lengthy reviews at a time when most of us are also teaching.  I didn't expect lengthy reviews.  I was happy to get timely reviews with reasonable consensus about the quality of the paper and whether to include it on the program.
     
    Having said all of that, the year I did this, I was surprised that some very capable SIMiams did not sign up to review.  They may have had good reasons.  But, it certainly made my job more difficult.  I encourage SIMians to view this contribution to the profession as your professional responsibility.  This is also a topic I wrote about last year when I was still Associate Editor of AMR.
     
     
    Linda Treviño
    Chair, SIM Division
     
     
     
     
     
    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Barnett, Michael
    Sent:     Sunday, March 22, 2009 10:29 AM
    To:     SIM@AOMLISTS.PACE.EDU
    Subject:     [SIM] Is the Academy review process broken? What to do?
     
    Dear fellow SIMians:
     
    I think the review process for the Academy of Management conference might be broken, within SIM and perhaps beyond.  I'd like to see if others share my concern, and if it's not just me, I'd like to push the division, or the Academy in general, to do something to fix it. 
     
    This morning, I received notification that my paper was accepted at SIM.  That's good.  Then I read my three reviews – not good.  The first reviewer offered a reasonable analysis and a couple good suggestions.  It wasn't a journal quality review, but it was a solid review – basically, of the caliber we should probably expect from a conference review, given time constraints, etc. 
     
    Then there's the other two.  Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), here's the entirety of the sage feedback I received from the other two reviewers:
     
    R#2:
    Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.
     
    R#3:
    I have an issue with your title.  I think it is inappproriate for an academic conference.  I would leave off "Taking a Chit".
     
    Include some up-to-date references (at least 2007 and above).
     
    So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine – that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have "some" post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).
     
    Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof.  Seriously – it's not that I'm a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but that's not the point), but it's that these reviews don't even address what I wrote. 
     
    Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, I'm concerned about introducing this level of noise into the review process.  How can the program chair make selections when the majority of data are at best useless, and at worst, wrong?  Does this selection process lead to a program that is most useful to Academy members? 
     
    This is not a slam at Barry Mitnick, the SIM division program chair.  If anything, it's a testament to his hard work and dedication, despite the troubled process.  I can't imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time.  Rather, it's a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.
     
    So I think we first need to decide if the current model is broken in a way that can't be fixed.  I think it is.  The problem is that we can't obtain an adequate pool of competent and engaged reviewers.  Therefore, we rely on the incompetent and disengaged, and to an increasing degree.  This is a problem for our major journals (see the recent pleas for reviewers), and so it's going to be even worse for SIM at Academy (can't even imagine the challenge at conferences with lower profiles).  I don't think there's a way to get up to adequate quality while retaining or even increasing quantity.  We could better screen potential reviewers (must have some basic qualifications – don't think we do this now), and we could disallow bad reviewers from reviewing again (don't think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.
     
    If we accept that the current system is broken and can't be fixed, then that frees us up to search for a new system.  I hope the executive committee will do just this.  What are some options?  Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat – 40% get spots.  I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews.  If the result is the same, while the efficiency is much greater, we should do this.
     
    Another option is to decrease the number of reviews required for each paper.  Two quality reviews would be better than three reviews where two are useless or wrong.  This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged. 
     
    A third option is to unblind the review process and allow selection, perhaps by a committee, based on record.  This has many obvious chicken-egg problems – how does one break in to the system? – but it would increase the likelihood of securing a quality program.  And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).
     
    A fourth option is to make all papers theme dependent.  The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldn't be accepted.  Reviewers would need only determine fit.
     
    I'm sure there are more options, or even combinations.  The things I've suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than we're doing now.
     
    Best,
    Mike 
     
    ********************
    Michael L. Barnett, PhD
    University of South Florida
    College of Business Administration
    Department of Management & Organization
    4202 E. Fowler Avenue, BSN 3527
    Tampa, FL 33620-5500
    Phone: 813-974-1727
    Fax: 813-974-1734
    Webpage: http://www.coba.usf.edu/barnett
     
    View my research on my SSRN Author page:
    <    http://ssrn.com/author=414796>
     
    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________ _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 11.  Is the Academy review process broken? What to do?

    Posted 03-23-2009 09:14

    I am not an economist, but I do think Don's pricing model makes a sense if we can come up with pricing level that would not become a barrier for unfunded Ph.D. students (or maybe create an alternative system for students).  

    Another suggestion I have is to reduce the length of papers to present only the essence of the paper.  I was speaking with a friend in Physics recently, and he was astonished that our papers are often 40-50 pages long.  His words: "how can you possibly read all of them and reflect on them?"  Meaning if we spend too much time reading, we don't have enough time/energy/motivation to think....  Or, if we are pressed against time, then we simply do not read the papers very carefully and come up with just perfunctory comments.  As all of you know, good review takes time and good review on 40 pages takes very long time.  

    There is another advantage of limiting paper length. I believe writing a good short paper that just contains the essence would be more difficult than writing a longer paper.  And reviewing a very short paper for quality is probably easier, because authors cannot hide behind long quotes, lit. reviews and rhetoric etc. I tested this hypothesis with my class (student essays), and it seems to be true J   So, the burden of proof falls on authors, not reviewers.

    I also think that since we will hear the presentations, we really do not need the full paper.... Papers can be submitted to journals, not AOM conference.

    So, my suggestion is to limit the length of papers to something like 2000 words and allow reviewers more time to think.

    That's my 2 cents.

     

    Paul

     

    Min-Dong Paul Lee, Assistant Professor

    Department of Management & Organization

    <st1:place w:st="on"><st1:placetype w:st="on">College</st1:placetype> of <st1:placename w:st="on">Business</st1:placename></st1:place>

    <st1:place w:st="on"><st1:city w:st="on">University of South</st1:city> <st1:state w:st="on">Florida</st1:state></st1:place>

    <st1:street w:st="on"><st1:address w:st="on">4202 E. Flowler Ave.</st1:address></st1:street>, BSN 3403

    <st1:place w:st="on"><st1:city w:st="on">Tampa</st1:city>, <st1:state w:st="on">FL</st1:state>  <st1:postalcode w:st="on">33620</st1:postalcode></st1:place>

    (813) 974-1721

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Donald S Siegel
    Sent: Sunday, March 22, 2009 10:49 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

     

    HI Mike:

    It's always a pleasure to read your lucid and insightful comments on this list-serve.  I understand your frustration.  At the risk of sounding like a stereotypical economist,  I view this as a pricing problem.  With the proliferation of journals, increasing submissions to the AOM meetings,  and greater pressure to publish (tenure committees are often focused more on quantity than quality), it is difficult to find reviewers.  One solution is to raise the cost of submission, which is now zero, while simultaneously compensating reviewers (i.e., the revenue generated from submissions could be used to pay reviewers, with additional funds coming from the division).  In other fields, especially in economics and finance, journals pay reviewers for on-time reviews or offer to donate a certain sum of money to charity.  The latter option might be especially appealing to SIM members.

    Best regards,

    Don

     

    Dr. Donald S. Siegel

    Dean and Professor

    <st1:place w:st="on"><st1:placetype w:st="on">School</st1:placetype> of <st1:placename w:st="on">Business</st1:placename></st1:place>

    University at <st1:city w:st="on"><st1:place w:st="on">Albany</st1:place></st1:city>, SUNY

    <st1:street w:st="on"><st1:address w:st="on">1400 Washington Avenue</st1:address></st1:street>

    <st1:place w:st="on"><st1:city w:st="on">Albany</st1:city>, <st1:state w:st="on">NY</st1:state> <st1:postalcode w:st="on">12222</st1:postalcode></st1:place>

    Tel: (518) 442-4910

    DSiegel@uamail.albany.edu

    http://www.albany.edu/business/

    http://www.albany.edu/business/news_and_events/DonSiegel-CV.pdf

    http://econpapers.repec.org/RAS/psi32.htm

    http://ssrn.com/author=33607

     

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Barnett, Michael
    Sent: Sunday, March 22, 2009 10:29 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: [SIM] Is the Academy review process broken? What to do?

     

    Dear fellow SIMians:

     

    I think the review process for the <st1:place w:st="on"><st1:placetype w:st="on">Academy</st1:placetype> of <st1:placename w:st="on">Management</st1:placename></st1:place> conference might be broken, within SIM and perhaps beyond.  I'd like to see if others share my concern, and if it's not just me, I'd like to push the division, or the Academy in general, to do something to fix it. 

     

    This morning, I received notification that my paper was accepted at SIM.  That's good.  Then I read my three reviews – not good.  The first reviewer offered a reasonable analysis and a couple good suggestions.  It wasn't a journal quality review, but it was a solid review – basically, of the caliber we should probably expect from a conference review, given time constraints, etc. 

     

    Then there's the other two.  Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), here's the entirety of the sage feedback I received from the other two reviewers:

     

    R#2:

    Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.

     

    R#3:

    I have an issue with your title.  I think it is inappproriate for an academic conference.  I would leave off "Taking a Chit".

     

    Include some up-to-date references (at least 2007 and above).

     

    So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine – that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have "some" post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).

     

    Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof.  Seriously – it's not that I'm a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but that's not the point), but it's that these reviews don't even address what I wrote. 

     

    Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, I'm concerned about introducing this level of noise into the review process.  How can the program chair make selections when the majority of data are at best useless, and at worst, wrong?  Does this selection process lead to a program that is most useful to Academy members? 

     

    This is not a slam at Barry Mitnick, the SIM division program chair.  If anything, it's a testament to his hard work and dedication, despite the troubled process.  I can't imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time.  Rather, it's a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.

     

    So I think we first need to decide if the current model is broken in a way that can't be fixed.  I think it is.  The problem is that we can't obtain an adequate pool of competent and engaged reviewers.  Therefore, we rely on the incompetent and disengaged, and to an increasing degree.  This is a problem for our major journals (see the recent pleas for reviewers), and so it's going to be even worse for SIM at Academy (can't even imagine the challenge at conferences with lower profiles).  I don't think there's a way to get up to adequate quality while retaining or even increasing quantity.  We could better screen potential reviewers (must have some basic qualifications – don't think we do this now), and we could disallow bad reviewers from reviewing again (don't think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.

     

    If we accept that the current system is broken and can't be fixed, then that frees us up to search for a new system.  I hope the executive committee will do just this.  What are some options?  Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat – 40% get spots.  I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews.  If the result is the same, while the efficiency is much greater, we should do this.

     

    Another option is to decrease the number of reviews required for each paper.  Two quality reviews would be better than three reviews where two are useless or wrong.  This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged. 

     

    A third option is to unblind the review process and allow selection, perhaps by a committee, based on record.  This has many obvious chicken-egg problems – how does one break in to the system? – but it would increase the likelihood of securing a quality program.  And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).

     

    A fourth option is to make all papers theme dependent.  The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldn't be accepted.  Reviewers would need only determine fit.

     

    I'm sure there are more options, or even combinations.  The things I've suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than we're doing now.

     

    Best,

    Mike 

     

    ********************

    Michael L. Barnett, PhD

    <st1:place w:st="on"><st1:city w:st="on">University of South</st1:city> <st1:state w:st="on">Florida</st1:state></st1:place>

    <st1:place w:st="on"><st1:placetype w:st="on">College</st1:placetype> of <st1:placename w:st="on">Business Administration</st1:placename></st1:place>

    Department of Management & Organization

    <st1:street w:st="on"><st1:address w:st="on">4202 E. Fowler Avenue</st1:address></st1:street>, BSN 3527

    <st1:place w:st="on"><st1:city w:st="on">Tampa</st1:city>, <st1:state w:st="on">FL</st1:state> <st1:postalcode w:st="on">33620-5500</st1:postalcode></st1:place>

    Phone: 813-974-1727

    Fax: 813-974-1734

     

    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>

     

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 12.  Is the Academy review process broken? What to do?

    Posted 03-23-2009 09:41

    I think Paul has a great idea here.  Though it certainly would not guarantee good reviews, it would ease the burden on reviewers considerably.  It would also ease the burden on authors in some ways.  Most of us are pressed to meet the deadline in early Jan.  Papers often have a few good sections but overall, many parts of the papers are clearly rushed.  It's like buying an album and getting 2/3rds filler in order to get to the few songs you do like.  Of course, no one buys albums anymore – the system has been improved so that you need only buy what you want.  Perhaps this is a fitting analogy – time to change our system to better reflect the realities of the day.  It means having to write an extended abstract, and so it could be more work for authors, but as Paul points out, it could also be developmental, forcing authors to hone their arguments.  The additional benefit is that you become freed to develop your paper in response to the feedback we hope this might generate.  Right now, when I submit a complete paper to Academy, I'm pretty far down a path and so the limited (if any) useful (and delayed) feedback I get may be too late to stomach (especially the feedback many responders have suggested, which comes only in August, 8 months after submission).  So, I think Paul's idea fires on a lot of cylinders.  It's simple, and it's common in other conferences, so there's plenty of precedent.  Shall we??  Anyone know why we haven't already (or why we've been doing it the full paper way for so long?).

     

    ***********************

    Michael L. Barnett, PhD

    <st1:place w:st="on"><st1:city w:st="on">University of South</st1:city> <st1:state w:st="on">Florida</st1:state></st1:place>

    <st1:place w:st="on"><st1:placetype w:st="on">College</st1:placetype> of <st1:placename w:st="on">Business Administration</st1:placename></st1:place>

    Department of Management & Organization

    <st1:street w:st="on"><st1:address w:st="on">4202 E. Fowler Avenue</st1:address></st1:street>, BSN 3213

    <st1:place w:st="on"><st1:city w:st="on">Tampa</st1:city>, <st1:state w:st="on">FL</st1:state> <st1:postalcode w:st="on">33620-5500</st1:postalcode></st1:place>

    Phone: 813-974-1727

    Fax: 813-974-1734

    E-mail: mbarnett@coba.usf.edu

    Webpage: http://www.coba.usf.edu/barnett

     

    View my research on my SSRN Author page:

    <http://ssrn.com/author=414796>

    **************************************************

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of <st1:personname w:st="on">Lee, Min Dong</st1:personname>
    Sent: Monday, March 23, 2009 9:14 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

     

    I am not an economist, but I do think Don's pricing model makes a sense if we can come up with pricing level that would not become a barrier for unfunded Ph.D. students (or maybe create an alternative system for students).  

    Another suggestion I have is to reduce the length of papers to present only the essence of the paper.  I was speaking with a friend in Physics recently, and he was astonished that our papers are often 40-50 pages long.  His words: "how can you possibly read all of them and reflect on them?"  Meaning if we spend too much time reading, we don't have enough time/energy/motivation to think....  Or, if we are pressed against time, then we simply do not read the papers very carefully and come up with just perfunctory comments.  As all of you know, good review takes time and good review on 40 pages takes very long time.  

    There is another advantage of limiting paper length. I believe writing a good short paper that just contains the essence would be more difficult than writing a longer paper.  And reviewing a very short paper for quality is probably easier, because authors cannot hide behind long quotes, lit. reviews and rhetoric etc. I tested this hypothesis with my class (student essays), and it seems to be true J   So, the burden of proof falls on authors, not reviewers.

    I also think that since we will hear the presentations, we really do not need the full paper.... Papers can be submitted to journals, not AOM conference.

    So, my suggestion is to limit the length of papers to something like 2000 words and allow reviewers more time to think.

    That's my 2 cents.

     

    Paul

     

    Min-Dong Paul Lee, Assistant Professor

    Department of Management & Organization

    <st1:place w:st="on"><st1:placetype w:st="on">College</st1:placetype> of <st1:placename w:st="on">Business</st1:placename></st1:place>

    <st1:place w:st="on"><st1:city w:st="on">University of South</st1:city> <st1:state w:st="on">Florida</st1:state></st1:place>

    <st1:street w:st="on"><st1:address w:st="on">4202 E. Flowler Ave.</st1:address></st1:street>, BSN 3403

    <st1:place w:st="on"><st1:city w:st="on">Tampa</st1:city>, <st1:state w:st="on">FL</st1:state>  <st1:postalcode w:st="on">33620</st1:postalcode></st1:place>

    (813) 974-1721

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Donald S Siegel
    Sent: Sunday, March 22, 2009 10:49 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

     

    HI Mike:

    It's always a pleasure to read your lucid and insightful comments on this list-serve.  I understand your frustration.  At the risk of sounding like a stereotypical economist,  I view this as a pricing problem.  With the proliferation of journals, increasing submissions to the AOM meetings,  and greater pressure to publish (tenure committees are often focused more on quantity than quality), it is difficult to find reviewers.  One solution is to raise the cost of submission, which is now zero, while simultaneously compensating reviewers (i.e., the revenue generated from submissions could be used to pay reviewers, with additional funds coming from the division).  In other fields, especially in economics and finance, journals pay reviewers for on-time reviews or offer to donate a certain sum of money to charity.  The latter option might be especially appealing to SIM members.

    Best regards,

    Don

     

    Dr. Donald S. Siegel

    Dean and Professor

    <st1:place w:st="on"><st1:placetype w:st="on">School</st1:placetype> of <st1:placename w:st="on">Business</st1:placename></st1:place>

    University at <st1:place w:st="on"><st1:city w:st="on">Albany</st1:city></st1:place>, SUNY

    <st1:street w:st="on"><st1:address w:st="on">1400 Washington Avenue</st1:address></st1:street>

    <st1:place w:st="on"><st1:city w:st="on">Albany</st1:city>, <st1:state w:st="on">NY</st1:state> <st1:postalcode w:st="on">12222</st1:postalcode></st1:place>

    Tel: (518) 442-4910

    DSiegel@uamail.albany.edu

    http://www.albany.edu/business/

    http://www.albany.edu/business/news_and_events/DonSiegel-CV.pdf

    http://econpapers.repec.org/RAS/psi32.htm

    http://ssrn.com/author=33607

     

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of <st1:personname w:st="on">Barnett, Michael</st1:personname>
    Sent: Sunday, March 22, 2009 10:29 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: [SIM] Is the Academy review process broken? What to do?

     

    Dear fellow SIMians:

     

    I think the review process for the <st1:place w:st="on"><st1:placetype w:st="on">Academy</st1:placetype> of <st1:placename w:st="on">Management</st1:placename></st1:place> conference might be broken, within SIM and perhaps beyond.  I'd like to see if others share my concern, and if it's not just me, I'd like to push the division, or the Academy in general, to do something to fix it. 

     

    This morning, I received notification that my paper was accepted at SIM.  That's good.  Then I read my three reviews – not good.  The first reviewer offered a reasonable analysis and a couple good suggestions.  It wasn't a journal quality review, but it was a solid review – basically, of the caliber we should probably expect from a conference review, given time constraints, etc. 

     

    Then there's the other two.  Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), here's the entirety of the sage feedback I received from the other two reviewers:

     

    R#2:

    Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.

     

    R#3:

    I have an issue with your title.  I think it is inappproriate for an academic conference.  I would leave off "Taking a Chit".

     

    Include some up-to-date references (at least 2007 and above).

     

    So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine – that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have "some" post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).

     

    Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof.  Seriously – it's not that I'm a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but that's not the point), but it's that these reviews don't even address what I wrote. 

     

    Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, I'm concerned about introducing this level of noise into the review process.  How can the program chair make selections when the majority of data are at best useless, and at worst, wrong?  Does this selection process lead to a program that is most useful to Academy members? 

     

    This is not a slam at Barry Mitnick, the SIM division program chair.  If anything, it's a testament to his hard work and dedication, despite the troubled process.  I can't imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time.  Rather, it's a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.

     

    So I think we first need to decide if the current model is broken in a way that can't be fixed.  I think it is.  The problem is that we can't obtain an adequate pool of competent and engaged reviewers.  Therefore, we rely on the incompetent and disengaged, and to an increasing degree.  This is a problem for our major journals (see the recent pleas for reviewers), and so it's going to be even worse for SIM at Academy (can't even imagine the challenge at conferences with lower profiles).  I don't think there's a way to get up to adequate quality while retaining or even increasing quantity.  We could better screen potential reviewers (must have some basic qualifications – don't think we do this now), and we could disallow bad reviewers from reviewing again (don't think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.

     

    If we accept that the current system is broken and can't be fixed, then that frees us up to search for a new system.  I hope the executive committee will do just this.  What are some options?  Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat – 40% get spots.  I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews.  If the result is the same, while the efficiency is much greater, we should do this.

     

    Another option is to decrease the number of reviews required for each paper.  Two quality reviews would be better than three reviews where two are useless or wrong.  This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged. 

     

    A third option is to unblind the review process and allow selection, perhaps by a committee, based on record.  This has many obvious chicken-egg problems – how does one break in to the system? – but it would increase the likelihood of securing a quality program.  And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).

     

    A fourth option is to make all papers theme dependent.  The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldn't be accepted.  Reviewers would need only determine fit.

     

    I'm sure there are more options, or even combinations.  The things I've suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than we're doing now.

     

    Best,

    Mike 

     

    ********************

    Michael L. Barnett, PhD

    <st1:place w:st="on"><st1:city w:st="on">University of South</st1:city> <st1:state w:st="on">Florida</st1:state></st1:place>

    <st1:place w:st="on"><st1:placetype w:st="on">College</st1:placetype> of <st1:placename w:st="on">Business Administration</st1:placename></st1:place>

    Department of Management & Organization

    <st1:street w:st="on"><st1:address w:st="on">4202 E. Fowler Avenue</st1:address></st1:street>, BSN 3527

    <st1:place w:st="on"><st1:city w:st="on">Tampa</st1:city>, <st1:state w:st="on">FL</st1:state> <st1:postalcode w:st="on">33620-5500</st1:postalcode></st1:place>

    Phone: 813-974-1727

    Fax: 813-974-1734

     

    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>

     

    _______________________________________________________________________

    To send a message to the list, send your email to <st1:personname w:st="on">SIM@aomlists.pace.edu</st1:personname>

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 13.  Is the Academy review process broken? What to do?

    Posted 03-23-2009 09:52

    I just quickly refreshed over the first three replies below.  I have a question and  it's very naïve.  I'm  a first year professor and I know that my comments for reviews are not ever good enough, but then I'm faced with two issues to overcome: 1) being a neophyte in the business and sometimes, I can barely understand some papers because of my lack of experience for which I'm supposedly qualified (review mismatch) and 2) the theory is way over my head.  So in order for me to make sense of a 40 page paper I have to read a lot of the lit review articles to get up to speed.  Then, because it's way past my level of expertise, how does one comment on improvements?  On some of the reviews that I did early on, I got paper that did not meet the  style guidelines.  Those I kicked to the curb as automatically unreadable.  45 pages of 10 point font is inappropriate.  But beyond basic formatting and grammar and style, some of the papers that I've read for the Academy leave me cold because they are so long that I can't invest.  I prefer to submit a five to ten page abstract or prospectus and then take comments for improvement of the final paper  that will be submitted to a journal.  In fact, the process is onerous to me that I prefer not to participate in the AoM for a while longer.  In the tenure review process, getting a paper accepted at AoM is the same as getting a paper accepted at other smaller conferences as long as they are peer reviewed we aren't too picky!  To submit elsewhere does not mean that a paper lacks quality or contribution, and I think too many people feel that AoM is the only publication in town that has quality and contribution. 

     

    No need to go through the headache of an AoM review unless I just want it for free and for fun to prepare for publication elsewhere.  The process is just onerous either as an author or reviewer.  I'm in favor of 20 page papers that get to the point, but there is always one nay-sayer that asks why a particular topic wasn't asked or covered when, indeed, in the longer paper it was.  Then that same visionless reviewer  might not recommend a paper based on the fact that things aren't covered...never asking if it could have been covered in the space allotted. Sigh, win-lose, all the way around.

     

    I have inflation-itis-

    My $.10

    Linda

    --

     

     

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Lee, Min Dong
    Sent: Monday, March 23, 2009 9:14 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

     

    I am not an economist, but I do think Don's pricing model makes a sense if we can come up with pricing level that would not become a barrier for unfunded Ph.D. students (or maybe create an alternative system for students).  

    Another suggestion I have is to reduce the length of papers to present only the essence of the paper.  I was speaking with a friend in Physics recently, and he was astonished that our papers are often 40-50 pages long.  His words: "how can you possibly read all of them and reflect on them?"  Meaning if we spend too much time reading, we don't have enough time/energy/motivation to think....  Or, if we are pressed against time, then we simply do not read the papers very carefully and come up with just perfunctory comments.  As all of you know, good review takes time and good review on 40 pages takes very long time.  

    There is another advantage of limiting paper length. I believe writing a good short paper that just contains the essence would be more difficult than writing a longer paper.  And reviewing a very short paper for quality is probably easier, because authors cannot hide behind long quotes, lit. reviews and rhetoric etc. I tested this hypothesis with my class (student essays), and it seems to be true J   So, the burden of proof falls on authors, not reviewers.

    I also think that since we will hear the presentations, we really do not need the full paper.... Papers can be submitted to journals, not AOM conference.

    So, my suggestion is to limit the length of papers to something like 2000 words and allow reviewers more time to think.

    That's my 2 cents.

     

    Paul

     

    Min-Dong Paul Lee, Assistant Professor

    Department of Management & Organization

    College of Business

    University of South Florida

    4202 E. Flowler Ave., BSN 3403

    Tampa, FL  33620

    (813) 974-1721

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Donald S Siegel
    Sent: Sunday, March 22, 2009 10:49 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

     

    HI Mike:

    It's always a pleasure to read your lucid and insightful comments on this list-serve.  I understand your frustration.  At the risk of sounding like a stereotypical economist,  I view this as a pricing problem.  With the proliferation of journals, increasing submissions to the AOM meetings,  and greater pressure to publish (tenure committees are often focused more on quantity than quality), it is difficult to find reviewers.  One solution is to raise the cost of submission, which is now zero, while simultaneously compensating reviewers (i.e., the revenue generated from submissions could be used to pay reviewers, with additional funds coming from the division).  In other fields, especially in economics and finance, journals pay reviewers for on-time reviews or offer to donate a certain sum of money to charity.  The latter option might be especially appealing to SIM members.

    Best regards,

    Don

     

    Dr. Donald S. Siegel

    Dean and Professor

    School of Business

    University at Albany, SUNY

    1400 Washington Avenue

    Albany, NY 12222

    Tel: (518) 442-4910

    DSiegel@uamail.albany.edu

    http://www.albany.edu/business/

    http://www.albany.edu/business/news_and_events/DonSiegel-CV.pdf

    http://econpapers.repec.org/RAS/psi32.htm

    http://ssrn.com/author=33607

     

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Barnett, Michael
    Sent: Sunday, March 22, 2009 10:29 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: [SIM] Is the Academy review process broken? What to do?

     

    Dear fellow SIMians:

     

    I think the review process for the Academy of Management conference might be broken, within SIM and perhaps beyond.  I'd like to see if others share my concern, and if it's not just me, I'd like to push the division, or the Academy in general, to do something to fix it. 

     

    This morning, I received notification that my paper was accepted at SIM.  That's good.  Then I read my three reviews – not good.  The first reviewer offered a reasonable analysis and a couple good suggestions.  It wasn't a journal quality review, but it was a solid review – basically, of the caliber we should probably expect from a conference review, given time constraints, etc. 

     

    Then there's the other two.  Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), here's the entirety of the sage feedback I received from the other two reviewers:

     

    R#2:

    Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.

     

    R#3:

    I have an issue with your title.  I think it is inappproriate for an academic conference.  I would leave off "Taking a Chit".

     

    Include some up-to-date references (at least 2007 and above).

     

    So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine – that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have "some" post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).

     

    Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof.  Seriously – it's not that I'm a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but that's not the point), but it's that these reviews don't even address what I wrote. 

     

    Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, I'm concerned about introducing this level of noise into the review process.  How can the program chair make selections when the majority of data are at best useless, and at worst, wrong?  Does this selection process lead to a program that is most useful to Academy members? 

     

    This is not a slam at Barry Mitnick, the SIM division program chair.  If anything, it's a testament to his hard work and dedication, despite the troubled process.  I can't imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time.  Rather, it's a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.

     

    So I think we first need to decide if the current model is broken in a way that can't be fixed.  I think it is.  The problem is that we can't obtain an adequate pool of competent and engaged reviewers.  Therefore, we rely on the incompetent and disengaged, and to an increasing degree.  This is a problem for our major journals (see the recent pleas for reviewers), and so it's going to be even worse for SIM at Academy (can't even imagine the challenge at conferences with lower profiles).  I don't think there's a way to get up to adequate quality while retaining or even increasing quantity.  We could better screen potential reviewers (must have some basic qualifications – don't think we do this now), and we could disallow bad reviewers from reviewing again (don't think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.

     

    If we accept that the current system is broken and can't be fixed, then that frees us up to search for a new system.  I hope the executive committee will do just this.  What are some options?  Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat – 40% get spots.  I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews.  If the result is the same, while the efficiency is much greater, we should do this.

     

    Another option is to decrease the number of reviews required for each paper.  Two quality reviews would be better than three reviews where two are useless or wrong.  This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged. 

     

    A third option is to unblind the review process and allow selection, perhaps by a committee, based on record.  This has many obvious chicken-egg problems – how does one break in to the system? – but it would increase the likelihood of securing a quality program.  And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).

     

    A fourth option is to make all papers theme dependent.  The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldn't be accepted.  Reviewers would need only determine fit.

     

    I'm sure there are more options, or even combinations.  The things I've suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than we're doing now.

     

    Best,

    Mike 

     

    ********************

    Michael L. Barnett, PhD

    University of South Florida

    College of Business Administration

    Department of Management & Organization

    4202 E. Fowler Avenue, BSN 3527

    Tampa, FL 33620-5500

    Phone: 813-974-1727

    Fax: 813-974-1734

     

    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>

     

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 14.  Is the Academy review process broken? What to do?

    Posted 03-23-2009 10:26
    Hi Paul and Mike,
     
    I think your suggestions on a 2000 words paper for AOM might be more relevant to theoretical and quantitative papers. But, hey, there are some managament scholars, like myself, who do qualitative research too. Have you forgotten your next door neighbor and others like me?! :-)
     
    Life will be too tough if qualitative researchers are expected to submit a 2000 words paper. In my research, I am constantly struggling to mantain the right balance between presenting the theoretical reviews and findings, and theorizing based on my analysis. Reviewers of qualitative papers want to see quotes (AND MORE QUOTES) that support our findings. If we do not provide enough quotes, then they start thinking our analysis is weak and our findings not reliable. When we provide more quotes, then the paper length goes up. And that is because, our research subjects, unfortunately, don't convey (or know how to convey) their thoughts in less than 20 words, either when they are speaking with us during interviews, or engaging in dialogue with others that we record during meetings.
     
    Junior scholars like myself want to submit our work to the Academy because we want to get some critical reviews on the actual paper which we plan to submit to journals in the future. Submitting a condensed version may not be very helpful. I think those colleagues who do not have time to review should not sign up to be a reviewer at the first place. It is a voluntary service. I also think that reviewing is a skill which scholars develop over a period of time. The more we publish/ get revise and resubmits and/or rejects, the better we get at reviewing other's work too. So, let's also give reviewers opportunity to develop their skill.
     
    Sincerely,
    Your junior qualitative colleague, Aarti :-)
     
     
     
     
     
     
    Aarti Sharma, Ph.D.
    Instructor of Strategic Management
    Department of Management & Organization
    College of Business Administration
    University of South Florida
    4202 E. Fowler Ave., BSN-3524
    Tampa, FL-33620-5500
    Tel: 813-974-4354
    Fax: 813- 974-1734
    Email: asharma@coba.usf.edu
    Web: http://www.coba.usf.edu/departments/management/faculty/sharma/index.html
     

    From: Social Issues in Management Listserv on behalf of Barnett, Michael
    Sent: Mon 3/23/2009 9:41 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

    I think Paul has a great idea here.  Though it certainly would not guarantee good reviews, it would ease the burden on reviewers considerably.  It would also ease the burden on authors in some ways.  Most of us are pressed to meet the deadline in early Jan.  Papers often have a few good sections but overall, many parts of the papers are clearly rushed.  It's like buying an album and getting 2/3rds filler in order to get to the few songs you do like.  Of course, no one buys albums anymore – the system has been improved so that you need only buy what you want.  Perhaps this is a fitting analogy – time to change our system to better reflect the realities of the day.  It means having to write an extended abstract, and so it could be more work for authors, but as Paul points out, it could also be developmental, forcing authors to hone their arguments.  The additional benefit is that you become freed to develop your paper in response to the feedback we hope this might generate.  Right now, when I submit a complete paper to Academy, I'm pretty far down a path and so the limited (if any) useful (and delayed) feedback I get may be too late to stomach (especially the feedback many responders have suggested, which comes only in August, 8 months after submission).  So, I think Paul's idea fires on a lot of cylinders.  It's simple, and it's common in other conferences, so there's plenty of precedent.  Shall we??  Anyone know why we haven't already (or why we've been doing it the full paper way for so long?).

     

    ***********************

    Michael L. Barnett, PhD

    University of South Florida

    College of Business Administration

    Department of Management & Organization

    4202 E. Fowler Avenue, BSN 3213

    Tampa, FL 33620-5500

    Phone: 813-974-1727

    Fax: 813-974-1734

    E-mail: mbarnett@coba.usf.edu

    Webpage: http://www.coba.usf.edu/barnett

     

    View my research on my SSRN Author page:

    <http://ssrn.com/author=414796>

    **************************************************

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Lee, Min Dong
    Sent: Monday, March 23, 2009 9:14 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

     

    I am not an economist, but I do think Don's pricing model makes a sense if we can come up with pricing level that would not become a barrier for unfunded Ph.D. students (or maybe create an alternative system for students).  

    Another suggestion I have is to reduce the length of papers to present only the essence of the paper.  I was speaking with a friend in Physics recently, and he was astonished that our papers are often 40-50 pages long.  His words: "how can you possibly read all of them and reflect on them?"  Meaning if we spend too much time reading, we don't have enough time/energy/motivation to think....  Or, if we are pressed against time, then we simply do not read the papers very carefully and come up with just perfunctory comments.  As all of you know, good review takes time and good review on 40 pages takes very long time.  

    There is another advantage of limiting paper length. I believe writing a good short paper that just contains the essence would be more difficult than writing a longer paper.  And reviewing a very short paper for quality is probably easier, because authors cannot hide behind long quotes, lit. reviews and rhetoric etc. I tested this hypothesis with my class (student essays), and it seems to be true J   So, the burden of proof falls on authors, not reviewers.

    I also think that since we will hear the presentations, we really do not need the full paper.... Papers can be submitted to journals, not AOM conference.

    So, my suggestion is to limit the length of papers to something like 2000 words and allow reviewers more time to think.

    That's my 2 cents.

     

    Paul

     

    Min-Dong Paul Lee, Assistant Professor

    Department of Management & Organization

    College of Business

    University of South Florida

    4202 E. Flowler Ave., BSN 3403

    Tampa, FL  33620

    (813) 974-1721

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Donald S Siegel
    Sent: Sunday, March 22, 2009 10:49 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

     

    HI Mike:

    It's always a pleasure to read your lucid and insightful comments on this list-serve.  I understand your frustration.  At the risk of sounding like a stereotypical economist,  I view this as a pricing problem.  With the proliferation of journals, increasing submissions to the AOM meetings,  and greater pressure to publish (tenure committees are often focused more on quantity than quality), it is difficult to find reviewers.  One solution is to raise the cost of submission, which is now zero, while simultaneously compensating reviewers (i.e., the revenue generated from submissions could be used to pay reviewers, with additional funds coming from the division).  In other fields, especially in economics and finance, journals pay reviewers for on-time reviews or offer to donate a certain sum of money to charity.  The latter option might be especially appealing to SIM members.

    Best regards,

    Don

     

    Dr. Donald S. Siegel

    Dean and Professor

    School of Business

    University at Albany, SUNY

    1400 Washington Avenue

    Albany, NY 12222

    Tel: (518) 442-4910

    DSiegel@uamail.albany.edu

    http://www.albany.edu/business/

    http://www.albany.edu/business/news_and_events/DonSiegel-CV.pdf

    http://econpapers.repec.org/RAS/psi32.htm

    http://ssrn.com/author=33607

     

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Barnett, Michael
    Sent: Sunday, March 22, 2009 10:29 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: [SIM] Is the Academy review process broken? What to do?

     

    Dear fellow SIMians:

     

    I think the review process for the Academy of Management conference might be broken, within SIM and perhaps beyond.  I'd like to see if others share my concern, and if it's not just me, I'd like to push the division, or the Academy in general, to do something to fix it. 

     

    This morning, I received notification that my paper was accepted at SIM.  That's good.  Then I read my three reviews – not good.  The first reviewer offered a reasonable analysis and a couple good suggestions.  It wasn't a journal quality review, but it was a solid review – basically, of the caliber we should probably expect from a conference review, given time constraints, etc. 

     

    Then there's the other two.  Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), here's the entirety of the sage feedback I received from the other two reviewers:

     

    R#2:

    Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.

     

    R#3:

    I have an issue with your title.  I think it is inappproriate for an academic conference.  I would leave off "Taking a Chit".

     

    Include some up-to-date references (at least 2007 and above).

     

    So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine – that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have "some" post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).

     

    Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof.  Seriously – it's not that I'm a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but that's not the point), but it's that these reviews don't even address what I wrote. 

     

    Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, I'm concerned about introducing this level of noise into the review process.  How can the program chair make selections when the majority of data are at best useless, and at worst, wrong?  Does this selection process lead to a program that is most useful to Academy members? 

     

    This is not a slam at Barry Mitnick, the SIM division program chair.  If anything, it's a testament to his hard work and dedication, despite the troubled process.  I can't imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time.  Rather, it's a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.

     

    So I think we first need to decide if the current model is broken in a way that can't be fixed.  I think it is.  The problem is that we can't obtain an adequate pool of competent and engaged reviewers.  Therefore, we rely on the incompetent and disengaged, and to an increasing degree.  This is a problem for our major journals (see the recent pleas for reviewers), and so it's going to be even worse for SIM at Academy (can't even imagine the challenge at conferences with lower profiles).  I don't think there's a way to get up to adequate quality while retaining or even increasing quantity.  We could better screen potential reviewers (must have some basic qualifications – don't think we do this now), and we could disallow bad reviewers from reviewing again (don't think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.

     

    If we accept that the current system is broken and can't be fixed, then that frees us up to search for a new system.  I hope the executive committee will do just this.  What are some options?  Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat – 40% get spots.  I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews.  If the result is the same, while the efficiency is much greater, we should do this.

     

    Another option is to decrease the number of reviews required for each paper.  Two quality reviews would be better than three reviews where two are useless or wrong.  This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged. 

     

    A third option is to unblind the review process and allow selection, perhaps by a committee, based on record.  This has many obvious chicken-egg problems – how does one break in to the system? – but it would increase the likelihood of securing a quality program.  And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).

     

    A fourth option is to make all papers theme dependent.  The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldn't be accepted.  Reviewers would need only determine fit.

     

    I'm sure there are more options, or even combinations.  The things I've suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than we're doing now.

     

    Best,

    Mike 

     

    ********************

    Michael L. Barnett, PhD

    University of South Florida

    College of Business Administration

    Department of Management & Organization

    4202 E. Fowler Avenue, BSN 3527

    Tampa, FL 33620-5500

    Phone: 813-974-1727

    Fax: 813-974-1734

     

    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>

     

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 15.  Is the Academy review process broken? What to do?

    Posted 03-23-2009 15:54
    Linda,
    Just a couple of thoughts.
    When you receive a submission that does not meet submission guidelines, you should quickly notify the Program Chair and return the paper unreviewed.  Such a paper should not be entered in the review process.
     
    Also, if your assessment is that you are not competent to review a paper (without having to read massive amounts) you should again notify the Program Chair and return the paper.
     
     

    Linda K. Trevino

    Distinguished Professor of Organizational Behavior and Ethics

    Smeal College of Business

    402 Business Building

    Smeal College of Business

    The Pennsylvania State University

    University Park, PA  16802

    Phone: 814-865-2194  Fax: 814-863-7261

    Email: ltrevino@psu.edu

    From: Social Issues in Management Listserv [SIM@AOMLISTS.PACE.EDU] On Behalf Of Linda C. Rodriguez [blondie@pobox.com]
    Sent: Monday, March 23, 2009 9:51 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

    I just quickly refreshed over the first three replies below.  I have a question and  it's very naïve.  I'm  a first year professor and I know that my comments for reviews are not ever good enough, but then I'm faced with two issues to overcome: 1) being a neophyte in the business and sometimes, I can barely understand some papers because of my lack of experience for which I'm supposedly qualified (review mismatch) and 2) the theory is way over my head.  So in order for me to make sense of a 40 page paper I have to read a lot of the lit review articles to get up to speed.  Then, because it's way past my level of expertise, how does one comment on improvements?  On some of the reviews that I did early on, I got paper that did not meet the  style guidelines.  Those I kicked to the curb as automatically unreadable.  45 pages of 10 point font is inappropriate.  But beyond basic formatting and grammar and style, some of the papers that I've read for the Academy leave me cold because they are so long that I can't invest.  I prefer to submit a five to ten page abstract or prospectus and then take comments for improvement of the final paper  that will be submitted to a journal.  In fact, the process is onerous to me that I prefer not to participate in the AoM for a while longer.  In the tenure review process, getting a paper accepted at AoM is the same as getting a paper accepted at other smaller conferences as long as they are peer reviewed we aren't too picky!  To submit elsewhere does not mean that a paper lacks quality or contribution, and I think too many people feel that AoM is the only publication in town that has quality and contribution. 

     

    No need to go through the headache of an AoM review unless I just want it for free and for fun to prepare for publication elsewhere.  The process is just onerous either as an author or reviewer.  I'm in favor of 20 page papers that get to the point, but there is always one nay-sayer that asks why a particular topic wasn't asked or covered when, indeed, in the longer paper it was.  Then that same visionless reviewer  might not recommend a paper based on the fact that things aren't covered...never asking if it could have been covered in the space allotted. Sigh, win-lose, all the way around.

     

    I have inflation-itis-

    My $.10

    Linda

    --

     

     

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Lee, Min Dong
    Sent: Monday, March 23, 2009 9:14 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

     

    I am not an economist, but I do think Don's pricing model makes a sense if we can come up with pricing level that would not become a barrier for unfunded Ph.D. students (or maybe create an alternative system for students).  

    Another suggestion I have is to reduce the length of papers to present only the essence of the paper.  I was speaking with a friend in Physics recently, and he was astonished that our papers are often 40-50 pages long.  His words: "how can you possibly read all of them and reflect on them?"  Meaning if we spend too much time reading, we don't have enough time/energy/motivation to think....  Or, if we are pressed against time, then we simply do not read the papers very carefully and come up with just perfunctory comments.  As all of you know, good review takes time and good review on 40 pages takes very long time.  

    There is another advantage of limiting paper length. I believe writing a good short paper that just contains the essence would be more difficult than writing a longer paper.  And reviewing a very short paper for quality is probably easier, because authors cannot hide behind long quotes, lit. reviews and rhetoric etc. I tested this hypothesis with my class (student essays), and it seems to be true J   So, the burden of proof falls on authors, not reviewers.

    I also think that since we will hear the presentations, we really do not need the full paper.... Papers can be submitted to journals, not AOM conference.

    So, my suggestion is to limit the length of papers to something like 2000 words and allow reviewers more time to think.

    That's my 2 cents.

     

    Paul

     

    Min-Dong Paul Lee, Assistant Professor

    Department of Management & Organization

    College of Business

    University of South Florida

    4202 E. Flowler Ave., BSN 3403

    Tampa, FL  33620

    (813) 974-1721

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Donald S Siegel
    Sent: Sunday, March 22, 2009 10:49 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: Re: [SIM] Is the Academy review process broken? What to do?

     

    HI Mike:

    It's always a pleasure to read your lucid and insightful comments on this list-serve.  I understand your frustration.  At the risk of sounding like a stereotypical economist,  I view this as a pricing problem.  With the proliferation of journals, increasing submissions to the AOM meetings,  and greater pressure to publish (tenure committees are often focused more on quantity than quality), it is difficult to find reviewers.  One solution is to raise the cost of submission, which is now zero, while simultaneously compensating reviewers (i.e., the revenue generated from submissions could be used to pay reviewers, with additional funds coming from the division).  In other fields, especially in economics and finance, journals pay reviewers for on-time reviews or offer to donate a certain sum of money to charity.  The latter option might be especially appealing to SIM members.

    Best regards,

    Don

     

    Dr. Donald S. Siegel

    Dean and Professor

    School of Business

    University at Albany, SUNY

    1400 Washington Avenue

    Albany, NY 12222

    Tel: (518) 442-4910

    DSiegel@uamail.albany.edu

    http://www.albany.edu/business/

    http://www.albany.edu/business/news_and_events/DonSiegel-CV.pdf

    http://econpapers.repec.org/RAS/psi32.htm

    http://ssrn.com/author=33607

     

    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Barnett, Michael
    Sent: Sunday, March 22, 2009 10:29 AM
    To: SIM@AOMLISTS.PACE.EDU
    Subject: [SIM] Is the Academy review process broken? What to do?

     

    Dear fellow SIMians:

     

    I think the review process for the Academy of Management conference might be broken, within SIM and perhaps beyond.  I'd like to see if others share my concern, and if it's not just me, I'd like to push the division, or the Academy in general, to do something to fix it. 

     

    This morning, I received notification that my paper was accepted at SIM.  That's good.  Then I read my three reviews – not good.  The first reviewer offered a reasonable analysis and a couple good suggestions.  It wasn't a journal quality review, but it was a solid review – basically, of the caliber we should probably expect from a conference review, given time constraints, etc. 

     

    Then there's the other two.  Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), here's the entirety of the sage feedback I received from the other two reviewers:

     

    R#2:

    Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.

     

    R#3:

    I have an issue with your title.  I think it is inappproriate for an academic conference.  I would leave off "Taking a Chit".

     

    Include some up-to-date references (at least 2007 and above).

     

    So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine – that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have "some" post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).

     

    Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof.  Seriously – it's not that I'm a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but that's not the point), but it's that these reviews don't even address what I wrote. 

     

    Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, I'm concerned about introducing this level of noise into the review process.  How can the program chair make selections when the majority of data are at best useless, and at worst, wrong?  Does this selection process lead to a program that is most useful to Academy members? 

     

    This is not a slam at Barry Mitnick, the SIM division program chair.  If anything, it's a testament to his hard work and dedication, despite the troubled process.  I can't imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time.  Rather, it's a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.

     

    So I think we first need to decide if the current model is broken in a way that can't be fixed.  I think it is.  The problem is that we can't obtain an adequate pool of competent and engaged reviewers.  Therefore, we rely on the incompetent and disengaged, and to an increasing degree.  This is a problem for our major journals (see the recent pleas for reviewers), and so it's going to be even worse for SIM at Academy (can't even imagine the challenge at conferences with lower profiles).  I don't think there's a way to get up to adequate quality while retaining or even increasing quantity.  We could better screen potential reviewers (must have some basic qualifications – don't think we do this now), and we could disallow bad reviewers from reviewing again (don't think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.

     

    If we accept that the current system is broken and can't be fixed, then that frees us up to search for a new system.  I hope the executive committee will do just this.  What are some options?  Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat – 40% get spots.  I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews.  If the result is the same, while the efficiency is much greater, we should do this.

     

    Another option is to decrease the number of reviews required for each paper.  Two quality reviews would be better than three reviews where two are useless or wrong.  This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged. 

     

    A third option is to unblind the review process and allow selection, perhaps by a committee, based on record.  This has many obvious chicken-egg problems – how does one break in to the system? – but it would increase the likelihood of securing a quality program.  And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).

     

    A fourth option is to make all papers theme dependent.  The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldn't be accepted.  Reviewers would need only determine fit.

     

    I'm sure there are more options, or even combinations.  The things I've suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than we're doing now.

     

    Best,

    Mike 

     

    ********************

    Michael L. Barnett, PhD

    University of South Florida

    College of Business Administration

    Department of Management & Organization

    4202 E. Fowler Avenue, BSN 3527

    Tampa, FL 33620-5500

    Phone: 813-974-1727

    Fax: 813-974-1734

     

    View my research on my SSRN Author page:
    <http://ssrn.com/author=414796>

     

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________



  • 16.  Is the Academy review process broken? What to do?

    Posted 03-23-2009 16:21

    This is all a bit "horses for courses'.  Many conferences use a 1/2, 1/4 paper model and they have positives and negatives.  The positive is that it reduces the burden on the reviewers and can speed up the process.  I have run conferences and served as track chair for those doing each. However, the negative is that your ability to differentiate is reduced significantly.  Indeed, it would be my anecdotal argument that the quality of the presentations declines with the length of the submission (within the submission processes I know).  Here is where the problems arise.

    1.  The shorter the paper the less likely that key ideas can be presented.  Hence, as noted, empirical papers are more easily accepted since a table of results is easier to include that long complicated qualitative inferences.
    2.  As there is less distinguishing between papers the likelihood that a good jnr person can show themselves as being an up and comer is harder.  Given that people quickly recognize that the quality of submissions is lower, there is a tendency to focus on established scholars.  Although the reviewers do not know this the organizers do and will make biased judgements.  Similarly conferences with short submissions have a reputation that leads people to want to see the usual suspects (since they know what they are getting).
    3.  If you don't have a paper ready, why are you bothering?  I hate to say it but if your don't have a paper ready why bother even submitting to the conference?  Would you really much rather be in Anaheim?  I do not write papers for conferences.  Never had, never will.  I might use a conference deadline as a motivator for something I am doing.  But who really cares?  I know that your university won't pay if you don't have a paper but from a career standpoint whether you are at every conference or not really makes no difference.  No amount of conference proceedings and attendance substitutes for the real thing (some may have seen my comment before that the correlation between proceedings and publications is low.  The correlation even between conference best papers and ultimate publication is also very low).
    4.  Isn't it a waste of time writing a paper just for a conference when the output of that process is going nowhere?  For example, I never write a specific proposal to a specific conference unless there is a very compelling reason and it is generally a real waste of time cutting down a 30 page paper into a format just to have it for a conference proceedings volume no one is going to read anyway (this is our dirty little secret).  The only value I see is for PhD students and others where you can use it as a motivator to get them to put down their ideas in a succinct way.  When I do bite the bullet and do something for a conference there are two motivations.  One is that the idea is new and this is a way of testing it (but are there really 6,000 new ideas out there every year?) and/or the conference is in some exotic location (not New Orleans, Chicago, or Anaheim but Milan, Paris or Beijing).
    5.  There is a real quality problem with all conferences.  Another way of reducing the burden on reviewers is to use the Informs model (where nearly everything gets accepted based on an abstract) or to go the other way and simply reduced the size of the overall conference.  Informs has a reputation for your never knowing what you are going to get and some sessions have nothing but presenters.  The other route would imply that you really only accept things that are of some journal quality.  In the case of AOM and my experience as a program chair that would reduce your acceptance rate to probably 20%.  You could even go so far as letting the conference chair make the decisions based on one review.

    My point is that each of these systems has pluses and minuses.  You want the plus, you take the minus.



    Prof. Timothy Devinney

    Global Numbers: +61 2 8006 0048 or +1 412 5677440

    Other Numbers:
    Australia (Mob): +61 412 276 467
    USA (Mob): +1 (847) 207 3202
    Germany (Mob): +49 (151) 5316 7974
    China (Mob): +86 (136) 5116 1087
    Email: T.Devinney@unsw.edu.au

    Web 1 (Academic):     http://www.agsm.edu.au/tdevinney
    Web 2 (China Related):     http://vincitveritas.spaces.live.com/
    Flickr:     http://Flickr.com ... Search for the "tag" Timothy_Devinney or use http://flickr.com/search/people/?q=t.devinney
    "Lee, Min Dong" <mdlee@coba.usf.edu>
    Sent by: Social Issues in Management Listserv <SIM@AOMLISTS.PACE.EDU>

    24/03/2009 12:14 AM

    Please respond to
    "Lee, Min Dong" <mdlee@coba.usf.edu>
    To
    		SIM@AOMLISTS.PACE.EDU
    cc
    Subject
    		Re: [SIM] Is the Academy review process broken?  What to do?





    I am not an economist, but I do think Don's pricing model makes a sense if we can come up with pricing level that would not become a barrier for unfunded Ph.D. students (or maybe create an alternative system for students).  
    Another suggestion I have is to reduce the length of papers to present only the essence of the paper.  I was speaking with a friend in Physics recently, and he was astonished that our papers are often 40-50 pages long.  His words: "how can you possibly read all of them and reflect on them?"  Meaning if we spend too much time reading, we don't have enough time/energy/motivation to think....  Or, if we are pressed against time, then we simply do not read the papers very carefully and come up with just perfunctory comments.  As all of you know, good review takes time and good review on 40 pages takes very long time.  
    There is another advantage of limiting paper length. I believe writing a good short paper that just contains the essence would be more difficult than writing a longer paper.  And reviewing a very short paper for quality is probably easier, because authors cannot hide behind long quotes, lit. reviews and rhetoric etc. I tested this hypothesis with my class (student essays), and it seems to be true J   So, the burden of proof falls on authors, not reviewers.
    I also think that since we will hear the presentations, we really do not need the full paper.... Papers can be submitted to journals, not AOM conference.
    So, my suggestion is to limit the length of papers to something like 2000 words and allow reviewers more time to think.
    That's my 2 cents.
     
    Paul
     
    Min-Dong Paul Lee, Assistant Professor
    Department of Management & Organization
    College of Business
    University of South Florida
    4202 E. Flowler Ave., BSN 3403
    Tampa, FL  33620
    (813) 974-1721


    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Donald S Siegel
    Sent:     Sunday, March 22, 2009 10:49 AM
    To:     SIM@AOMLISTS.PACE.EDU
    Subject:     Re: [SIM] Is the Academy review process broken? What to do?
     
    HI Mike:
    It's always a pleasure to read your lucid and insightful comments on this list-serve.  I understand your frustration.  At the risk of sounding like a stereotypical economist,  I view this as a pricing problem.  With the proliferation of journals, increasing submissions to the AOM meetings,  and greater pressure to publish (tenure committees are often focused more on quantity than quality), it is difficult to find reviewers.  One solution is to raise the cost of submission, which is now zero, while simultaneously compensating reviewers (i.e., the revenue generated from submissions could be used to pay reviewers, with additional funds coming from the division).  In other fields, especially in economics and finance, journals pay reviewers for on-time reviews or offer to donate a certain sum of money to charity.  The latter option might be especially appealing to SIM members.
    Best regards,
    Don
     
    Dr. Donald S. Siegel
    Dean and Professor
    School of Business
    University at Albany, SUNY
    1400 Washington Avenue
    Albany, NY 12222
    Tel: (518) 442-4910
    DSiegel@uamail.albany.edu
    http://www.albany.edu/business/
    http://www.albany.edu/business/news_and_events/DonSiegel-CV.pdf
    http://econpapers.repec.org/RAS/psi32.htm
    http://ssrn.com/author=33607
     
    From: Social Issues in Management Listserv [mailto:SIM@AOMLISTS.PACE.EDU] On Behalf Of Barnett, Michael
    Sent:     Sunday, March 22, 2009 10:29 AM
    To:     SIM@AOMLISTS.PACE.EDU
    Subject:     [SIM] Is the Academy review process broken? What to do?
     
    Dear fellow SIMians:
     
    I think the review process for the Academy of Management conference might be broken, within SIM and perhaps beyond.  I'd like to see if others share my concern, and if it's not just me, I'd like to push the division, or the Academy in general, to do something to fix it.  
     
    This morning, I received notification that my paper was accepted at SIM.  That's good.  Then I read my three reviews – not good.  The first reviewer offered a reasonable analysis and a couple good suggestions.  It wasn't a journal quality review, but it was a solid review – basically, of the caliber we should probably expect from a conference review, given time constraints, etc.  
     
    Then there's the other two.  Bearing in mind that the paper I submitted is a theory paper and has an intentionally edgy title designed to play off the existing literature and catch attention (Take a chit: Cognitive constraints on stakeholder response to corporate misconduct), here's the entirety of the sage feedback I received from the other two reviewers:
     
    R#2:
    Authors provide an interesting perspective on why CSR efforts may/often fall short. Paper would make an interesting conference discussion, and I would be interested to see the authors developing testable hypotheses from the propositions and collecting data.
     
    R#3:
    I have an issue with your title.  I think it is inappproriate for an academic conference.  I would leave off "Taking a Chit".
     
    Include some up-to-date references (at least 2007 and above).
     
    So, R#2 appears unfamiliar with the concept of a theory paper, mistaking it for a research proposal (not that uncommon outside our world, but we should be getting reviews by people in our world), and R#3 focuses on the title (fine – that was expected) and takes issue with references being from earlier than the prior year (this was submitted at the end of 2008; obviously written prior to that; also, I suspect I did have "some" post 2006 cites; and interestingly, R#1 rightfully suggests I add some cites from several decades prior).
     
    Two-thirds of my reviewers were disinterested, incompetent, or some combination thereof.  Seriously – it's not that I'm a prima donna author who is angered that others have not fully appreciated my written genius (maybe I am but that's not the point), but it's that these reviews don't even address what I wrote.  
     
    Beyond my anger that these two reviewers, for whatever it is worth to them, will be allowed to claim to have engaged in service to the profession, I'm concerned about introducing this level of noise into the review process.  How can the program chair make selections when the majority of data are at best useless, and at worst, wrong?  Does this selection process lead to a program that is most useful to Academy members?  
     
    This is not a slam at Barry Mitnick, the SIM division program chair.  If anything, it's a testament to his hard work and dedication, despite the troubled process.  I can't imagine the headaches he has had, trying to sort information from noise, incompetence from earnestness, and so on, all on a grand scale and with limited time.  Rather, it's a call to change the process so that it becomes more useful, and perhaps efficient, if at all possible.
     
    So I think we first need to decide if the current model is broken in a way that can't be fixed.  I think it is.  The problem is that we can't obtain an adequate pool of competent and engaged reviewers.  Therefore, we rely on the incompetent and disengaged, and to an increasing degree.  This is a problem for our major journals (see the recent pleas for reviewers), and so it's going to be even worse for SIM at Academy (can't even imagine the challenge at conferences with lower profiles).  I don't think there's a way to get up to adequate quality while retaining or even increasing quantity.  We could better screen potential reviewers (must have some basic qualifications – don't think we do this now), and we could disallow bad reviewers from reviewing again (don't think we do this either), but we seem to need any semi-warm body we can find if we are to have the numbers.
     
    If we accept that the current system is broken and can't be fixed, then that frees us up to search for a new system.  I hope the executive committee will do just this.  What are some options?  Most cynically, since the current system introduces a great deal of randomness, we might as well draw names from a hat – 40% get spots.  I seriously believe the outcome will approximate the current process in a few short years, as the reviewer pool further degrades under increased demand for reviews.  If the result is the same, while the efficiency is much greater, we should do this.
     
    Another option is to decrease the number of reviews required for each paper.  Two quality reviews would be better than three reviews where two are useless or wrong.  This would allow for more stringent screening of reviewers, enabling us to leave out more of the incompetent and disengaged.  
     
    A third option is to unblind the review process and allow selection, perhaps by a committee, based on record.  This has many obvious chicken-egg problems – how does one break in to the system? – but it would increase the likelihood of securing a quality program.  And if high status folks come to abuse their ability to get in by offering up bad papers, they could be barred from future seating based on record and instead have to go through a more intensive selection process (so this process could also degrade in time).
     
    A fourth option is to make all papers theme dependent.  The committee could establish a set of topics the prior year, and then papers would need to fit with one of the themes, else it wouldn't be accepted.  Reviewers would need only determine fit.
     
    I'm sure there are more options, or even combinations.  The things I've suggested are off the top of my head. I hope we can put more of our heads together, so that we can do better than we're doing now.
     
    Best,
    Mike  
     
    ********************
    Michael L. Barnett, PhD
    University of South Florida
    College of Business Administration
    Department of Management & Organization
    4202 E. Fowler Avenue, BSN 3527
    Tampa, FL 33620-5500
    Phone: 813-974-1727
    Fax: 813-974-1734
    Webpage: http://www.coba.usf.edu/barnett
     
    View my research on my SSRN Author page:
    <    http://ssrn.com/author=414796>
     
    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________

    _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________ _______________________________________________________________________

    To send a message to the list, send your email to SIM@aomlists.pace.edu

    _______________________________________________________________________

    Visit the SIM Division website at: http://sim.aomonline.org _______________________________________________________________________

    If you wish to unsubscribe from this list or change your delivery options, you can do so online at: http://aomlists.pace.edu/scripts/wa.exe?SUBED1=sim&A=1 _______________________________________________________________________