© evileditor.blogspot.com.au
Perhaps it’s just that I’ve been at this for a while, or maybe it’s a real trend. Regardless, many of my colleagues and I are now of the opinion that the quality of editing in scientific journals is on the downhill slide.
Yes – we (scientists) all complain about negative decisions from journals to which we’ve submitted our work. Being rejected is part of the process. Aiming high is necessary for academic success, but when a negative decision is made on the basis of (often one) appalling review, it’s a little harder to swallow.
I suppose I can accept the inevitability of declining review quality for the simple reason that there are now SO MANY papers to review that finding willing volunteers is difficult. This means that there will always be people who only glance cursorily at the paper, miss the detail and recommend rejection based on their own misunderstanding or bias. It’s far easier to skim a paper and try to find a reason to reject than actually putting in the time to appraise the work critically and fairly.
This means that the traditional model of basing the decision to accept or reject a manuscript on only two reviews is fraught because the probability of receiving poor reviews is rising. For example, a certain undisclosed journal of unquestionably high quality for which I edit does not accept anything less than six recommendations for reviewers per manuscript, and none that I’m aware of is accepted or rejected based on only two reviews. But I think this is the exception rather than the rule – there are simply too many journals now of low to medium quality to be able to get that many reviewers to agree to review.
I won’t spend too much time trying to encourage you to do the best job you can when reviewing – that should go without saying. Remember what goes around comes around. If you are a shit reviewer, you will receive shit reviews.
As you move up the career ladder, however, you will be asked more frequently to become part of a journal’s editorial board. This is a noble thing to do in its own right, but it also gives you much more insight into the publication process. While time-consuming, it’s generally worth the effort (to a degree, of course).
If you become or already are an editor, then this post is for you. What should you do about bad reviews?
The first thing to remember is that you are not merely an administrator – you are, in essence, the overseer/gatekeeper/high priest of scientific integrity. If you fail, science fails. This means that you actually have to read the paper – in its entirety. It is not acceptable to skim the abstract and make your decision based only on the reviewers’ comments. If you don’t read the paper, how can you possible judge the quality of the reviews assessing it?
Nearly all journals for which I have edited or am currently editing have sent around annual e-mails to the editorial board that go something like this:
“The Journal of XXXX has had an X-fold increase in the number of submissions over the last X months. We therefore strongly encourage you to be extremely critical of what you let go to review. If a paper is not within the top X% of all articles in the journal, you should consider outright rejection instead of sending to review.”
Sounds harsh, I know, but it is the sentiment behind the ubiquitous “space is at a premium in our journal, so we cannot accept all articles regardless of their merit” that you’ll get when you receive that rejection e-mail. Of course, this is mostly bullshit – space is definitely no longer at a premium as the entire world shifts to online publication. There are no space restrictions on the internet. The main reasons this excuse is given are to (i) maximise profits for the publishing company, (ii) reduce the workload for largely volunteer editorial staff and (iii) increase the journal’s impact factor.
I’ll let you decide on the morality of this reality, but I need to make it clear that editors are now increasingly directed from on high to reject papers as often as they can. It is therefore understandable that when a negative review comes in, the easiest thing to do is just accept it at face value and hit the ‘reject’ button.
But please resist that temptation. A couple of pointers about how to recognise a bad review (both unfairly negative or suspiciously supportive ones) are in order:
- If a review is only a paragraph long, be very, very suspicious of its quality and objectivity. Unless it’s a review from someone I trust absolutely to give an educated and objective opinion, I nearly always dismiss it.
- Likewise, if the reviewer is clearly adversarial, or appears to take offence at the audacity of the authors even to write such rubbish, then take the review with a big shaker of salt.
- Always, always, always check the collaborative relationship between a reviewer and the authors. Good mates tend to overlook major faults.
- If the review focuses almost exclusively on ‘lack of novelty’ without critiquing the methods or results, perhaps give the authors a little more benefit of the doubt. Novelty is a poisoned chalice – don’t fall into the trap of thinking that all papers have to be absolutely, ground-shakingly and Earth-shatteringly novel. True novelty is very rare.
After deciding on whether a review should be believed in its entirety or dismissed as a poor-quality and subjective tirade, then the next thing to do is weigh the evidence for or against. If you have one high-quality review (whether negative or positive) and one poor-quality review, you simply cannot make an informed decision. Of course, add in your own appraisal, but I would always recommend obtaining at least one more (high-quality) review before deciding which way to go. This will of course extend the time to publication, but to me this is far more important than rejecting a really good paper based on a shaky assessment, or worse, accepting one that is fundamentally flawed.
So being an editor means that you have to edit, and do it well or don’t do it at all. It’s an essential component of the greater process of scientific knowledge generation.
ConservationBytes.com 
[…] and be as sickly sweet as possible, while still exposing the flaws in the review and decision. Editors are human too, and so they will respond negatively to insulting language and obvious anger. […]
LikeLike
[…] free (submitted articles), has its construction and quality control done for free (reviewing & editing), and then sells its final products for immense profit back to the very people who started the […]
LikeLike
[…] ‘lacking novelty’ excuse is an old trick that you will see in most rejections that do not explicitly highlight any […]
LikeLike
[…] own approaches for reviewing their colleagues’ work. But just as it’s my opinion that many journal editors do an awful job of editing, I know that many reviewers do rather a shit job at their assigned tasks. This perspective comes […]
LikeLike
[…] Be a good reviewer, but be a better editor. […]
LikeLike
A good blog Corey
I, like everyone else wish I had the time to review all manuscripts that come across my desk (as people “do this for me”) but the issue these days is more about having the time to do this. It is not that people do not want to review new manuscripts (irrespective of whether they are good or bad – although I understand that this is the gist of this blog) the fact is that our employers often do not factor these types of activities into our busy workloads and so most reviewers end up doing this in their own “spare” time – and we each only have so much of this.
Therein lies the problem, the journals and submitters want people to give up their time in an already busy world and that is a big ask. Now (and dare I say this), if reviewers were paid then people could say to their employer, I’m getting x amount for this work thus this could be “factored” into their work time. Now I appreciate that this is a slippery slope but how else do you persuade employers to account for your work time?
Book companies do this – pay people for editing and reviewing so why are so goddamn stupid in our profession that we have to do everything for free – or as a volunteer. Today, I get so many calls to volunteer for this and that in conservation that I could be a full-time volunteer – it is draining and equally, it is hard to say no when there are just so many “good causes” out there.
When are people going to wake up in our profession, just like lawyers and doctors have done, and realise that goodwill goes so far but at some stage or another we have to make a living – we need to start recompensing people for their time. No longer, should we be considered a cheap voluntary profession. We spend a long time at university (which costs a crapload of money), we spend even longer building our reputation and we still get paid peanuts compared to the other “professions” and on top of this many people/
agencies/journals still expect us to give up what valuable time we have left.
I could do more reviewing if I was paid for my time but this probably won’t ever happen unless we decide that we become true professionals and charge (even a small amount for our time). I may be out of line, but I honestly think our profession needs to take a long hard look at itself. Given this, I will continue to adopt the policy that if a manuscript is sent to me for review, I have the option to say no “if I don’t have the time”. If I feel I have the time, then yes I will assist wherever I can – what more can one ask?
LikeLike
Excellent piece Corey. Having been on the editorial board of eight journals, and still serving on five, finding reviewers is the bane of my existence. I am also finding that the papers I am asked to handle have increased significantly compared to 3-4 years ago. Some of these journals require only 2 reviewers, but I prefer at least 3, but even finding 2 willing reviewers is very hard. Add to that the pressure to review papers fast and sometimes the pressure from the authors themselves. Right now, I have a paper for which I sent the 13th reviewer request. I had only one person accept so far and that’s a late review. Last year, I had a paper submitted which finally received 2 reviews after 3 months. I could have made a decision right there, but these were opposing reviews and after having read the paper, I was also on the fence about it. To give the authors a fair chance, I sent it to a third person, which added another month to the review. During that period, the senior prof in the paper wrote a scathing complaint to the editor of the journal about the lateness of the review. A good deed never goes unpunished. With this in mind, I no longer go out of my way to look for more reviewers than the journal requires. I still invite at least three people, but if I get the required 2 reviews (or whatever number), I make my decision based on that. We have a broken system where papers are skyrocketing in number, everyone wants multiple good reviews fast, but fewer and fewer people want to review any papers.
LikeLike
Thanks for the advice Corey. And Cagan, thanks also for that story. I will never complain to an editor about lateness!
LikeLike
Corey, once again I couldn’t agree more with you. If I was pro-wedding, I would marry you!
LikeLike
Bises, alors!
LikeLike
Wow it doesn’t take much to get a marriage proposal these days does it?! On a more serious note, Corey thank you for your blog entry on journal editors. I agree with you and also wanted to add the issue of editors who hold grudges…and subjective culling. Such an editor I used to know had a serious grudge against someone she conducted her PhD with and took every opportunity she could to make this person’s publishing attempts in the journal in question a battle in vain. The slightest negative comment from a review would be enough for that person who was the only expert in her field at that time to reject the paper. How does one protect himself/herself from such a subjective process other than avoid publishing in the said journal? And no this is not a case of the ‘friend’ being me… ;-)
LikeLike
I think most of us have stories like that. My suggestion is to cast your submission net widely (i.e., submit to many different journals) to avoid individual monopolies like that. Remembering that editors are (barely) human too can help understand the less-than-perfect aspects of many editorial decisions. I’d also highly recommend that in the cases of clear bias (and you must have evidence), appeals can lead to positive outcomes.
LikeLike