I have written previously about Journal Impact Factors (here and here). The response to these articles has been great and earlier this year I was asked to write something about JIFs and citation distributions for one of my favourite journals. I agreed and set to work.
Things started off so well. A title came straight to mind. In the style of quantixed, I thought The Number of The Beast would be amusing. I asked for opinions on Twitter and got an even better one (from Scott Silverman @sksilverman) Too Many Significant Figures, Not Enough Significance. Next, I found an absolute gem of a quote to kick off the piece. It was from the eminently quotable Sydney Brenner.
Before we develop a pseudoscience of citation analysis, we should remind ourselves that what matters absolutely is the scientific content of a paper and that nothing will substitute for either knowing it or reading it.
That quote was from a Loose Ends piece that Uncle Syd penned for Current Biology in 1995. Wow, 1995… that is quite a few years ago I thought to myself. Never mind. I pressed on.
There’s a lot of literature on JIFs, research assessment and in fact there are whole fields of scholarly activity (bibliometrics) devoted to this kind of analysis. I thought I’d better look back at what has been written previously. The “go to” paper for criticism of JIFs is Per Seglen’s analysis in the BMJ, published in 1997. I re-read this and I can recommend it if you haven’t already seen it. However, I started to feel uneasy. There was not much that I could add that hadn’t already been said, and what’s more it had been said 20 years ago.
Around about this time I was asked to review some fellowship applications for another EU country. The applicants had to list their publications, along with the JIF. I found this annoying. It was as if SF-DORA never happened.
There have been so many articles, blog posts and more written on JIFs. Why has nothing changed? It was then that I realised that it doesn’t matter how many things are written – however coherently argued – people like JIFs and they like to use them for research assessment. I was wasting my time writing something else. Sorry if this sounds pessimistic. I’m sure new trainees can be reached by new articles on this topic, but acceptance of JIF as a research assessment tool runs deep. It is like religious thought. No amount of atheist writing, no matter how forceful, cogent, whatever, will change people’s minds. That way of thinking is too deeply ingrained.
As the song says, “If I can’t change your mind, then no-one will”.
So I declared defeat and told the journal that I felt like I had said all that I could already say on my blog and that I was unable to write something for them. Apologies to all like minded individuals for not continuing to fight the good fight.
But allow me one parting shot. I had a discussion on Twitter with a few people, one of whom said they disliked the “JIF witch hunt”. This caused me to think about why the JIF has hung around for so long and why it continues to have support. It can’t be that so many people are statistically illiterate or that they are unscientific in choosing to ignore the evidence. What I think is going on is a misunderstanding. Criticism of a journal metric as being unsuitable to judge individual papers is perceived as an attack on journals with a high-JIF. Now, for good or bad, science is elitist and we are all striving to do the best science we can. Striving for the best for many scientists means aiming to publish in journals which happen to have a high JIF. So an attack of JIFs as a research assessment tool, feels like an attack on what scientists are trying to do every day.
Because of this intense focus on high-JIF journals… what people don’t appreciate is that the reality is much different. The distribution of JIFs is as skewed as that for the metric itself. What this means is that focussing on a minuscule fraction of papers appearing in high-JIF journals is missing the point. Most papers are in journals with low-JIFs. As I’ve written previously, papers in journals with a JIF of 4 get similar citations to those in a journal with a JIF of 6. So the JIF tells us nothing about citations to the majority of papers and it certainly can’t predict the impact of these papers, which are the majority of our scientific output.
So what about those fellowship applicants? All of them had papers in journals with low JIFs (<8). The applicants’ papers were indistinguishable in that respect. What advice would I give to people applying to such a scheme? Well, I wouldn’t advise not giving the information asked for. To be fair to the funding body they also asked for number of citations for each paper, but for papers that are only a few months old, this number is nearly always zero. My advice would be to try and make sure that your paper is available freely for anyone to read. Many of the applicants’ papers were outside my expertise and so the title and abstract didn’t tell me much about the significance of the paper. So I looked at some of these papers to look at the quality of the data in there… if I had access. Applicants who had published in closed access journals are at a disadvantage here because if I couldn’t download the paper then it was difficult to assess what they had been doing.
I was thinking that this post would be a meta-meta-blogpost. Writing about an article which was written about something I wrote on my blog. I suppose it still is, except the article was never finished. I might post again about JIFs, but for now I doubt I will have anything new to say that hasn’t already been said.
—
The post title is taken from “If I Can’t Change Your Mind” by Sugar from their LP Copper Blue. Bob Mould was once asked about song-writing and he said that the perfect song was like a maths puzzle (I can’t find a link to support this, so this is from memory). If you are familiar with this song, songwriting and/or mathematics, then you will understand what he means.
Edit @ 08:22 16-05-20 I found an interview with Bob Mould where he says song-writing is like city-planning. Maybe he just compares song-writing to lots of different things in interviews. Nonetheless I like the maths analogy.
Reblogged this on In the Dark and commented:
Why do people still use Journal Impact Factors?
I don’t know either, but they should stop..
“It was then that I realised that it doesn’t matter how many things are written – however coherently argued – people like JIFs and they like to use them for research assessment. I was wasting my time writing something else.”
You are so dead on – also with your comparison to religion. That’s not pessimistic, it’s realistic. Sometimes reality is depressing.
Just like you, I’ve given up on changing people’s minds. If evidence doesn’t work nothing will.
I still think it’s worth trying to change the reality, though. If we can’t change people’s minds about JIF, then we need to get rid of the JIF. That entails getting rid of journals. No journals, no JIF. Whenever I suggest that, people usually exclaim we should not through the baby out with the bathwater. To which I can only say: we don’t own the journals, so we don’t even have a baby to bath. Our baby is being held ransom by a corporate oligarchy and if we want to just see it, we have to pay to the tune of US$10 billion a year. If we want to bath our baby and nurture it, we need to free it from its captor.
Thus, I say cancel all subscriptions. We’ll have access to all previously subscribed articles anyway. Anybody who wants to keep paying subscriptions should do so with their own money.
Thanks for the comment Björn. Getting rid of journals would certainly get rid of JIFs!
Many applicants think that the reviewers of their applications have free access to all journals through university libraries. Even when this is the case (it is often not), access can require several minutes of clicking, searching, finding the right issue, downloading, etc. It can be a huge pain for many and they simply won’t bother. This is another great reason for ensuring you either submit a preprint to BioRxiv and/or publish via open access (recognizing the additional costs).
Totally agree Jim. Add to that: archiving at PMC/Europe PubMedCentral or other depository (in the form stipulated by the journal).
The JIF caries some useful information and the problem is not so much with its existence but with its abuse to provide “high-precision” evaluations. So, one way to emphasize its low accuracy is to provide JIFs together with other metrics based on the same data, i.e., google rank scores and to highlight the discordance: http://www.northeastern.edu/slavovlab/blog/2015/11/02/deceptive-numbers/
I have concluded that using JIF can be OK in some cases as long as its limitations are acknowledged and it’s recognized to be the dull tool it really is.
Also, even though the arguments against JIF have remained the same for decades, I think they have penetrated the scientific community much deeper, thanks to efforts like yours. I know that the pace of progress has been frustratingly slow but it is not zero. The very existence of DORA is evidence for such progress.
Thanks Nikolai. Yes, you are right. The awareness is there. Often people preface some statement about JIFs by saying that they know that they are flawed. Maybe this is because they are talking to me and they have read my blog – I don’t know!
Steve, there is hope: https://sfari.org/updates-and-events/sfari-blog/2016/sfari-supports-preprints-for-the-life-sciences