Happy New Year as well! I’ve been taking some time off from writing for the holidays but now I feel it’s time to get back into it now that it is several days in to 2016! This year I hope to cover more of my future plans and how they relate not only to interesting new ideas but also to basic science. I’ll kick this year off though with a review of a review of a meta-analysis of how research is conducted. In essence, talking about science in a scientific way but about as far away from specific topics as possible.
This post is about an article published yesterday in the open access journal PLos One: Biology called: “Reproducible Research Practices and Transparency across the Biomedical Literature” by Iqbal et al. The goal of this paper is to highlight the issues surrounding the reproducibility or lack thereof in current literature. It’s fitting that this paper is published in the PLos One, perhaps the most recognized open access journal. There has been a great concern that many subscription journals are trying to “pull the wool” over the eyes of the general public. I tend to be of the opinion that my colleagues in biomedical sciences are mostly acting nobly and not actively trying to skew results or mislead people. Still, in an era of increased competition for funding and with enormous pressure to “publish or perish” shortcuts are sometimes taken. Even worse, in my mind, are the predatory open access journals that are essentially, “pay to publish”. You can find the now famous “Beall’s List” of these types of journals here: http://scholarlyoa.com/publishers/ . In other words, stay away from these journals. PLos One, however, has a history of producing quality peer-reviewed work and stands as somewhat of a beacon to shine light on research malpractice.
The short-comings discussed in the review article by Iqbal et al are, in their opinion, systemic problems rather than limited to one type of journal. The authors argue that, while there may be more willingness to share final results through open access, few, if any researchers share exactly how they arrived at those results. The authors of this paper found that out of 441 biomedical papers, only one published the exact protocols used and none showed the raw data. While greater than 50% of papers included empirical analysis of their results, few of them reported conflicts of interest. What’s more, in the conclusion the authors are critical of NIH plans to “improve data sharing” in an initiative started in 2003. They suggested that the NIH’s plans have been fruitful. The paper paints a valid picture of the state of biomedical research heading into the year 2016.
I must, however, respond to the author’s critique of the field of biology and medicine by pointing out the realities of research. One other point they bring up is the somewhat subjective topic of people claiming their research is “novel” versus “replications”. The reality is that people do not get funding for simply “repeating” other people’s research. Journals don’t want to publish repeats because, by and large, people don’t care about them. They would rather just read the initial papers in the field and cite those papers. If they do publish work repeating other people’s work exactly it is usually to contradict or to find something the original authors missed.
Our goal as scientists is inherently to create new knowledge. Yes, people should be able to reproduce results. Repeating experiments helps to provide confidence in the original results. The check for this though is not to publish repeats, but rather to publish critiques and challenges of the original results if you find they are faulty. Finding fault with old results shouldn’t be a condemnation but it should be seen as scientific progress. This is a way of creating “new” ideas from old ones and is a proper use of the Scientific Method.
The author’s worried about the lack of full disclosure of data in science. Let me make this clear that the STANDARD in the scientific community is to keep all data available for anybody who wants it for a minimum of 5 years after the paper is published. Researchers don’t simply throw samples and data away right after the paper is published, that would be tantamount to scientific misconduct. Furthermore, protocols are to be kept infinitely and should be given to anybody who asks for it. I have often run into incomplete protocols in my work. In fact, one paper mentioned a molecule that is well cited in the literature, but if you look at the original paper it references a paper that “submitted for publication”…since 2001!!! This is why the contact information for the corresponding author is listed on a publication and I can attest that contacting them works. Perhaps the authors should have tried to contact all 441 corresponding authors to support their hypothesis. I will have to disagree with them that needing to contact the authors constitutes a “lack of transparency”.
One other point I wanted to bring up is the feasibility of publishing raw data in open access formats. You may be aware that in graduate school I worked in a lab that analyzed data from images from a two photon microscope. This machine gives images that contain many bytes of information per pixel. The raw data for one experiment could run into tens or hundreds of GB! It is just not feasible to have a publication that houses Terabytes to Pentabytes of raw data on their servers 24/7. Perhaps this could be done on some sort of rotating system but, to me, interested parties contacting corresponding authors for data seems to be the best method.
Overall, I will say that I agree with the author’s call for increased transparency in theory. It would be amazing to have access to all data all of the time and to be able to have time to repeat experiments infinitely. The fact of the matter is that there are limited resources and limited time to achieve these goals. The authors believe that there is a culture of cutting corners and/or publishing research that is not verifiable. In my opinion, the best way to combat this type of culture is to take responsible conduct of research training seriously and to better train young researchers. Also, increased science education would allow the public to critique results which would help speed up the validation process.
Sorry to start the new year off on a semi-rant. I’ll be back to my own thoughts in the next post! As always feel free to leave comments! Have a Prosperous New Year!
Link to the paper: