The Role of Peer Review in Establishing Scientific Validity

By Sheeva Azma

Do issues with scientific peer review carry over into the legal world when dealing with scientific evidence?

Thanks to being a member of the National Association of Science Writers, I was lucky enough to virtually attend a two-day conference on science and the courts for free. The conference happened September 21 and 22, 2023 at the American Association for the Advancement of Science headquarters in Washington, DC. I was only able to virtually attend the first day, but it was exciting and packed with information all the same. You can read all my session recaps here.

Below, I discuss a session called “As a Matter of Fact: ‘General Acceptance’ in Emerging vs. Established Science.” The session concerned peer-review: how it is established, limitations and benefits, and ways that scientific peer review fits into the legal system.

What is peer review and how does it fit into science?

Long before science makes it to the courts, it must be vetted through a scientific process called peer review. In other words, peer-reviewed science is generally known as being rigorous and scientifically valid.

The session moderator, Dr. Brad Wible, Senior Editor in Policy and Education at Science Magazine was first to speak about peer review and the nature of science. “Contrary to what we learn in school, science is not a static body of facts, but a dynamic social process, mediated by shared practices and norms. It’s iterative, continually updating and overriding itself, refining and refuting established findings and methods and experts who champion them, developing and testing new methods,” said Wible.

He mentioned that the panel was unique to the conference because it would discuss how the scientific community decides what is valid scientific expertise in the first place, long before courts are involved. Peer review is the process of scientific journals vetting papers before publication in an effort to ensure scientific validity. One could say that peer review involves evaluating research to be published using a jury of scientific peers.

Arizona State University Law School professor Dr. Michael Saks provided a quick primer on the history of peer review. He mentioned that while peer review started in the 17th century, it is mostly a post-World War II phenomenon. Before that, before there was peer-review or even journals, people circulated articles and they got popular, went to conferences, etc. There were no editors and peer reviewers back then.

Tradeoffs in peer review

A lack of peer review evokes fears of a “race to the bottom,” while too much peer-review may raise the bar too high to be able to publish in academic journals. Wible, the moderator, noted that the commercial nature of publishing may drive publishers to resort to clickbaity articles to increase revenue or even drive scientists to engage in fraud just to get published.

Dr. Bankston, former CEO and Managing Editor of the Journal of Science Policy and Governance, mentioned the COVID pandemic as an example for figuring out the tradeoffs between timeliness, credibility, and academic rigor versus peer-review. Peer review takes a long time, but the pandemic was a rapidly evolving crisis, so scientists published preprints, which are not peer-reviewed, but have a low barrier to publish and are easy to get out there.

What happens in the absence of peer review? Bankston says that scientists tend to refer each other as experts as well, and that trust develops from commonalities between the scientist and the audience. Bankston herself says she is driven to seek out people who propose things differently from the status quo.

While peer review is the gold standard for publishing scientific papers, more and more scientists have been publishing non-peer-reviewed preprint articles. When you think about iterative science and publishing better versions of science with input from the community, preprints become valuable, even if they have not been rigorously evaluated by fellow scientists.

Dr. Jake Yeston, an editor in research for physical sciences at Science Magazine, spoke about the nature of expertise in science versus in other spheres such as law. The source of validity is different between science and law. When dealing with an expert witness in a court, the main questions to establish validity are things like: “Is this person credible?” and “Do people believe their analysis is valid?” That person must convince non-experts. Outside of science, you might be qualified for something because you’ve been doing it for 20 years, he says. However, that’s not how peer review works in science. There are many different things scientists consider in peer review beyond how long a person has been publishing. Furthermore, what counts as rigorous differs between science and other fields.

Paradigm shifts in peer review

Dr. Michael Saks, who studies applied social psychology, also chimed in. “I think you can only tell there’s a paradigm shift [in science research] when you look back.” When he was a journal editor at Law and Human Behavior, he tried to create a paradigm shift after he tired of reading “article after article” about eyewitness identification. So, he took it upon himself to write an editorial, in which he basically said, “there’s a lot more going on in the law that might benefit from some psychological research than eyewitnesses than juries.” He listed a bunch of topics that might be interesting to researchers instead. The end result was that his effort didn’t work: he got a fair amount of pushback from scientists, who were angry that he was telling them what to research. He ended up agreeing with a critic of his that said that scientists must tackle a research area from multiple perspectives to really understand it.

Still, Saks remains a skeptic of new, untested science. “When anything comes along, you don’t want to put too much faith in it too quickly…we now have a lot of failures-to-replicate problems.” Scientific learning is a “slow, step-by-step, unpredictable process,” he says.

“Publish or perish,” toxic science culture, and structural issues with peer review

The session concluded with questions from the audience. Elise Walsh, a recent Master’s graduate of the Johns Hopkins Human Genetics program, stepped up to the microphone. She asked about “publish or perish,” which is the idea that scientists must publish often to be successful. Walsh recounted that when she was in graduate school, she lost funding because the government did not want to fund her research. She also asked about null results: how can we move forward with contradictions or null results?

Bankston replied that null results and contradictions are valuable, too, so should be published. She commented that “publish or perish” is part of a larger toxic science culture that needs to change. “There are a lot of other things [beyond publish or perish] that make a scientist a valuable person who can talk about technical issues, because those publications aren’t everything you do. I think it’s a broader culture thing,” Bankston says.

Yeston chimed in and stated that publishing datasets becomes valuable when considering research that has no result or contradictory results; crediting, citing, and linking to data can be a way to share the data and get others looking at it, especially when the data’s significance is not yet clear. Yeston is an advocate of publishing data, recounting that when he went to grad school in the 1990s, he would send a stack of papers with the data to the journal editors. Some people may not want to publish their data, he says, for fear of losing their upper hand in their research area, but as he argued in this session, it is a way to increase transparency in science and help create mechanisms to further validate published (and thereby established) science.

Learn More about Science in the Courts

If you’re interested in the role of science in the legal system, you might want to check out my other articles about the AAAS Scientific Evidence and the Courts conference. Check them out here. You can also watch this session, or all of the conference sessions, on YouTube.