How can we prevent users from cherry-picking references?

Question

A far larger number of answers than I expected use articles in scientific journals as references. And it turns out that there are several answers with a lot of references already that I consider very misleading, even more so as they look rather impressive due to the sheer number of references.

References are not a panacea to a quality problem, requiring them like Skeptics does is not a complete solution to preventing bad answers. It can filter out an enormous amount of crap, but it is still pretty easy to post a referenced answer that is completely misleading or wrong.

A single study is often not conclusive on its own, especially if it is in cell cultures or animals. Drawing conclusions from such basic research to human health is often premature. For any subject that is sufficiently well reasearched, you can probably find a study that supports any conclusion. The important part is evaluating the whole body of research.

Just picking out studies that agree with a preconceived idea can lead to an impressive, but wrong answer. And this site doesn't seem to be well equipped to deal with that issue.

Any idea on how to handle this problem?

Answer 1

For any subject that is sufficiently well researched, you can probably find a study that supports any conclusion. The important part is evaluating the whole body of research.

You're right about this for the most part.

First and foremost, use your down vote if you think the answer is unhelpful.

Rather than placing onerous burdens on the user in the form of more restrictive answering guidelines (no paywalls, no small studies, no animal studies, must represent every viewpoint out there), if you feel the answers are erroneous or misleading, why don't you provide another answer which is better, and let the readers decide for themselves?

Or as an alternative, you can comment on what you think is wrong under the answer and hope that the user is gracious enough to edit their answer (I have done this and been met with an impressive amount of graciousness.) The problem with this is that comment wars can ensue.

Finally (and probably the least helpful response), you can flag the moderator if you think the answer poses a threat to readers. The mod can monitor that particular user's answers for a while.

I feel that it is not only voluntary of users to answer, but an answer benefits the site. Of course we need to safeguard the quality of the answers. Place enough restrictions on them, though, and there will be many fewer answers.

_{This reminds me of a story heard at an Emergency Medicine conference. The push in medicine for a number of years now has been towards evidence based medicine, which sounds fine on the surface of things, but a lot of medicine is based on half a century of practice that has worked. The speaker mentioned that some things don't need to be studied to know what is useful. For example, no one has ever tested the hypothesis that parachutes save lives. He said we needed a rigorous, double blinded study for evidence. It would entail 100 people, 50 of whom were provided with parachutes in backbacks and instructions in their use, and 50 of whom were given backpacks stuffed with newspapers and the same instructions. Then toss them all out of a plane. Only then could we say with certainty that parachutes save lives because it's been studied! (And we could easily identify the placebo effect as well.) My point is, not everything will have the study you want to see. Doctors, at least, have used animal models for centuries.}

Of the 105 Nobel Prizes awarded for Physiology or Medicine, 91 were directly dependent on animal research. Animal research underpinned the very first Nobel Prize to be awarded for Physiology or Medicine to Emil von Behring in 1901 for developing serum therapy against diphtheria, as it did the most recent awarded in 2014 to John O’Keefe, May-Britt Moser and Edvard Moser for studies of the inner GPS of the brain of a rat.

Answer 2

I have been thinking about this too, and came up with a couple of solutions:

1. Have a whitelist of references

2. Ensure 3+ references per answer

3. Require the study to be peer reviewed

4. Disallow non-study references.

As you can see by the strike through, none of these seem to solve the problem for me, which might make you think that we have no solution?

Not quite.

I realised that there is no "catch all" statement that is going to fix this, and so we need to use this site the way it is intended. Vote.

Voting is central to our model of providing quality questions and answers; it is how

• Good content rises to the top
• Incorrect content falls to the bottom
• Users who consistently provide useful content accrue reputation and are granted more privileges on the site

That's our solution. All we have to do is downvote anything that we judge isn't high enough quality for us.

This is going to be a little opinionated - some people will be freer with their downvotes, but the SE model works.

If you don't think something is good enough, downvote it. If it is good, upvote it. If you can't decide, move on. I often find myself clicking downvote, and then thinking "actually... It's not that bad" and simply not voting.

One thing. If you downvote, please, please, please comment if appropriate - an anonymous downvote hurts the site more that people understand. If the studies aren't good enough, downvote and comment to say so - maybe even give them some sites that may have some suitable studies? Or even edit it yourself - you have the power to.