19

A far larger number of answers than I expected use articles in scientific journals as references. And it turns out that there are several answers with a lot of references already that I consider very misleading, even more so as they look rather impressive due to the sheer number of references.

References are not a panacea to a quality problem, requiring them like Skeptics does is not a complete solution to preventing bad answers. It can filter out an enormous amount of crap, but it is still pretty easy to post a referenced answer that is completely misleading or wrong.

A single study is often not conclusive on its own, especially if it is in cell cultures or animals. Drawing conclusions from such basic research to human health is often premature. For any subject that is sufficiently well reasearched, you can probably find a study that supports any conclusion. The important part is evaluating the whole body of research.

Just picking out studies that agree with a preconceived idea can lead to an impressive, but wrong answer. And this site doesn't seem to be well equipped to deal with that issue.

Any idea on how to handle this problem?

  • 5
    I don't think this deserves to be marked as a duplicate. The referenced question is about what makes a quality source, this is about people selecting only studies that show what they want. While they both reference published studies, they have a different purpose. – JohnP Apr 10 '15 at 14:40
  • I agree here, having good references doesn't mean that all the information is being shown. – Joe W Apr 11 '15 at 17:02
  • Related: How to account for publication bias? – Franck Dernoncourt Apr 23 '15 at 2:04
  • Vaccinations cause autism! [link to study] – bjb568 Apr 23 '15 at 2:37
11

For any subject that is sufficiently well researched, you can probably find a study that supports any conclusion. The important part is evaluating the whole body of research.

You're right about this for the most part.

First and foremost, use your down vote if you think the answer is unhelpful.

Rather than placing onerous burdens on the user in the form of more restrictive answering guidelines (no paywalls, no small studies, no animal studies, must represent every viewpoint out there), if you feel the answers are erroneous or misleading, why don't you provide another answer which is better, and let the readers decide for themselves?

Or as an alternative, you can comment on what you think is wrong under the answer and hope that the user is gracious enough to edit their answer (I have done this and been met with an impressive amount of graciousness.) The problem with this is that comment wars can ensue.

Finally (and probably the least helpful response), you can flag the moderator if you think the answer poses a threat to readers. The mod can monitor that particular user's answers for a while.

I feel that it is not only voluntary of users to answer, but an answer benefits the site. Of course we need to safeguard the quality of the answers. Place enough restrictions on them, though, and there will be many fewer answers.

This reminds me of a story heard at an Emergency Medicine conference. The push in medicine for a number of years now has been towards evidence based medicine, which sounds fine on the surface of things, but a lot of medicine is based on half a century of practice that has worked. The speaker mentioned that some things don't need to be studied to know what is useful. For example, no one has ever tested the hypothesis that parachutes save lives. He said we needed a rigorous, double blinded study for evidence. It would entail 100 people, 50 of whom were provided with parachutes in backbacks and instructions in their use, and 50 of whom were given backpacks stuffed with newspapers and the same instructions. Then toss them all out of a plane. Only then could we say with certainty that parachutes save lives because it's been studied! (And we could easily identify the placebo effect as well.) My point is, not everything will have the study you want to see. Doctors, at least, have used animal models for centuries.

Of the 105 Nobel Prizes awarded for Physiology or Medicine, 91 were directly dependent on animal research. Animal research underpinned the very first Nobel Prize to be awarded for Physiology or Medicine to Emil von Behring in 1901 for developing serum therapy against diphtheria, as it did the most recent awarded in 2014 to John O’Keefe, May-Britt Moser and Edvard Moser for studies of the inner GPS of the brain of a rat.

  • 4
    Yes! Yes! Downvote - this is precisely what the button is for! I'm glad it wasn't just me thinking this! – Tim Apr 9 '15 at 20:22
  • 4
    "Or as an alternative, you can comment on what you think is wrong under the answer and hope that the user is gracious enough to edit their answer" This is very good advice. People have commented on what they have seen wrong with some of my answers, and it really does help. ;) – michaelpri Apr 9 '15 at 20:23
  • 2
    Or even suggest an edit! These tools are all here for us to use, and we seem to be forgetting. – Tim Apr 9 '15 at 20:58
  • 1
    @Tim - I am very reluctant to correct any part of an answer with an edit. I'll gladly add a quote and a link, but I don't think (my problem, not anyone else's) that changing an answer is a good use of editing. This may deviate from SE recommendations, but that's just me. :-) – anongoodnurse Apr 9 '15 at 21:24
  • @anongoodnurse that's fair enough. I suppose it is the sites I am used to - I often correct things in them. – Tim Apr 9 '15 at 21:25
  • 5
    I've downvoted and comments, that just doesn't do enough. The worrying part is that the audience of this site is skewed far towards the general public and not medical professionals, and how is the general public supposed to evaluate those answers in detail? And while we don't have any mods yet, they're also not really supposed to evaluate whether an answer is right or wrong. – Mad Scientist Apr 9 '15 at 21:46
  • @MadScientist - I agree about the mod's function, but... If, say, six people on the site are in a position to know the answer is bad medicine, three of them read the answer, and two of them downvote, one leaves a comment, the other upvotes the comment, it still doesn't really correct the problem. And it is a problem. I had an answer that was diametrically opposed to another answer, and the OP chose that answer. What can I do? Eventually it sorted itself out. But it was still not a good situation to be in. – anongoodnurse Apr 9 '15 at 21:59
  • @MadScientist - I just read another answer where the user misinterpreted the question and actually answered a different question. – anongoodnurse Apr 9 '15 at 22:26
  • Your parachute example is very misleading. The control comparison of no parachute has never been tested in man, because the basic science underlying it, along with the critical pre-human testing in animals, says it is way to dangerous. – StrongBad Apr 11 '15 at 0:39
  • @StrongBad - Exactly. – anongoodnurse Apr 11 '15 at 3:25
  • @StrongBad - a good example of a drug that hasn't been "tested" properly (in my lifetime, anyway) is epinephrine. In anaphylaxis, for ethical reasons it is not possible to conduct randomized, placebo-controlled trials. – anongoodnurse Apr 12 '15 at 4:06
  • @anongoodnurse in most cases once there is a gold standard of care, RCTs with placebos are not possible. Since the test has never been done, we lack a piece of evidence that epinhrine is better than placebo, but what we do know is that epinephrine is just as good as everything else ever tested. – StrongBad Apr 12 '15 at 10:59
  • 1
    We might not have an RCT, but we definitely have observational studies on parachute efficacy. – Fomite Apr 14 '15 at 19:21
  • 1
    @anongoodnurse I will admit my bias, in that I consider observational studies to be evidence-based. – Fomite Apr 15 '15 at 2:37
  • 1
    Considering writing a grant on parachutes...wondering if I can get ones sewn small enough for a BALB\c mouse... so our answer to this to 1) DV, 2) comment, 3) throw the correct answer up...Sounds like SE101 but I guess we might have higher stakes than most (now wondering if there's an Explosives SE) – Atl LED Jun 1 '15 at 3:20
7

I have been thinking about this too, and came up with a couple of solutions:

  1. Have a whitelist of references

  2. Ensure 3+ references per answer

  3. Require the study to be peer reviewed

  4. Disallow non-study references.

As you can see by the strike through, none of these seem to solve the problem for me, which might make you think that we have no solution?

Not quite.

I realised that there is no "catch all" statement that is going to fix this, and so we need to use this site the way it is intended. Vote.

Voting is central to our model of providing quality questions and answers; it is how

  • Good content rises to the top
  • Incorrect content falls to the bottom
  • Users who consistently provide useful content accrue reputation and are granted more privileges on the site

That's our solution. All we have to do is downvote anything that we judge isn't high enough quality for us.

This is going to be a little opinionated - some people will be freer with their downvotes, but the SE model works.

If you don't think something is good enough, downvote it. If it is good, upvote it. If you can't decide, move on. I often find myself clicking downvote, and then thinking "actually... It's not that bad" and simply not voting.

One thing. If you downvote, please, please, please comment if appropriate - an anonymous downvote hurts the site more that people understand. If the studies aren't good enough, downvote and comment to say so - maybe even give them some sites that may have some suitable studies? Or even edit it yourself - you have the power to.

enter image description here

  • Wait, then how do people vote on my posts? – bjb568 Apr 23 '15 at 2:39
  • @bjb568 :P yep... – Tim Apr 23 '15 at 9:45
  • you want me to vote? I vote to remove the cat – Ooker Jul 3 '15 at 10:20
  • @Ooker umm okay...? – Tim Jul 3 '15 at 10:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .