Study delivers bleak verdict on validity of psychology experiment results

[PokerGuy]

http://www.theguardian.com/science/...-on-validity-of-psychology-experiment-results

75% of social psychology experiments and half of cognitive studies failed the replication test

These aren't the things published in some random rag somewhere, this was looking at studies in the top ranked psychology journals. Replication of results and verification are very core to the scientific process. What does it say about the state of the field when 75% of study results can't be replicated?

Not that this is shocking, I suspect many of the social psychology studies are really just SJW types in academia trying to find a way to support a view. That also explains why cognitive psychology studies fared better (but those results still couldn't be reproduced 50% of the time either).

A good reminder that social psychology is a lot more squishy than hard science.

 

[Brainonska511]

The new analysis, called the Reproducibility Project, found no evidence of fraud or that any original study was definitively false. Rather, it concluded that the evidence for most published findings was not nearly as strong as originally claimed.
....
Strictly on the basis of significance — a statistical measure of how likely it is that a result did not occur by chance — 35 of the studies held up, and 62 did not. (Three were excluded because their significance was not clear.) The overall “effect size,” a measure of the strength of a finding, dropped by about half across all of the studies. Yet very few of the redone studies contradicted the original ones; their results were simply weaker.

http://nyti.ms/1U8G10F

 

[PokerGuy]

Their conclusions, reported Thursday in the journal Science, have confirmed the worst fears of scientists who have long worried that the field needed a strong correction.

The field has taken some serious hits in terms of credibility, but those were specific isolated incidents. This is much more of a blow because it is a systematic review of more than a hundred studies published in the top 3 journals in the field.

 

[Moonbeam]

The beautiful thing about science is that its whole purpose is to achieve real understanding, not the understanding one hopes for. In this way it is self correcting and self critical. Sadly, for those with a CBD, the purpose is self deception and any sign of real self criticism like science has is seen as proof of weakness and any self criticism science makes will be turned into proof that science and its results deserve condemnation and aren't worth listening to.

 

[unokitty]

Chronicle of Higher Education
Less Than Half of Major Social-Science Studies Are Reproducible, Analysis Says

A study has found that fewer than half of 100 major studies in the social sciences could be replicated to produce similar results, The New York Times reports. Published in Science, the analysis was conducted by a team of researchers led by Brian Nosek, a psychologist at the University of Virginia.

The team sought to reproduce 100 studies published in 2008, but in more than 60 cases, the results came out different. That’s not a result of fraud, the researchers say, but an indication that the evidence supporting the studies’ conclusions was not as strong as originally stated.

“We see this is a call to action, both to the research community to do more replication and to funders and journals to address the dysfunctional incentives,” Mr. Nosek told the Times.

So for over half of 100 major social sciences studies:
peer-review != reproducible

Makes me feel old to be able to remember when scientific research meant that the results were quantifiable and reproducible...

Uno

 

[SMOGZINN]

Not that this is shocking, I suspect many of the social psychology studies are really just SJW types in academia trying to find a way to support a view. That also explains why cognitive psychology studies fared better (but those results still couldn't be reproduced 50% of the time either).

This has nothing to do with SJW, this is about the 'publish or die' culture of our universities.

 

[Moonbeam]

This has nothing to do with SJW, this is about the 'publish or die' culture of our universities.

That would be interesting to try to prove. What proportion of papers not reproducible contain claims are positives with SJW warrior claims. Until it is proved, however, it's speculation. To me the idea that what psychologists want to study and what they think might get funding, would certainly include some cunning appraisal of the proclivities of academic funding.

 

[werepossum]

Chronicle of Higher Education
Less Than Half of Major Social-Science Studies Are Reproducible, Analysis Says

So for over half of 100 major social sciences studies:
peer-review != reproducible

Makes me feel old to be able to remember when scientific research meant that the results were quantifiable and reproducible...

Uno

You and me both. The soft sciences have become more like notions.

This has nothing to do with SJW, this is about the 'publish or die' culture of our universities.

That's a good point, although I suspect that progressive views have a lot to do with this. Far too much of science has become like high school science - we know the correct answer, so we just have to edit the data until they give that answer. Publish or die might be the strongest factor though - better ten eventually totally debunked published studies than one lasting study if one is chasing tenure and grants.

 

[woolfe9998]

I majored in psych undergrad because it was a cushy course of study if you wanted to go to law school. Some of what I studied was interesting. The research we covered varied a lot, but much of it was pointless, especially in social psychology.

I recall one study where a person stands on campus and asks random people for directions, then records the results, i.e. how many people were willing to help. In the experimental group, the guy steps on each person's foot, then asks them for directions. Guess what. They made the shocking discovery that people are less likely to offer assistance after you step on their foot. No, this is no joke. They actually designed a study around this, and it was later cited in a text book. I discovered that with psych research, you really have to separate the wheat from the chaff.

 

[MongGrel]

I have my own opinions from the past I won't bother to bore everyone with.

I'll just say I'm not surprised.

 

[maluckey]

just wow.

Here's a good read from a scientist who understands. He wasn't big on psychology as a whole and was unashamed to call them out. This is also one of my favorite speeches of all time.

http://neurotheory.columbia.edu/~ken/cargo_cult.html

 

[Greenman]

The beautiful thing about science is that its whole purpose is to achieve real understanding, not the understanding one hopes for. In this way it is self correcting and self critical. Sadly, for those with a CBD, the purpose is self deception and any sign of real self criticism like science has is seen as proof of weakness and any self criticism science makes will be turned into proof that science and its results deserve condemnation and aren't worth listening to.

I'm not sure that I find questionable research being used to prove questionable research all that convincing.

 

[Moonbeam]

I have my own opinions from the past I won't bother to bore everyone with.

You are very strange.............................and I like it.

 

[Moonbeam]

I'm not sure that I find questionable research being used to prove questionable research all that convincing.

You are free to put to the test any such you find.

 

[OverVolt]

The beautiful thing about science is that its whole purpose is to achieve real understanding, not the understanding one hopes for. In this way it is self correcting and self critical. Sadly, for those with a CBD, the purpose is self deception and any sign of real self criticism like science has is seen as proof of weakness and any self criticism science makes will be turned into proof that science and its results deserve condemnation and aren't worth listening to.

What you do is not science so there is that.

Its more like people lately and on this board in particular just tack on "science said so" type arguments and thats not how any of it works, thats not how it works at all. Science is not some type of authority, science is just the science. The facts as they are. Peoples added interpretations are bullshit. You guys are not entitled to such interpretations.

 

[Moonbeam]

What you do is not science so there is that.

Its more like people lately and on this board in particular just tack on "science said so" type arguments and thats not how any of it works, thats not how it works at all. Science is not some type of authority, science is just the science. The facts as they are. Peoples added interpretations are bullshit. You guys are not entitled to such interpretations.

Telling me that what I do is not science is not science. You, sir, are an idiot, so there.

 

[mizzou]

what % of studies are merely used to finish the masters/doctoral degrees?

 

[Belegost]

This has nothing to do with SJW, this is about the 'publish or die' culture of our universities.

Publish or die is the pressure, but my experience suggests that it's greatly helped along by fundamentally shit skills in statistical analysis amongst many in the social sciences.

Working on machine intelligence from the engineering background I have worked with a lot of cognitive scientists, neuroscientists, and a few human factors psychologists. Only a handful had any solid understanding of the mathematical principles behind the statistical tools they tried to apply to their data, and multiple times I have had to rip apart half a paper and redo the entire analysis myself because the math was bad.

The paradigm that seemed common would be to just grab some off-the-shelf analysis package, throw the data into it, and poke things that were only dimly understood until the results looked good. It was never a case of malfeasance that I could tell, it was simply being ignorant of how to estimate parameters correctly, how to properly set initial conditions, and what knobs could be tweaked, and which ones need to be learned from the data (or designed into the experiment.)

Of course the pressure to put out a half dozen papers a year makes it difficult to put the correct amount of time and effort into learning how to do the math right.

 

[Greenman]

What you do is not science so there is that.

Its more like people lately and on this board in particular just tack on "science said so" type arguments and thats not how any of it works, thats not how it works at all. Science is not some type of authority, science is just the science. The facts as they are. Peoples added interpretations are bullshit. You guys are not entitled to such interpretations.

If you can't measure it, it ain't science.

 

[nickqt]

Peer-review consists of:

Publishing data.
Replication of the experiment/study.

Psychology isn't measurable in the same way abstract math or chemistry is measurable. Anyone who has studied biology beyond what is taught in middle school knows this.

Identical twins with the same DNA sequence can and will have different psychological problems.

There are so many different neurotransmitters, hormones, and environmental factors affecting brain chemistry that it would be absurd to think that all experiments/studies can be replicated between different individuals and more accurately, groups of individuals, even if the first experiment/study has good data.

So, yes, psychological experiments/studies are much less likely be replicated. Such is the inherent difficult in human biology.

 

[unokitty]

Peer-review consists of:
Publishing data.
Replication of the experiment/study.

Peer review doesn't, and hasn't ever, included replication of the experiment.

Peer review can't guarantee that the data itself is accurate or not-fabricated. That's not really the point of peer review: the point is to verify that the methodology, analysis, conclusions, and writing given in the paper adhere to scientific standards. See: Fake data used in study claiming efficacy of gay canvassers

Uno

 

[DrPizza]

just wow.

Here's a good read from a scientist who understands. He wasn't big on psychology as a whole and was unashamed to call them out. This is also one of my favorite speeches of all time.

http://neurotheory.columbia.edu/~ken/cargo_cult.html

Big thumbs up to that writer.

 

[Moonbeam]

Big thumbs up to that writer.

I hear, it's all about whether you are interested in the truth whatever it may be, or you are interested in your ego, your fame, reputation, career, grant money etc.

 

[WHAMPOM]

Don't we know that phycology experment results are most always subjective?

 

[MagickMan]

What you do is not science so there is that.

The vast majority of the time it's not even comprehensible. :\