Study delivers bleak verdict on validity of psychology experiment results

[MagickMan]

I hear, it's all about whether you are interested in the truth whatever it may be, or you are interested in your ego, your fame, reputation, career, grant money etc.

or mere post count, in your case.

 

[Blue_Max]

It's musing how many people have seen this sort of thing coming 1-2 generations ago.

From Starship Troopers;
Jean Rasczak: All right, let's sum up. This year we explored the failure of democracy. How our social scientists brought our world to the brink of chaos.

 

[nickqt]

Peer review doesn't, and hasn't ever, included replication of the experiment.

Uno

Publishing the methodology, statistics and results allows others to analyze the data. That is peer review.

No one gives a shit about 1 experiment/study. Ever.

Eggs cause heart attacks!
Eggs protect from heart attacks!
Eggs have no discernible affect on heart attacks!

Hence, the same methodology will be used by other researchers, which is how you determine if the actual results are worthy of adding to the knowledge base, or throwing away.

First, you publish your data in a journal where peers review the data.
Second, it is replicated by others, or it exists as simply one study that no one gives a shit about.

I'm sorry that I skipped over the middle part. But yes, replicating the experiment/study is part of the process, assuming that anyone gives a shit about the original experiment/study.

 

[OverVolt]

Most of the knowledge is shared among faculty/staff/professors at universities. They have a generally good idea of what is going to happen when they run the experiment. A good example is the first study to use PCR. They did something like sequence the sickle cell gene. That had jack shit balls to do with anything they really wanted to do. What they really wanted to do was publish the PCR methodology.

By the time something is published online you are the last one to hear about it.

 

[Brainonska511]

Most of the knowledge is shared among faculty/staff/professors at universities.

Most "knowledge" is closely guarded until it is published or near publication. You don't want someone to scoop your idea and beat you to a publication or pull off the low hanging fruit on your project. The only reason professors, post docs, researchers, and graduate students are "in the know" is because it is their field and it is their job to know everything they possibly can about their system of interest while keeping an eye on the literature of their own field and related fields in order to gain insight about their own problems.

They have a generally good idea of what is going to happen when they run the experiment.

No. People have a hypothesis about what will happen when they run their experiment, formulated on existing knowledge. If something doesn't go as hypothesized, then it's back to the drawing board to figure out why something didn't work as anticipated. If we knew everything would work when we set up an experiment, we wouldn't bother to set up experiments.

A good example is the first study to use PCR. They did something like sequence the sickle cell gene. That had jack shit balls to do with anything they really wanted to do. What they really wanted to do was publish the PCR methodology.

It's not always good enough to publish a methodology on its own. It's a good idea to show a real application of the tool. And at the time, it was a novel way of combining existing tools in the field to do something new.

By the time something is published online you are the last one to hear about it.

That last line is hardly true at all.

 

[realibrad]

Publishing the methodology, statistics and results allows others to analyze the data. That is peer review.

No one gives a shit about 1 experiment/study. Ever.

Eggs cause heart attacks!
Eggs protect from heart attacks!
Eggs have no discernible affect on heart attacks!

Hence, the same methodology will be used by other researchers, which is how you determine if the actual results are worthy of adding to the knowledge base, or throwing away.

First, you publish your data in a journal where peers review the data.
Second, it is replicated by others, or it exists as simply one study that no one gives a shit about.

I'm sorry that I skipped over the middle part. But yes, replicating the experiment/study is part of the process, assuming that anyone gives a shit about the original experiment/study.

The problem with peer review is that there is an assumption peers are reviewing. Right now the way things work is that once something is submitted, its skimmed over and published. If the journal says its not worth publishing, and another journal does, then only the one that does publish gets the benefit. Many of the papers today are not being reviewed deeply and so most journals publish almost everything they can get their hands on. So long as the paper is complicated enough, it will get through.

http://www.nature.com/news/publishers-withdraw-more-than-120-gibberish-papers-1.14763

 

[nickqt]

The problem with peer review is that there is an assumption peers are reviewing. Right now the way things work is that once something is submitted, its skimmed over and published. If the journal says its not worth publishing, and another journal does, then only the one that does publish gets the benefit. Many of the papers today are not being reviewed deeply and so most journals publish almost everything they can get their hands on. So long as the paper is complicated enough, it will get through.

http://www.nature.com/news/publishers-withdraw-more-than-120-gibberish-papers-1.14763

I never said there wasn't a problem. It's been said here, often the push to publish to keep your position at an institution or retain grants means that more and more data is just put into article form and dumped onto the laps of a bunch of different journals that are all swimming in them.

That said, peer review of the actual data either leads to further studies, or it is quickly forgotten about, which was my point and the reason I replied to someone who I guess thinks I'm purposefully stupid because he found some information on a wiki

Peer review doesn't necessarily mean that every experiment/study is replicated, but experiments/studies that go on to make a difference (either for or against the original experiment/study, by the way) are replicated, with the peer review data providing the parameters to do the replication.

 

[OverVolt]

That last line is hardly true at all.

Everything happens in the real world. The internet is the last to know. The types who frequent this board find that hard to swallow. So I'll just reiterate it. Worst is probably how information spreads 2nd & 3rd hand when its reposted as a science based news article and the meaning of the study is usually twisted.

No. People have a hypothesis about what will happen when they run their experiment, formulated on existing knowledge. If something doesn't go as hypothesized, then it's back to the drawing board to figure out why something didn't work as anticipated. If we knew everything would work when we set up an experiment, we wouldn't bother to set up experiments.

Oh how naive. So they didn't expect to find the higgs boson when they spent all those billions? In order to actually make a big discovery in 2015 the projects are large, collaborative, and they are generally seeking a result they expect to find. They didn't just build the biggest collider ever to "see what happens." Those days are long gone.

 

[Brainonska511]

Everything happens in the real world. The internet is the last to know. The types who frequent this board find that hard to swallow. So I'll just reiterate it. Worst is probably how information spreads 2nd & 3rd hand when its reposted as a science based news article and the meaning of the study is usually twisted.

Yes, science reporting in the news is woefully inadequate and many people are horribly misinformed.

Oh how naive. So they didn't expect to find the higgs boson when they spent all those billions? In order to actually make a big discovery in 2015 the projects are large, collaborative, and they are generally seeking a result they expect to find. They didn't just build the biggest collider ever to "see what happens." Those days are long gone.

They had a good idea of what would happen, but they could still have been wrong, hence, hypothesis. But it's not just "big projects" that this applies to. It's every experiment, from the small to the large - you design an experiment with some idea (hypothesis) about what will happen when you execute it. But when it comes to the new things, you don't always get what you expect or know what you're going to see, hence, the need to experiment and interpret results. But it's not like I would know anything about science and how it works. I only live and breathe it at the moment in my line of work.

 

[werepossum]

just wow.

Here's a good read from a scientist who understands. He wasn't big on psychology as a whole and was unashamed to call them out. This is also one of my favorite speeches of all time.

http://neurotheory.columbia.edu/~ken/cargo_cult.html

Awesome read, thanks.

Peer review doesn't, and hasn't ever, included replication of the experiment.

Uno

Sadly it's often even worse than that. We've seen global warming research that is "peer reviewed" without raw data (some of which eventually was claimed to be lost to avoid that step) and often with incomplete if not very rudimentary methodology. As Realibrad says, it is skimmed and blessed and then published. As long as it's what "we all know" anyway. And that's in hard sciences, which are inherently quantifiable. I'm frankly surprised that any significant percentage of the soft sciences' experiments are reproducible.

EDIT: On the bright side, there is Steve 'n' Seagulls, a Finnish hillbilly hard rock/acid metal bluegrass band. https://www.youtube.com/watch?v=e4Ao...3RfSfOgcNnzq3Z

When you can hear an up-tempo version of Thunderstruck or Paradise City played on banjo, spoons, accordion and mandolin, can things really be so bad?

 

[OverVolt]

Yes, science reporting in the news is woefully inadequate and many people are horribly misinformed.

They had a good idea of what would happen, but they could still have been wrong, hence, hypothesis. But it's not just "big projects" that this applies to. It's every experiment, from the small to the large - you design an experiment with some idea (hypothesis) about what will happen when you execute it. But when it comes to the new things, you don't always get what you expect or know what you're going to see, hence, the need to experiment and interpret results. But it's not like I would know anything about science and how it works. I only live and breathe it at the moment in my line of work.

Oh I think modern science is going down the wrong path so its not surprising we disagree . Science has plateaued when it comes to new discoveries. Not sure how long they can afford to bang their head against the wall.

 

[Brainonska511]

Science has plateaued when it comes to new discoveries. Not sure how long they can afford to bang their head against the wall.

Where do you get this idea from? Discoveries rarely ever come in big leaps. In general, it's a slow, incremental process that builds on previous work.