Experiments and Research

Not all experiments or research studies are created equal. Not by a long shot. Some can be downright misleading if they are misinterpreted. Almost all research is prone to bias. And there are many different sorts of bias. For example, if you decided to find out the incidence of obesity in women and chose to survey only women at the gym, your results would suffer from "selection bias". 

Here for your consideration is the famed "Evidence Pyramid" which depicts the range of different types of scientific studies and their levels of validity. At the bottom are represented the very numerous case studies and expert opinions, which are of low relative validity. At the top are the difficult and expensive to produce, massive, time consuming, but very informative and unfortunately few Meta-analyses. 

THE EVIDENCE PYRAMID

Who Knew ? 

Each type of study is dictated by the type of condition it is designed to study, as well as cost and other practical considerations. Each type produces results which are limited by certain types and amounts of bias. Said another way, the answers we get are dependent on the way we ask our questions. 

Animal and Lab studies - Cheap and easy to do, but results may not apply to humans or outside of test tubes. They provide suggestions for further study.  

Case studies - These are studies of one, which can be interesting, but findings may not be generalizable. 

Expert opinion - Well, of course this depends on the expert. Even experts disagree. 

Case controlled studies - A comparison of those with a disease to similar people without the disease. Interesting associations may be uncovered, but causality is not determined. 

Cohort Studies - Comparison of large exposed vs. non-exposed populations over long time frames. Cause and effect may be suggested at, but with many potential biases. 

RCT, randomized controlled trials - These are done prospectively, double blind, and randomized, useful for determining cause and effect. Prospective means you start observing before anything has happened. You introduce a variable, say a new medicine, then watch the effects as time goes forward. Randomized means part of the study participants are randomly assigned to get placebo, and the rest get the real treatment.  Blinded means no one knows who got which. Double blinded means the researchers don't know either. RCTs are known as the best way to minimize bias. 

Systematic reviews - Synthesis of all relevant studies on a particular topic. A study of studies. Quality of results depend on the quality of the studies included. 

Meta-analysis - A Study of Studies whose of results are summarized numerically. Has great power to be informative and relatively free of bias. 

The research community tends to use the most valid type of study that money and practicality can afford. Because of these constraints, research results of varying validity are produced. Those of us depending on their results must bear in mind how they were produced so that we can use the information properly in clinical practice. 

The Media commonly report on research results in science and medicine. They rarely mention the study design. Moreover, they tend to simplify and dramatize the material because that's what holds the public's interest. It is human nature to want simple answers and quick miraculous cures. It is natural to want cheap easy solutions to life's problems. This is why unscrupulous advertisers and practitioners can tout nonscientific theories and advertise unproven products. It is also why people hang on to unreasonable ideas despite their better judgement. 

Who's your farmer ?  

err... source of information ? 

I warned you you were going down a Rabbit Hole. 

Now those of you who made it all the way through Levels of Evidence deserve a little treat. Here is one of my favorite songs, Crosseyed and Painless, played by one of my favorite bands, The Talking Heads. To me, the song is about the limitations of hard statistical facts when they are brought to bear in messy real life situations. Look up the lyrics sometime and see what you think. 

Meanwhile, click HERE to go back to Information and Learning.