Why are some research studies flawed?

This page provides reasons why individual research studies into autism interventions may not be fully valid or reliable.

Purpose of the research

The researchers who carried out the study may not be completely independent. For example,

they may have a biased hypothesis (they are setting out to prove something they already know is right) or
they may stand to gain financially if they claim that an intervention is successful.

The study may not have lasted long enough for the results to be valid. For example, the study may only have lasted for a few weeks, while the effect of the intervention may not be apparent until months later.
The study may have used techniques that are more likely to produce a biased result. For example, an observational study is more likely to be biased than an experimental study, and a non-randomisd control study is more likely to be biased than a randomised control trial.
The researchers may not have reported any confounders i.e. things which may have influenced the results.

The study may not include enough participants for the results to be statistically significant. For example, a study which looks at one or two people is much less likely to be statistically significant than a study which looks at more than twenty people.
The study may not record vital information about the participants in the study group, such as the type of autistic spectrum disorder they had or whether they had any other conditions which might have affected the result.
The study may not record whether the intervention is specifically relevant to people with autistic spectrum disorders. This sometimes happens when people with autism are included in a research study alongside other groups of people.

The study may not record detailed information about the intervention and how it was delivered. This makes it more difficult for other researchers to replicate the study and check the accuracy of the findings.
The study may not record detailed information about the intervention received by the control group. This makes it more difficult to compare the effects of the intervention on the study group compared to the intervention received by the control group.
The study may not record information about other things which may have affected the result. For example, family circumstances or other treatments being received at the same time as the intervention could affect the results of the study into the intervention.

The study may not use standardised outcome measures i.e. ways of measuring improvements in the participants. This makes it difficult to compare the results with other studies
The study may use different measures before and after the intervention. This makes it difficult to compare the effect of the intervention.
The study may only use subjective measures e.g. parental observations of children’s behaviour. This makes the results less reliable.
The assessors – the people evaluating the study – may know which participants received which intervention. This makes it more likely they could influence the results, however sub-consciously.

Last Updated : 31/10/2008 Back to Top