Another Autism Miracle: Something Smells Fishy
Travis Thompson, PhD
Dr. Michael Leon shown right.
A recently published study raises autism magic to a new level. It is yet one more attempt to add credibility to sensory integration, this time a poorly done study conducted by two experts in the anatomy and physiology of smell. I don’t doubt the researchers motivation is pure and well intentioned, but their study is not well designed and poses very significant problems leading to misleading results and even more beguiling interpretation. The paper is being cited all over the internet as showing sensory experience reduces autism symptoms, but the study shows no such thing as far as I can tell. If there are any treatment effects at all, they are extremely small and likely of no practical significance whatsoever.
A VERY OLD IDEA: The article reports that exposing children with autism to a variety of stimulating sensory experiences over a period of time, significantly increases their cognitive skills and reduces their autism symptoms. This idea dates back to Eduardo Seguin and Maria Montessori who believed that the deficits of children with disabilities could be overcome by their sensory experiences. In the 1960s, it was discovered that brain weights of rats exposed to supplementary sensory experiences were greater than those without those supplementary experiences. Research on neuroplasticity by William Greenough and associates also suggest that supplementary sensory experience can increase synapse formation. None of these studies were directly relevant to autism. None showed symptoms similar to those in autism could be reduced by such sensory experiences. Occupational therapist Jean Ayres popularized the notion that supplementary stimulatory experiences would reduce autism symptoms, which has been repeatedly found not to be true in controlled studies. Major professional societies have stated that sensory integration and auditory integration training are not effective autism treatments and warned parents and practitioners against depending on them rather than proven effective methods. Numerous studies on ABA intervention have used comparison groups involving sensory integration, generally shown little or no effect of sensory integration but very large effects of ABA treatment.
Now in a new paper, Woo and Leon in the journal Behavioral Neuroscience May 20, 2013 have conducted a study with a comparison group allegedly showing that supplementary sensory experience actually does reduce autism symptoms and increase cognitive functioning. There are numerous problems with this study which render such an interpretation seriously questionable.
MISLEADING BACKGROUND: In the introduction of the article the authors site a study by Charles Nelson and colleagues with institutionalized Romanian refugee children who had experienced extensive social, language and sensory deprivation, later shown to exhibit some autism symptoms. If the children had autism, it was likely a very different condition from that which spontaneously occurs in autism. Woo and Leon state that the Nelson study showed that these symptoms could be reduced or eliminated by supplementary experience. That is not at all what the Nelson study showed. According to the authors of the Nelson et.al. study, “The cognitive outcome of children who remained in the institution was markedly below that of never-institutionalized children and children taken out of the institution and placed into foster care.” Being placed in foster care, or living their entire lives outside of an institution, involved a complex array of social and language experiences totally unlike the sensory experiences in the present study. They did not merely have supplementary sensory experiences, not at all. To compare them is very misleading.
DIAGNOSIS AND SEVERITY: The study included children referred to the authors as having autism. After the children were identified for the study, an ADOS was administered. They report, “Diagnoses of autism were confirmed using the Autism Diagnostic Observation Schedule (ADOS). The communication and social interactions ADOS scores were combined, and to qualify for our study, the total score had to be within the ADOS autism classification and not simply within the autism spectrum classification.’ Depending on the ADOS module used for testing, scores can vary widely and still fall within the autism range. Without presenting actual numerical scores, it is impossible to know whether there were no relevant differences in baseline autism characteristics of the groups of children. Typically for a study of this type the ADI-R is used that can be quantified, which the ADOS-R is not designed to be used to measure severity (though there are now complex ways to do that).
WHAT IS STANDARD CARE? The children were then randomly assigned to either a sensorimotor enrichment group plus “standard care” (13 children), or a group receiving “standard care” alone (15 children), matched for age and autism severity, as assessed by their Childhood Autism Rating Scale (CARS) score. It is stated that most of the “standard of care” group children were receiving behavioral intervention, but there is no report of the kind or intensity of such intervention, which means it could have ranged from a few hours of week in preschool to 30 hours a week of home based therapy. Some had speech therapy others none. Without that information it is impossible to know whether the treatments were comparable. Only a few children receiving an effective treatment in one group could easily have yielded a difference of the magnitude reported. Several well-documented studies have shown that with children with more severe autism, less than 20 hours per week of structured ABA therapy is likely to engender less improvement.
INCONSEQUENTIAL DIFFERENCE: The primary autism severity measure used was the Childhood Autism Rating Scale, which is a screening test for autism (not diagnostic) nor a test that is a valid measure of severity. Most credible studies of autism treatment outcomes use the ADOS or ADI-R as outcome measures. The CARS is most often used when the magnitude of improvements are very large, which was not the case in this study. Reported improvements on the CARS in the present study was 2.8 scale points. That is an extremely small clinical difference. For example, the Very low level of Autism-Related symptoms CARS range is from 25-39 and for Low Level 40-44. A 2.8 point difference would represent no clinically significant improvement within either of these CARS ranges.
STATISTICAL ERROR: The authors state, “However, greater improvement in behavioral responses was observed in the enriched group relative to the control group in 11 out of the 15 items,” reflects a lack of understanding of statistical data. A numerical score difference cannot be described as “greater improvement,” if it is not statistically significant. What they mean is that there was a directional difference, with the supplementary group having a numerical score that was greater on 11 scales. If there were truly no differences one would expect by chance 7 of 15 scale scores would be higher, which means that they found four more scales scored lower in the supplementary stimulation group than would be expected by chance, but the magnitude of those differences was non-significant. The authors report, ‘Moreover, 42% of the enriched group and 7% of the standard care group had what we considered to be a clinically significant improvement of at least 5 points on that scale (Figure 2; two-sample test for proportions, p _ .03).” Whether that would be considered reliably clinically significant improvement depends on where within the range of scores that difference fell.
NO LANGUAGE DIFFERENCE: The Expressive One-Word Picture Vocabulary Test revealed no difference in outcomes between the groups. It’s worth comparing the communication findings with this study, with those of one of the typical ABA outcome studies by Sallows and Graupner (2005). Sallows and Graupner study reported an increase among rapid learners from 49.9 to 85.7 standard score points on the expressive language subscale of the WISC and 60.8 to 105.09 on the Vineland composite communication scale. The present study found no significant improvements in the One Word Vocabulary test scores.
SUMMARY: This paper is seriously flawed and is certainly not cause for everyone to rush out and expose their children to smells, lights and sounds expecting that it would overcome their autism symptoms. Shame on the science press for failing to read the article carefully (Woo, CC and Leon, M [2013] Environmental Enrichment as an Effective Treatment for Autism: A Randomized Controlled Trial. Behavioral Neuroscience, May 20 , 2013, no pagination) Dr. Woo is an assistant project scientist and Dr. Leon is Professor of neurobiology and behavior at the University of California at Irvine. Both are experts on neuroanatomy of olfaction)
A VERY OLD IDEA: The article reports that exposing children with autism to a variety of stimulating sensory experiences over a period of time, significantly increases their cognitive skills and reduces their autism symptoms. This idea dates back to Eduardo Seguin and Maria Montessori who believed that the deficits of children with disabilities could be overcome by their sensory experiences. In the 1960s, it was discovered that brain weights of rats exposed to supplementary sensory experiences were greater than those without those supplementary experiences. Research on neuroplasticity by William Greenough and associates also suggest that supplementary sensory experience can increase synapse formation. None of these studies were directly relevant to autism. None showed symptoms similar to those in autism could be reduced by such sensory experiences. Occupational therapist Jean Ayres popularized the notion that supplementary stimulatory experiences would reduce autism symptoms, which has been repeatedly found not to be true in controlled studies. Major professional societies have stated that sensory integration and auditory integration training are not effective autism treatments and warned parents and practitioners against depending on them rather than proven effective methods. Numerous studies on ABA intervention have used comparison groups involving sensory integration, generally shown little or no effect of sensory integration but very large effects of ABA treatment.
Now in a new paper, Woo and Leon in the journal Behavioral Neuroscience May 20, 2013 have conducted a study with a comparison group allegedly showing that supplementary sensory experience actually does reduce autism symptoms and increase cognitive functioning. There are numerous problems with this study which render such an interpretation seriously questionable.
MISLEADING BACKGROUND: In the introduction of the article the authors site a study by Charles Nelson and colleagues with institutionalized Romanian refugee children who had experienced extensive social, language and sensory deprivation, later shown to exhibit some autism symptoms. If the children had autism, it was likely a very different condition from that which spontaneously occurs in autism. Woo and Leon state that the Nelson study showed that these symptoms could be reduced or eliminated by supplementary experience. That is not at all what the Nelson study showed. According to the authors of the Nelson et.al. study, “The cognitive outcome of children who remained in the institution was markedly below that of never-institutionalized children and children taken out of the institution and placed into foster care.” Being placed in foster care, or living their entire lives outside of an institution, involved a complex array of social and language experiences totally unlike the sensory experiences in the present study. They did not merely have supplementary sensory experiences, not at all. To compare them is very misleading.
DIAGNOSIS AND SEVERITY: The study included children referred to the authors as having autism. After the children were identified for the study, an ADOS was administered. They report, “Diagnoses of autism were confirmed using the Autism Diagnostic Observation Schedule (ADOS). The communication and social interactions ADOS scores were combined, and to qualify for our study, the total score had to be within the ADOS autism classification and not simply within the autism spectrum classification.’ Depending on the ADOS module used for testing, scores can vary widely and still fall within the autism range. Without presenting actual numerical scores, it is impossible to know whether there were no relevant differences in baseline autism characteristics of the groups of children. Typically for a study of this type the ADI-R is used that can be quantified, which the ADOS-R is not designed to be used to measure severity (though there are now complex ways to do that).
WHAT IS STANDARD CARE? The children were then randomly assigned to either a sensorimotor enrichment group plus “standard care” (13 children), or a group receiving “standard care” alone (15 children), matched for age and autism severity, as assessed by their Childhood Autism Rating Scale (CARS) score. It is stated that most of the “standard of care” group children were receiving behavioral intervention, but there is no report of the kind or intensity of such intervention, which means it could have ranged from a few hours of week in preschool to 30 hours a week of home based therapy. Some had speech therapy others none. Without that information it is impossible to know whether the treatments were comparable. Only a few children receiving an effective treatment in one group could easily have yielded a difference of the magnitude reported. Several well-documented studies have shown that with children with more severe autism, less than 20 hours per week of structured ABA therapy is likely to engender less improvement.
INCONSEQUENTIAL DIFFERENCE: The primary autism severity measure used was the Childhood Autism Rating Scale, which is a screening test for autism (not diagnostic) nor a test that is a valid measure of severity. Most credible studies of autism treatment outcomes use the ADOS or ADI-R as outcome measures. The CARS is most often used when the magnitude of improvements are very large, which was not the case in this study. Reported improvements on the CARS in the present study was 2.8 scale points. That is an extremely small clinical difference. For example, the Very low level of Autism-Related symptoms CARS range is from 25-39 and for Low Level 40-44. A 2.8 point difference would represent no clinically significant improvement within either of these CARS ranges.
STATISTICAL ERROR: The authors state, “However, greater improvement in behavioral responses was observed in the enriched group relative to the control group in 11 out of the 15 items,” reflects a lack of understanding of statistical data. A numerical score difference cannot be described as “greater improvement,” if it is not statistically significant. What they mean is that there was a directional difference, with the supplementary group having a numerical score that was greater on 11 scales. If there were truly no differences one would expect by chance 7 of 15 scale scores would be higher, which means that they found four more scales scored lower in the supplementary stimulation group than would be expected by chance, but the magnitude of those differences was non-significant. The authors report, ‘Moreover, 42% of the enriched group and 7% of the standard care group had what we considered to be a clinically significant improvement of at least 5 points on that scale (Figure 2; two-sample test for proportions, p _ .03).” Whether that would be considered reliably clinically significant improvement depends on where within the range of scores that difference fell.
NO LANGUAGE DIFFERENCE: The Expressive One-Word Picture Vocabulary Test revealed no difference in outcomes between the groups. It’s worth comparing the communication findings with this study, with those of one of the typical ABA outcome studies by Sallows and Graupner (2005). Sallows and Graupner study reported an increase among rapid learners from 49.9 to 85.7 standard score points on the expressive language subscale of the WISC and 60.8 to 105.09 on the Vineland composite communication scale. The present study found no significant improvements in the One Word Vocabulary test scores.
SUMMARY: This paper is seriously flawed and is certainly not cause for everyone to rush out and expose their children to smells, lights and sounds expecting that it would overcome their autism symptoms. Shame on the science press for failing to read the article carefully (Woo, CC and Leon, M [2013] Environmental Enrichment as an Effective Treatment for Autism: A Randomized Controlled Trial. Behavioral Neuroscience, May 20 , 2013, no pagination) Dr. Woo is an assistant project scientist and Dr. Leon is Professor of neurobiology and behavior at the University of California at Irvine. Both are experts on neuroanatomy of olfaction)