There’s Evidence and…Then There's Evidence:
Autism Early Behavioral Treatment
It seems reasonable that treatments paid for by tax payers or private insurance companies should be proven to be effective and safe. That is true whether the treatment is a medication, surgery or behavioral intervention. Exactly what constitutes “evidence” for effective treatment or intervention is less clear. Equally unclear is who determines effectiveness and safety and by what criteria.
We often get the impression that the rules testing new treatments has been handed down by Moses, or maybe his doctor, on a clay tablet, but in fact the notion of “randomized clinical trials” is relatively new in medicine. There were originally two motives for the development of explicit procedures for testing medical substances. First were several cases of disasters resulting from introduction of drugs that produced terrible side effects, one a constituent of an antibiotic that caused the Elixir Sulfanilamide disaster caused by ethylene glycol, the other being thalidomide causing birth defects in Europe.
The second, was the proliferation of a wild-west mentality in the introduction of new drugs without sufficient evidence of efficacy or safety. Something needed to be done to decide safety (most importantly) but also effectiveness, of treatments.
Until the Kefauver-Harris Amendments of 1962 was passed in direct response to the thalidomide disaster in Europe leading to numerous birth defects, there was essential no agreed upon standards for evaluating efficacy or safety of medicines. These Amendments mandated that efficacy, in addition to safety, be established for a new drug and instituted stricter control over testing new drugs. The standards for determining clinical effectiveness was set by physicians who prescribed drugs to treat people with known diseases or other health disorders.
Over the intervening years, the Food and Drug Administration developed standards for testing of new drugs and other medical treatments. Those regulations were determined by committees of physicians who administered those treatments, such as new medications or medical devices, such as cardiac pacemakers and overseen by, in theory, unbiased government officials. In practice, those government officials are subject to intense political pressure from lobbying groups via Congress.
Confounding this equation, is that these evidential determinations are powerfully influenced by political forces having nothing directly to do with the quality of evidence. Physicians who control these determinations have a vested interest in showing the services they provide should be reimbursed. In the case of individuals with autism, those services are largely for pediatric and psychiatric diagnostic assessments and prescribing medications. Physicians do not generally provide the one to one behavioral therapy services that are most beneficial to children with autism in the long run. Psychologists in the sub-discipline, applied behavior anlaysis, generally do not have a place at the table in determining which services are reimbursed and which are not.
Research findings by various practitioner groups are submitted to committees of the American Medical Association, Medicaid and private insurance company committees that review the findings, and arrive at reimbursement decisions. Those decisions clearly favor the services provided by physicians and disadvantage behavior analysts whose services, evidence clearly shows, provide the most effective long-term benefits. In addition, other powerful lobbying groups have succeeded in obtaining reimbursement for services for which there is virtually no evidence of efficacy, such as “sensory integration therapy” which has been lobbied for by professional occupational therapists which can cost $1,870 per year per child with autism via Medicaid and far more through private insurance.
Now we come to the crux of why Evidence Based Medicine is a problem. What is evidence? If “evidence” is defined in such a way that nearly no one who works with individual patients or clients can produce “evidence,” then it is very difficult for ABA/EIBI practitioners and researchers to satisfy the criteria for evidence. Much of the problem stems for the difference in what is considered evidence in a typical clinical drug trial and a typical ABA autism clinical study.
Evidence for the antipsychotic risperidone (Risperdal®): A psychiatrist administers a standard rating scale to a youngster diagnosed with autism, then flips a coin and gives him either a placebo pill or Risperdal for two months and then gives the same rating scale again. Cost of the assessment… next to nothing. Two rating scales, maybe 15-30 minutes each, twice. Study done. If the psychiatrist does this with enough children s/he treats over a year, say 50 kids, a change in rating from before to after treatment of a couple of rating points counts as statistically significant. The evidence for Risperdal indicates this occurs for about half of the children treated, a small improvement in one scale, the Irritability Scale of the Aberrant Behavior Checklist. The FDA committee that evaluates such evidence pats the psychiatrist on the head and says, “Well done, you’ve proven Risperdal has medical benefit,” and approves the drug for treating “irritability” in children and youth with autism. That approval translates directly into dollars for all physicians who treat children and youth with autism because now they can charge for prescribing the drug and monitoring its effects. There have been 3-5 such Randomized Clinical Trials of resperidone in autism. This minimally effective treatment for about half of individuals with autism, with significant adverse side effects, is approved for reimbursement.
Evidence for ABA Therapy in Autism: ABA therapy requires extensive baseline measurement of skills, intellectual, social and language functioning often requiring several hours or at times, days of professional time. Then from 20-40 hours per week of 1:1 therapy is performed, typically for 2-3 years. Throughout this period, the procedures are gradually adjusted as the child’s behavior improves. Then the same battery of tests is re-administered. Rarely is it possible to randomly assign children to ABA therapy or no treatment because parents refuse to have their child receive no treatment. Sometimes it is possible to find a “Business as Usual” comparison, in which case the comparison group receives whatever services most youngsters in that area receive, such as attending special education classes. Those assignments cannot be random, they have to be based on objective criteria for entry and parents usually have strong opinions about which group their child will be in. Because the treatment is so time consuming and invidualized, it takes considerable time (years) to accumulate enough participants for statistical analysis, often years.
Of the currently available 34 replication studies, only two have involved entirely random assignment, largely because parents won’t agree to the possibility of assigning their child to a treatment they believe is less beneficial. The ethical implications are obvious. It would be like asking breast cancer patients to agree to an effective treatment or a placebo, just to prove the effective treatment is really effective. In general, sample sizes have been fairly small, 15-30 children per study, not the 50 to 60 subjects that is typical of a clinical drug trial. Using the criteria established for “Evidence Based Medicine” those are the only two studies that have been considered as providing evidence according to most insurance companies with the conclusion, insufficient evidence is available to determine the effectiveness of EIBI (ABA) therapy for young children with autism.
The most recent quantitative review study by Warren, McPheeters et.al. under contract to the Center for Medicaid and Medicare Services concluded that EIBI “resulted in some improvements in cognitive performance, language skills, and adaptive behavior skills in some young children with ASDs, although the literature is limited by methodologic concerns.” I don’t recall that the conclusions of the Risperdal studies were that “some improvements in irritability were observed in some children with ASDs although the literature is limited by methodologic concerns” though that is what happened.
The Oxford Centre for Evidence Based Medicine Level 1 Criteria which has been used in these clinical reviews as required for convincing evidence of effectiveness of EIBI treatments though the OCEBM clearly states, “The Levels is NOT intended to provide you with a definitive judgment about the quality of evidence.” (Oxford Centre for Evidence Based Medicine, "Explanation of the 2011 Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence (Background Document)." Nonetheless, several reviewers concluded just that, namely that the quality of the evidence based on the OCEBM Level 1 criteria is inadequate. To say this is ludicrous is an understatement.
I suppose health insurance companies will continue to stonewall reimbursing ABA autism services as long as possible. Until they begin losing enough lawsuits that it becomes clear it is no longer in their financial interest to deny essential services to children with autism or the federal government rather than individual states mandates coverage of effective ABA autism services.
Centre for Evidence Based Medicine (2012) Oxford, UK http://www.cebm.net/
Food and Drug Administration ( 2009) Centennial FDA: What We Do; History of the FDA http://www.fda.gov/AboutFDA/WhatWeDo/History/CentennialofFDA/default.htm 05/13/2009; [Accessed 6-23-12]
Lord CE, McGee J (2001) Educating children with autism Committee on Educational Interventions for Children with Autism, National Research Council. Washington DC
Lovaas, OI (1987) Behavioral treatment and normal educational and intellectual functioning in young autistic children. J Consult Clin Psychol 1987 Feb: 55(1): 3-9.
Reichow, B, Wolery, M (2009) Comprehensive synthesis of early intensive behavioral interventions for young children with autism based on the UCLA young autism project model. J Autism Dev Disord 2009 Jan: 39(1): 23-41.
Sallows, GO, Graupner, TD (2005) Intensive behavioral treatment for children with autism: four-year outcome and predictors. Am J Ment Retard 2005 Nov: 110(6): 417-38.
Thompson, T. (2012 in press) Autism Research and Services for Young Children: Progress and Challenges. Journal of Applied Research in Intellectual Disabilities.
Warren, Z., McPheeters, ML et.al. (20111) A systematic review of early intensive intervention for autism spectrum disorders. Pediatrics. 127(5):e1303-11
Working Group on Evidence-Based Medicine (1992) Evidence-based medicine. A new approach to teaching the practice of medicine. Journal of the American Medical Association. 268(17): 2420-5.