There’s Evidence and…Then There's Evidence:
Autism Early Behavioral Treatment
It seems reasonable that treatments paid for by tax payers or private insurance companies should be proven to be effective and safe. That is true whether the treatment is a medication, surgery or behavioral intervention. Exactly what constitutes “evidence” for effective treatment or intervention is less clear. Equally unclear is who determines effectiveness and safety and by what criteria.
We often get the impression that the rules testing new treatments has been handed down by Moses, or maybe his doctor, on a clay tablet, but in fact the notion of “randomized clinical trials” is relatively new in medicine. There were originally two motives for the development of explicit procedures for testing medical substances. First were several cases of disasters resulting from introduction of drugs that produced terrible side effects, one a constituent of an antibiotic that caused the Elixir Sulfanilamide disaster caused by ethylene glycol, the other being thalidomide causing birth defects in Europe.
The second, was the proliferation of a wild-west mentality in the introduction of new drugs without sufficient evidence of efficacy or safety. Something needed to be done to decide safety (most importantly) but also effectiveness, of treatments.
Until the Kefauver-Harris Amendments of 1962 was passed in direct response to the thalidomide disaster in Europe leading to numerous birth defects, there was essential no agreed upon standards for evaluating efficacy or safety of medicines. These Amendments mandated that efficacy, in addition to safety, be established for a new drug and instituted stricter control over testing new drugs. The standards for determining clinical effectiveness was set by physicians who prescribed drugs to treat people with known diseases or other health disorders.
Over the intervening years, the Food and Drug Administration developed standards for testing of new drugs and other medical treatments. Those regulations were determined by committees of physicians who administered those treatments, such as new medications or medical devices, such as cardiac pacemakers and overseen by, in theory, unbiased government officials. In practice, those government officials are subject to intense political pressure from lobbying groups via Congress.
But the motivation for evaluating new treatments has drastically changed. Evidence Based Medicine is now being used as a major cost control mechanism for Federal Medicaid and private health insurance companies. While that may seem reasonable on the surface, the nature of what is considered “evidence” determines the winners and losers in this reimbursement merry-go-round. It is in the insurers’ interest to try to show that whatever evidence exists for more costly services is inadequate, while accepting truly inadequate evidence for other relatively less costly services to the same people. That way the insurance companies can claim they are covering the cost of some services to a grouop of people with given diagnosis (e.g. autism) and hopefully obviate complaints about other denied services.
Confounding this equation, is that these evidential determinations are powerfully influenced by political forces having nothing directly to do with the quality of evidence. Physicians who control these determinations have a vested interest in showing the services they provide should be reimbursed. In the case of individuals with autism, those services are largely for pediatric and psychiatric diagnostic assessments and prescribing medications. Physicians do not generally provide the one to one behavioral therapy services that are most beneficial to children with autism in the long run. Psychologists in the sub-discipline, applied behavior anlaysis, generally do not have a place at the table in determining which services are reimbursed and which are not.
Research findings by various practitioner groups are submitted to committees of the American Medical Association, Medicaid and private insurance company committees that review the findings, and arrive at reimbursement decisions. Those decisions clearly favor the services provided by physicians and disadvantage behavior analysts whose services, evidence clearly shows, provide the most effective long-term benefits. In addition, other powerful lobbying groups have succeeded in obtaining reimbursement for services for which there is virtually no evidence of efficacy, such as “sensory integration therapy” which has been lobbied for by professional occupational therapists which can cost $1,870 per year per child with autism via Medicaid and far more through private insurance.
Confounding this equation, is that these evidential determinations are powerfully influenced by political forces having nothing directly to do with the quality of evidence. Physicians who control these determinations have a vested interest in showing the services they provide should be reimbursed. In the case of individuals with autism, those services are largely for pediatric and psychiatric diagnostic assessments and prescribing medications. Physicians do not generally provide the one to one behavioral therapy services that are most beneficial to children with autism in the long run. Psychologists in the sub-discipline, applied behavior anlaysis, generally do not have a place at the table in determining which services are reimbursed and which are not.
Research findings by various practitioner groups are submitted to committees of the American Medical Association, Medicaid and private insurance company committees that review the findings, and arrive at reimbursement decisions. Those decisions clearly favor the services provided by physicians and disadvantage behavior analysts whose services, evidence clearly shows, provide the most effective long-term benefits. In addition, other powerful lobbying groups have succeeded in obtaining reimbursement for services for which there is virtually no evidence of efficacy, such as “sensory integration therapy” which has been lobbied for by professional occupational therapists which can cost $1,870 per year per child with autism via Medicaid and far more through private insurance.
Since around 1990 insurance companies have consistently argued applied behavior anlaysis therapy is an educational not a medical service (and therefore should not be covered by health insurance) and is experimental, meaning that there is insufficient evidence of its efficacy. The distinction of medical versus educational is based on two main factors. First, is whether the treated condition has a recognized medical diagnosis, in this case it does, ICD 398.0 Autism. The second, is the purpose of the service. If the purpose is to achieve educational goals similar to those taught in schools, e.g. reading, writing and math, then the service would be considered educational. If the goal is to address the causes or symptoms of the recognized medical diagnostic disorder or disability, in this case symptoms of autism, i.e. communication deficit, lack of social awareness and social skill, rigid non-functional behavioral routines and emotional outbursts and insistence on sameness, then the service can properly be considered medical. Recently the U.S. Office of Personnel Management (OPM) which regulates the Federal Employees Health Benefits (FEHB) Program, concluded that indeed ABA autism services qualify as medical and not educational services, which would seem to settle that argument, at least for now.
Having lost that battle, the insurance companies and Federal Medicaid are now clinging to their last hope in their efforts to avoid reimbursing the cost of this essential service as long as possible, they are arguing the empirical evidence for EIBI efficacy is still inadequate, hence the “Evidence Based Medicine” argument. Though nearly all professional medical and allied health professional groups have endorsed ABA therapy as state of the science in autism treatment, the insurance companies continue to deny reimbursement.
Now we come to the crux of why Evidence Based Medicine is a problem. What is evidence? If “evidence” is defined in such a way that nearly no one who works with individual patients or clients can produce “evidence,” then it is very difficult for ABA/EIBI practitioners and researchers to satisfy the criteria for evidence. Much of the problem stems for the difference in what is considered evidence in a typical clinical drug trial and a typical ABA autism clinical study.
Evidence for the antipsychotic risperidone (Risperdal®): A psychiatrist administers a standard rating scale to a youngster diagnosed with autism, then flips a coin and gives him either a placebo pill or Risperdal for two months and then gives the same rating scale again. Cost of the assessment… next to nothing. Two rating scales, maybe 15-30 minutes each, twice. Study done. If the psychiatrist does this with enough children s/he treats over a year, say 50 kids, a change in rating from before to after treatment of a couple of rating points counts as statistically significant. The evidence for Risperdal indicates this occurs for about half of the children treated, a small improvement in one scale, the Irritability Scale of the Aberrant Behavior Checklist. The FDA committee that evaluates such evidence pats the psychiatrist on the head and says, “Well done, you’ve proven Risperdal has medical benefit,” and approves the drug for treating “irritability” in children and youth with autism. That approval translates directly into dollars for all physicians who treat children and youth with autism because now they can charge for prescribing the drug and monitoring its effects. There have been 3-5 such Randomized Clinical Trials of resperidone in autism. This minimally effective treatment for about half of individuals with autism, with significant adverse side effects, is approved for reimbursement.
Evidence for ABA Therapy in Autism: ABA therapy requires extensive baseline measurement of skills, intellectual, social and language functioning often requiring several hours or at times, days of professional time. Then from 20-40 hours per week of 1:1 therapy is performed, typically for 2-3 years. Throughout this period, the procedures are gradually adjusted as the child’s behavior improves. Then the same battery of tests is re-administered. Rarely is it possible to randomly assign children to ABA therapy or no treatment because parents refuse to have their child receive no treatment. Sometimes it is possible to find a “Business as Usual” comparison, in which case the comparison group receives whatever services most youngsters in that area receive, such as attending special education classes. Those assignments cannot be random, they have to be based on objective criteria for entry and parents usually have strong opinions about which group their child will be in. Because the treatment is so time consuming and invidualized, it takes considerable time (years) to accumulate enough participants for statistical analysis, often years.
Now we come to the crux of why Evidence Based Medicine is a problem. What is evidence? If “evidence” is defined in such a way that nearly no one who works with individual patients or clients can produce “evidence,” then it is very difficult for ABA/EIBI practitioners and researchers to satisfy the criteria for evidence. Much of the problem stems for the difference in what is considered evidence in a typical clinical drug trial and a typical ABA autism clinical study.
Evidence for the antipsychotic risperidone (Risperdal®): A psychiatrist administers a standard rating scale to a youngster diagnosed with autism, then flips a coin and gives him either a placebo pill or Risperdal for two months and then gives the same rating scale again. Cost of the assessment… next to nothing. Two rating scales, maybe 15-30 minutes each, twice. Study done. If the psychiatrist does this with enough children s/he treats over a year, say 50 kids, a change in rating from before to after treatment of a couple of rating points counts as statistically significant. The evidence for Risperdal indicates this occurs for about half of the children treated, a small improvement in one scale, the Irritability Scale of the Aberrant Behavior Checklist. The FDA committee that evaluates such evidence pats the psychiatrist on the head and says, “Well done, you’ve proven Risperdal has medical benefit,” and approves the drug for treating “irritability” in children and youth with autism. That approval translates directly into dollars for all physicians who treat children and youth with autism because now they can charge for prescribing the drug and monitoring its effects. There have been 3-5 such Randomized Clinical Trials of resperidone in autism. This minimally effective treatment for about half of individuals with autism, with significant adverse side effects, is approved for reimbursement.
Evidence for ABA Therapy in Autism: ABA therapy requires extensive baseline measurement of skills, intellectual, social and language functioning often requiring several hours or at times, days of professional time. Then from 20-40 hours per week of 1:1 therapy is performed, typically for 2-3 years. Throughout this period, the procedures are gradually adjusted as the child’s behavior improves. Then the same battery of tests is re-administered. Rarely is it possible to randomly assign children to ABA therapy or no treatment because parents refuse to have their child receive no treatment. Sometimes it is possible to find a “Business as Usual” comparison, in which case the comparison group receives whatever services most youngsters in that area receive, such as attending special education classes. Those assignments cannot be random, they have to be based on objective criteria for entry and parents usually have strong opinions about which group their child will be in. Because the treatment is so time consuming and invidualized, it takes considerable time (years) to accumulate enough participants for statistical analysis, often years.
Ivar Lovaas published his landmark study in 1987 showing that 47% of the children receiving three years of 40 hours per week of ABA therapy functioned in the typical range on several social, language and cognitive measures, and were successfully included in regular education classes. The comparison group (not randomly assigned) had 12% showing such an outcome. The basic findings of this study have been replicated over 30 times in published reports in credible peer reviewed journals.
Of the currently available 34 replication studies, only two have involved entirely random assignment, largely because parents won’t agree to the possibility of assigning their child to a treatment they believe is less beneficial. The ethical implications are obvious. It would be like asking breast cancer patients to agree to an effective treatment or a placebo, just to prove the effective treatment is really effective. In general, sample sizes have been fairly small, 15-30 children per study, not the 50 to 60 subjects that is typical of a clinical drug trial. Using the criteria established for “Evidence Based Medicine” those are the only two studies that have been considered as providing evidence according to most insurance companies with the conclusion, insufficient evidence is available to determine the effectiveness of EIBI (ABA) therapy for young children with autism.
The most recent quantitative review study by Warren, McPheeters et.al. under contract to the Center for Medicaid and Medicare Services concluded that EIBI “resulted in some improvements in cognitive performance, language skills, and adaptive behavior skills in some young children with ASDs, although the literature is limited by methodologic concerns.” I don’t recall that the conclusions of the Risperdal studies were that “some improvements in irritability were observed in some children with ASDs although the literature is limited by methodologic concerns” though that is what happened.
The Oxford Centre for Evidence Based Medicine Level 1 Criteria which has been used in these clinical reviews as required for convincing evidence of effectiveness of EIBI treatments though the OCEBM clearly states, “The Levels is NOT intended to provide you with a definitive judgment about the quality of evidence.” (Oxford Centre for Evidence Based Medicine, "Explanation of the 2011 Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence (Background Document)." Nonetheless, several reviewers concluded just that, namely that the quality of the evidence based on the OCEBM Level 1 criteria is inadequate. To say this is ludicrous is an understatement.
I suppose health insurance companies will continue to stonewall reimbursing ABA autism services as long as possible. Until they begin losing enough lawsuits that it becomes clear it is no longer in their financial interest to deny essential services to children with autism or the federal government rather than individual states mandates coverage of effective ABA autism services.
Of the currently available 34 replication studies, only two have involved entirely random assignment, largely because parents won’t agree to the possibility of assigning their child to a treatment they believe is less beneficial. The ethical implications are obvious. It would be like asking breast cancer patients to agree to an effective treatment or a placebo, just to prove the effective treatment is really effective. In general, sample sizes have been fairly small, 15-30 children per study, not the 50 to 60 subjects that is typical of a clinical drug trial. Using the criteria established for “Evidence Based Medicine” those are the only two studies that have been considered as providing evidence according to most insurance companies with the conclusion, insufficient evidence is available to determine the effectiveness of EIBI (ABA) therapy for young children with autism.
The most recent quantitative review study by Warren, McPheeters et.al. under contract to the Center for Medicaid and Medicare Services concluded that EIBI “resulted in some improvements in cognitive performance, language skills, and adaptive behavior skills in some young children with ASDs, although the literature is limited by methodologic concerns.” I don’t recall that the conclusions of the Risperdal studies were that “some improvements in irritability were observed in some children with ASDs although the literature is limited by methodologic concerns” though that is what happened.
The Oxford Centre for Evidence Based Medicine Level 1 Criteria which has been used in these clinical reviews as required for convincing evidence of effectiveness of EIBI treatments though the OCEBM clearly states, “The Levels is NOT intended to provide you with a definitive judgment about the quality of evidence.” (Oxford Centre for Evidence Based Medicine, "Explanation of the 2011 Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence (Background Document)." Nonetheless, several reviewers concluded just that, namely that the quality of the evidence based on the OCEBM Level 1 criteria is inadequate. To say this is ludicrous is an understatement.
I suppose health insurance companies will continue to stonewall reimbursing ABA autism services as long as possible. Until they begin losing enough lawsuits that it becomes clear it is no longer in their financial interest to deny essential services to children with autism or the federal government rather than individual states mandates coverage of effective ABA autism services.
References
Centre for Evidence Based Medicine (2012) Oxford, UK http://www.cebm.net/
Food and Drug Administration ( 2009) Centennial FDA: What We Do; History of the FDA http://www.fda.gov/AboutFDA/WhatWeDo/History/CentennialofFDA/default.htm 05/13/2009; [Accessed 6-23-12]
Lord CE, McGee J (2001) Educating children with autism Committee on Educational Interventions for Children with Autism, National Research Council. Washington DC
Lovaas, OI (1987) Behavioral treatment and normal educational and intellectual functioning in young autistic children. J Consult Clin Psychol 1987 Feb: 55(1): 3-9.
Reichow, B, Wolery, M (2009) Comprehensive synthesis of early intensive behavioral interventions for young children with autism based on the UCLA young autism project model. J Autism Dev Disord 2009 Jan: 39(1): 23-41.
Sallows, GO, Graupner, TD (2005) Intensive behavioral treatment for children with autism: four-year outcome and predictors. Am J Ment Retard 2005 Nov: 110(6): 417-38.
Thompson, T. (2012 in press) Autism Research and Services for Young Children: Progress and Challenges. Journal of Applied Research in Intellectual Disabilities.
Warren, Z., McPheeters, ML et.al. (20111) A systematic review of early intensive intervention for autism spectrum disorders. Pediatrics. 127(5):e1303-11
Working Group on Evidence-Based Medicine (1992) Evidence-based medicine. A new approach to teaching the practice of medicine. Journal of the American Medical Association. 268(17): 2420-5.