Questions and Answers, Autism Interventions Evidence Report

Theme 1: why the report relied on evidence from systematic reviews only

A number of questions and comments related to the fact that only evidence from systematic reviews (i.e., a review of individual studies) were included in the umbrella review. The full rationale for this is provided in the report, but in brief it was to ensure we included the highest quality of evidence from across each category of interventions.

We appreciate that there is a wide range of other research literature including qualitative studies of people's goals, preferences, and experiences with interventions; single case experimental design studies that are used to evaluate interventions; and quasi-experimental research which can offer insights. We also explicitly state in the report that the report represents a first step towards evidence-based guidelines. For the next step, which would translate the findings of the research synthesis (the Autism CRC report) to policy and practice, it is critical that there be comprehensive consultation with people with lived experience and practitioners to deliver consensus amongst all stakeholders.

The following is an example of a question raised and our specific responses:

Question: Why was the voice of autistic adult advocates regarding ABA strategies not included in the report?

Answer: The scope of the report was to synthesize the evidence from the scientific literature regarding interventions, including therapeutic benefits and adverse effects. The findings are enclosed in the report. It is important to be clear that this report is a piece to summarise the scientific evidence, and not a Guideline. A Guideline involves clinical wisdom and the voice of people with lived experience. As we wrote in the Discussion (p. 103): This report… “provides a robust foundation of evidence upon which consumers and policy-makers can combine efforts to develop guidelines that ensure the delivery of evidence-based supports to children on the autism spectrum within the Australian community.”

We firmly believe the next step of this research is to engage in a comprehensive, collaborative consultation process with people with lived experience and practitioners to ensure these scientific findings are placed within that important context and consensus is reached amongst all stakeholders.

Theme 2: Independence in completing the review

Several questions and comments raised this issue, including whether the authors were free to complete the report independently, and whether any personal or professional interests could have influenced the way the review was conducted and the findings reported. We welcome these as it is important that each person reading the report can understand the process and backgrounds of the authors and organisations involved.

It is proper scientific practice for researchers to disclose any roles or relationships that are relevant to their work on a project in the interests of full transparency. This is known as a Disclosure of Interests and was completed by the research team prior to starting the project. Furthermore, these disclosures were made publicly available at that point, and continue to be freely available via the Open Science Framework. The document is lengthy because the researchers wanted to provide as much information as possible.

A Conflict of Interest occurs when a researcher is unable to act independently. The research team had complete independence in completing and reporting the research synthesis. Accordingly, they report no conflicts of interest.

Disclosures of interest for each investigator are provided in the report, and also online on the Open Science Framework.

The research team worked independently in undertaking the analysis and producing the report, before submitting it to the client, the NDIA. It was then provided to the Autism CRC Board, Participants and the public. 

There were also questions on the connection between the report and the NDIA’s consultation paper on early interventions for children on the autism spectrum, such as the following example:

Question: How was Autism CRC involved in the development of the NDIA’s consultation paper on early interventions for children with autism, and were you involved in recommendations around funding?

Answer: Autism CRC was not involved in the development of the NDIA’s consultation paper, including the proposed levels of funded support it contains; and the our Autism Interventions Evidence Report did not consider matters of funding. The CRC has released a more detailed clarification of this matter in the article Autism CRC’s Interventions Evidence Report.

Theme 3: The review and publication process

A number of people were interested to learn about how the report came about, from the commissioning of the report, the selection of categories, right through to its publication. The report is deliberately lengthy so that each stage of the process, from the development of the protocol through to the presentation of the findings is clearly documented. This transparency was critical to the process, and allows any interested reader to identify and understand the data that led to the conclusions in the report.

The following are examples of three questions raised and our specific responses:

Question: Are you willing to share the briefing documentation received from the NDIA for this body of work?

Answer: The protocol contained in Appendix D of the report reflects the agreed scope of the evidence review of early interventions.

Question: What methodology was used for the umbrella review?

Answer: The umbrella review was conducted according to the methodology outlined in the JBI manual for evidence synthesis (Aromataris et al., 2020) and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Moher, Liberati, Tetzlaff, Altman, & Group, 2009). The research team’s application of the methodology was assessed and verified as sound by experts at JBI, which is a centre of excellence in research methodologies that support and guide evidence-based health care.

The full protocol for the umbrella review is provided as Appendix D of the report. The protocol was submitted to PROSPERO on 12 June 2020 and published on Open Science Framework on 8 July 2020. This is freely available to download.

Question: How were the findings from the report summarised?

Answer: Text summaries were based on the breadth of positive, inconsistent and null findings across the child and family outcomes. These findings can be viewed in Table 6 of the report (p. 73) and also in the Community Summary of the Umbrella Review.

Theme 4: How the method and findings of our report relate to previous reviews

We received several questions asking about how this review, including the method used and the findings, relate with those of previous systematic reviews and agency-commissioned reviews. We appreciate that differences can be confusing, which is why we completed an umbrella review. Put simply, this is a 'review of reviews' and ideally suited to situations in which there are previous reviews which differ according to aims, methods, and findings. To learn more about umbrella reviews we suggest this resource: Umbrella reviews from JBI Manual for Evidence Synthesis. 

The following are examples of two questions raised and our specific responses:

A previous report from the NDIA recommended between 15 and 25 hours per week of intervention for pre-school aged children with ASD. Are you saying that the conclusion of that report was inconsistent/null, given that that report has been included in your research?

Answer: The previous report in 2016 (referred to in the question) and the current report had different aims and different methods. The previous report sought to summarise general principles of early intervention through a non-systematic review of scientific literature and combine inputs from the sector on practice to provide guidance.

The current study adopted a structured and systematic literature search, and sought to summarise evidence derived from clinical trials. Of the 58 systematic reviews included in the study, 40 have been published in the last five years (i.e., since the 2016 report), so there have been significant additional quality research findings in recent years. It is therefore the most comprehensive description of the current evidence-base.

At a broad level, the review found that there is evidence for a positive effect for certain interventions delivered at ‘low intensity’ (e.g., 1-2 hours a week) and certain interventions delivered at ‘high intensity’ (e.g., 15-25 hours a week). It also found that there was insufficient or inconsistent evidence to determine whether particular delivery characteristics (e.g., individual or group delivery; clinic, home or school setting; amount of intervention) may maximise the effects of intervention and, if so, on which outcomes for which children. There is insufficient evidence to answer these questions at the current moment.

Best practice guidance is developed by combining the research evidence with the community context. As a next step, the report’s comprehensive foundation of current quality research evidence should be combined with similarly comprehensive consultation with people with lived experience and practitioners to provide high-quality consensus guidance on such questions.

Question: Were previous large reviews, like the NAC (National Autism Centre) National Standards Report, included in the report and reflected in your findings?

Answer: Yes. All of the reviews that were included in our research (and the report) are described in Appendix M.

Theme 5: Information or topics included or not included in the report

We received several questions asking whether or not certain measures or interventions were included in the review. The following are examples of two questions raised and our specific responses:

Question: Is there any information on studies working with children specifically in rural and remote communities?

Answer: The report does not contain information specifically about rural and remote communities. There was some information on telehealth, which is more commonly used with individuals living in rural/remote settings. This information is included in the findings of the Umbrella Review.

Question: What kind of quality of life measures are available?

Answer: Very few quality of life measures were used in the studies. We have highlighted this as an urgent priority for future research.

Question: How do the interventions examined relate to professional disciplines, such as speech pathology/therapy, occupational therapy and psychology?

Answer: The report focuses on intervention practices and categories (rather than professional disciplines), which is the typical method through which interventions are evaluated in the research literature. Professional disciplines, such as speech pathology/therapy, occupational therapy and psychology, draw on a range of these intervention practices and techniques in their day-to-day practice. This is described in p. 29 of the report.

Theme 6: Who can provide interventions to children on the autism spectrum and which professions are mentioned in the report

The CRC report includes a narrative review of interventions and their delivery, along with the umbrella review of intervention effects. A small number of questions sought clarification about which professions were mentioned.

The following is an example of one question and our specific response:

Question: Why did Autism CRC not recognise physiotherapists in the report?

Answer: The report and Autism CRC recognises there are a wide variety of clinical professionals who can play a role in supporting children on the autism spectrum. The following statement is on page 43 of the report: “Table 5 provides examples of the professional training, regulation, competencies, and scope of practice for certain disciplines. Similar details for other professions, such as physiotherapy, social work, nursing, and medicine can be obtained from AHPRA or the relevant professional college or society. This table is for illustrative purposes only, and does not present information for all clinical practitioners or educators who are relevant to working with children on the autism spectrum.”

Theme 7: Next steps by the NDIA and their consultation paper, Interventions for Young People on the Autism Spectrum

Several people asked about how the NDIA will use the evidence report to inform policy decisions. Autism CRC is not involved in setting policy for NDIA or other government entities. Our role is to be a national reference body for evidence for best practice.

We also received questions about whether we would be making a submission to the NDIS consultation paper, Interventions for children on the autism spectrum, which closed on Friday 14 May 2021. The following is an example of a question raised and our specific response:

Question: Are you making a submission response to the current NDIS proposal around ASD and proposed levels of funding or is that a conflict?

Answer: Yes, we made a submission to this consultation process. Our submission recommended an Early Intervention Guideline be developed. As previously mentioned, we firmly believe the next step of this research should be to engage in a comprehensive, collaborative consultation process with people with lived experience and practitioners to ensure the scientific findings of our report are placed in context and consensus is reached amongst all stakeholders.

Our submission also recommended further research be carried out to fill the many evidence gaps that currently exist, for example, the impact of interventions on quality of life for children and families.

You can read our full submission to the NDIS consultation process.