Open science, power and reproducibility in neurodevelopmental disorder research workshop
Held at the University of Surrey in June 2019, the workshop was the second part of a two-day Neurodevelopmental Disorders Annual Seminar (NDAS19) showcasing the progress of research in neurodevelopmental disorders in the UK.
Morning sessions introduced many open science and reproducibility concepts in the context of neurodevelopmental disorder research and beyond. Hannah Hobson discussed the difficulty of achieving substantial power due to challenges in the recruitment of participants with different developmental presentations. She highlighted that the mean power for Neurodevelopmental disorder research is 0.613.
Another variable that is important to consider is precision. Effect sizes are often large in neurodevelopmental disorder research, which means that power is often high even with small group sizes. However, small group sizes can have poor precision. Hannah emphasised that if the confidence interval (a measure of precision) is larger than the effect size, this is cause for concern. One way to compensate for small group sizes is to increase the number of trials. However, this can also be difficult for these atypical groups. Other suggestions such as collaborative multi-lab studies were also discussed.
Harry Purser gave a general introduction to Bayesian statistics as an alternative or complementary way of analysing and understanding data. Relevant to the issue of low power and low precision discussed by Hannah, Harry explained how Bayesian statistics can be used to differentiate between a true null effect and an underpowered study. Lorcan Kenny, a representative of Autistica UK, presented the charity’s views with respect to open and reproducible science practices. Lorcan emphasised that for research to have impact (e.g. on policy, clinical guidelines, evidence-based practice), it must be reproducible. Autistica UK are working to support the use of such methods from the funding application stage throughout the research process, a topic which incited much audience discussion.
Relevant to participant numbers, Lorcan introduced the Discover Network, which provides access (subject to ethical approval and open research practices) to a database of 12,000 members, including autistic people who are willing to take part in research. The Discover Network, among other events, also provides training on reproducible methods and open research. Finally, Hannah Hobson discussed pre-registrations and registered reports and their application in developmental research. She highlighted that, as yet, there are no published registered reports in developmental psychology.
Following the talks and discussions, the practical workshop sessions gave attendees the opportunity to delve into Bayesian statistics using the open-source software JASP. Attendees also opened their own Open Science Framework (OSF) accounts and explored the platform’s many functions. To end the day, everyone made pirate-plots in R. Pirate plots (similar to violin plots) represent all data points, in addition to the group mean and variance. This is particularly relevant with reference to current emphasis on individual differences and variability in neurodevelopmental disorder research.
All resources from the meeting are now accessible on the NDAS workshop OSF page.
Written by Emily Farran and Marta Topor.