Use of existing standards

[ahcvankampen]

Complementary reporting guidelines and standards

To facilitate and improve reproducibility throughout the complete research lifecycle, numerous guidelines, policies, and standards have been developed to guide and facilitate the detailed documentation of all steps.

Reporting guidelines
Many reporting guidelines provide structured tools specifying the minimum information required for specific study types and largely contribute to the transparency, understanding, and reproducibility of a study. Examples include guidelines for clinical trials (CONSORT), diagnostic accuracy studies (STARD), and observational studies (STROBE). Virtually all these minimum information standards and reporting guidelines require the specification of statistical and computational methods that were used in a study. However, precise requirements to specify such methods are often lacking.

Journal guidelines
In addition, an increasing number of scientific journals have their own guidelines. One example is the Nature Reporting Summary that partially relies on existing reporting guidelines and FAIR. Nature also has a specific software and algorithm policy with requirements about availability (e.g., using GitHub or Zenodo), use of an open-source license, and code review. Nature Computational Science and several other Nature journals have adopted Code Ocean, which is a cloud-based platform to share executable code to enable review. The PLOS Computational Biology journal requires that editors and reviewers can access the software, reproduce the results, and run the software on a deposited dataset with provided control parameters. The Science journal TOP guidelines require data and computer code to be available. Interestingly, a study published in 2018 showed that despite these guidelines, many computational studies were not reproducible.

The journal as an information repository
It has also been suggested to rethink the concept of a “journal” as a community-driven information repository containing, among others, the data and software, to enable reproducibility, reuse, and comparisons. In this scenario, academic publishers would have a key role in stimulating (standard-based) approaches to research dissemination. We believe that such an effort should be a joint undertaking of research communities and publishers aiming to improve reproducibility.

Guidelines/standards for computational projects
Clearly, computational projects require their own specific guidelines and standards to guarantee transparency and reproducibility. This was also recognized by the artificial intelligence community, which started initiatives to develop specific AI-oriented guidelines.
To the best of our knowledge, there are not many (practical) reporting guidelines nor standards available for computational research that are routinely used in practice. Nevertheless, there are various initiatives to improve this situation. For example, recently detailed guidelines for a standardized file system structure for scientific data were published by Spreckelsen and co-workers, which inspired the sFSS used in ENCORE.

Other examples of guidelines and standards for computational projects include the application of the FAIR principles to software, the ICERM implementation and archiving criteria for software, the Adaptive Immune Receptor Repertoire (AIRR) software guidelines, the Software Ontology to describe software used to store, manage and analyze data, and the EDAM ontology to describe bioinformatics operations. For simulation-based research there are initiatives like the Minimum Information About a Simulation Experiment (MIASE) guidelines, the corresponding simulation experiment description markup language (SED-ML), COMBINE/OMEX to share and reproduce modeling projects.

How to incorporate in ENCORE?
For the further development of ENCORE, we will need to consider which of these standards are relevant for ENCORE and how to incorporate them in the ENCORE approach. This may require the development of ENCORE specifications for different types of computational projects by different specialized working groups. However, the main challenge we see is the development of software tools to support and use ontologies and standards in the context of ENCORE without introducing much overhead while providing clear benefit.