Tools & Workflows

• The Conda package manager was used for tool packaging in EOSC-Life. It standardises software installation and is already used by many researchers: the Bioconda channel, which collects a majority of tools relevant to the life sciences, now contains more than 8000 packages.
• Singularity and Docker were used for “containerisation” to provide isolated execution
  environments and ensure their portability. Over 10,500 life sciences tools are now available in
  containerized form in the BioContainers registry. Through Continuous Integration, Bioconda packages are made available as Docker and Singularity containers.
• EOSC-Life promoted the use of workflow engines that make use of the above technologies, such as Nextflow and Snakemake, for running workflows on the command line, and the Galaxy platform, as a web-based user-friendly interface for building and running data analysis
  workflows. The project especially demonstrated that Galaxy makes tools and workflows available to non-expert users across many different communities through community instances (see Deliverable 2.2) and through national instances (e.g. usegalaxy.fr, usegalaxy.be, usegalaxy.es)

The Common Workflow Language (CWL) was selected in EOSC-Life as the standard for describing tools and workflows that can be executed by multiple workflow engines such as Nextflow and Snakemake. ELIXIR has invested in the support of CWL. CWL is also used by the EU’s BioExcel Centre of Excellence for Computational Biomolecular Research, and by the IBISBA ESFRI for Industrial Biotechnology.

To make computational workflows FAIR, EOSC-Life created the WorkflowHub, a registry for describing, sharing and publishing computational workflows. By mid-project, 250 workflows had already been registered in the WorkflowHub, covering multiple LS RIs involved in the project across more than 100 research groups and projects.

WorkflowHub makes it easier for users to find and re-use of workflows by leveraging a number of
standards. WP2 contributed to several of these, such as Workflow RO-Crate, a standard for packaging
executable workflows together with auxiliary files and enriched metadata, and Workflow Testing
RO-Crate, an extension of Workflow RO-Crate that adds a formalism to specify metadata related
to workflow testing.

WP2 also contributed tools that can be used to work with these standards, such as
ro-crate-py, a Python library and CLI to create and consume RO-Crates, and repo2rocrate, a
tool to generate Workflow (Testing) RO-Crates out of workflow repositories following community
standards.

Notably, the WorkflowHub has been included as part of the European COVID-19 Data Portal, adopted by Australian BioCommons and RIs outside LS RIs.

To facilitate workflow maintenance, and in particular to help with the automation and monitoring of workflow tests, EOSC-Life WP2 developed the LifeMonitor workflow testing service.

In addition to tools that facilitate the automatic execution of tests on continuous integration
services (e.g. GitHub Actions), LifeMonitor integrates with WorkflowHub, includes a web
application reporting on the status of all tested workflows, and can notify people who express interest when workflows show problems.

Interoperability between WorkflowHub and LifeMonitor is ensured by the adoption of Workflow
(Testing) RO-Crate as a common data exchange format.

The WorkflowHub and EOSC-Life Workflow Collaboratory developed a new metadata framework for
workflows.

• Bioschemas schema.org profiles for Computational Tool, Computational Workflow and
  Formal Parameter provide metadata about a workflow and its tools that are discipline
  independent.
• The Common Workflow Language as a canonical workflow description to accompany
  native workflow definitions – both CWL workflows (which are executable) and Abstract CWL
  (non-executable) description variants. This description presents the structure of workflows,
  composed of tools and external interface, in an interoperable way across workflow languages.
  As of the project end, 33 workflows in other formats used Abstract CWL.
• The EDAM Ontology can be used to add bioinformatics-specific metadata, such as strong typing
  of inputs and outputs, within Abstract CWL and Bioschemas annotations. EDAM can also be used
  to describe the overall workflow Topics and Operations (requested by users) to help find
  workflows. It is possible to search and filter using these properties. Note that the approach is
  not limited to EDAM, e.g. in terms of attribute names. This supports expansion beyond EDAM
  in the future to support non-life-sciences – although EDAM themselves are also currently
  going in that direction – topics are starting to support ecology, mathematics, language,
  physics, chemistry plus many more non-life science terms.
• RO-Crate is a community-developed standardised approach for research output
  packaging with rich metadata. RO-Crate enables users to package executable
  workflows, their components, such as example and test data, abstract CWL, diagrams and
  their documentation. This makes workflows more readily re-usable. RO-Crate is the base unit
  of upload and download at the WorkflowHub. RO-Crates are used to import into and export
  from WorkflowHub and are the currency of exchange between the services of the Workflow
  Collaboratory.
  Three RO-Crate profiles were developed for Workflows:
  o Workflow-RO-Crate which WorkflowHub can register and mint;
  o Workflow-Testing-RO-Crate which includes test information for LifeMonitor;
  o Workflow-Run-RO-Crate which includes provenance information after the execution
  of a workflow.
• Identifiers for all the components: RO-Crates can be metadata-rich bags of identifiers and can
  themselves be assigned permanent identifiers. This enables the user to make full descriptions of a
  computational analysis, extending from input data, over tools and workflows, to the final results.

• EOSC-Life Services that integrate with WorkflowHub include:
  bio.tools39, the ELIXIR registry of software tools, databases and services for bioinformatics and
  the life sciences: workflow entries on WorkflowHub can be annotated with tool identifiers,
  either by automatic parsing of the workflow description, or through a tagging mechanism that
  queries the bio.tools API. Users of WorkflowHub can filter through workflows that use a
  certain tool, and from a workflow entry that has been annotated with tools, they can click
  through to the respective bio.tools entry to see extra metadata. bio.tools can link to
  workflows that use a given tool, to show examples of use.
  ● LifeMonitor40 EOSC-Life service developed by BBMRI-ERIC (Task 2.5) that ensures the correct
  operation of the workflow over time through monitoring and triggering automated workflow
  tests and automated checks on metadata and adherence to best practices on the workflow’s
  source code Git repository. The service supports the maintenance of the Galaxy Intergalactic
  Workflow Collection (IWC) – the curated collection of Galaxy workflows – by monitoring the
  test status of the workflows and, in the future, by automatically publishing new or updated
  IWC workflow releases to WorkflowHub and LifeMonitor.
  ● LS-Login41 enables researchers to use their home organisation credentials or community or
  other identities (e.g. Google, Apple, ORCID, LS ID) to sign in and access data and services they
  need. It also allows service providers to control and manage access rights of their users and
  create different access levels for research groups or international projects.
  ● citation.cff42 is a plain text file format used to describe software or research artefacts such as
  datasets, models, and algorithms, to make it easier for researchers to cite in scholarly
  publications. It is designed to be machine- and human-readable, making it easy to include in
  software repositories, code repositories, or other places where research artefacts are shared.
  By harvesting a citation.cff file from Git repositories, WorkflowHub makes it easier for others
  to cite their workflow, increasing their visibility and impact.
  ● ORCID43 for researcher identifiers

What will happen to these services and technologies after the EOSC-Life project ends?

Further European-wide developments related to the Galaxy platform will continue under the new
EOSC project EuroScienceGateway, which started in September 2022.

EuroScienceGateway will leverage the Galaxy platform, the Pulsar Network (a wide distributed job execution system applied to scale computing power over heterogeneous resources) and FAIR workflow services pioneered in the EOSC-Life cluster (WorkflowHub, Workflow RO-Crate and metadata standards like schema.org).