[EPIC] Deployment framework for community and custom XNAT container service pipelines

[tclose]

Hypothesis Statement

• Funnel Entry Date: 15/6/22
• Epic Name: Framework to run community-generated pipelines in XNAT container service
• Epic Owner: @tclose

Description

For AIS users who design their own workflows or use community-developed pipelines the Container service pipelines deployment framework is a framework for deploying custom and community-generated pipelines to the AIS. Unlike handcrafting Docker images compatible with XNAT's container service, in our solution XNAT-CS-compatible Docker images are fully specified by recipes with minimal boiler plate (particularly in comparison to the XNAT command.JSON). The Docker images are automatically built and deployed to the AIS. Workflow provenance is also stored alongside the generated derivatives for future reference.

Outcomes

• Attract new users to AIS
• Improve the productivity of AIS users
• Improve reproducibility of scientific research using AIS by storing workflow provenance

Leading Indicators

• Number of pipelines run across the network
• Number of community-contributed recipes

Nonfunctional Requirements

• User satisfaction
• Execution performance

Features

Tasks

Preview Give feedback

1. [FEATURE] Auto-build of XNAT-CS Docker images #48
   0 of 1
   deploy-framework mapped pipelines +3
2. [FEATURE] Continuous delivery workflow for pipelines #49
   deploy-framework mapped pipelines +3
   
3. [FEATURE] Pydra wrapper for BIDS apps #47
   feature +1
4. [FEATURE] Implementation of selected BIDS apps #39
   0 of 1
   analysis-design mapped pipelines +3
   
5. [FEATURE] Create deployment testing framework #50
   0 of 11
   deploy-framework mapped pipelines +3
6. [FEATURE] Derivatives XNAT datatype #26
   deploy-framework mapped pipelines +3
7. [FEATURE] Arcana command-line interface #40
   deploy-framework mapped pipelines +3
8. [FEATURE] Implicit conversion between file formats #42
   deploy-framework mapped pipelines +3
9. [FEATURE] Polishing of Arcana package ahead of major release #44
   deploy-framework mapped pipelines +3
10. [FEATURE] Deploy pipeline specs to BIDS apps for wider publication #51
    deploy-framework pipelines to-map +3
11. [FEATURE] Improve test coverage of Arcana package to 80% #56
    deploy-framework mapped pipelines +3
12. [FEATURE] Automatic updating of deployed pipelines at node deployments #120
    deploy-framework mapped pipelines +3
13. [FEATURE] Support licenses on a per site/project basis #122
    deploy-framework feature mapped pipelines +4
14. Independent Stories
15. [STORY] Security screen pipeline inputs for special characters ArcanaFramework/arcana#105
    deploy-framework mid-level pipelines story +4
16. [STORY] Auto add new issues to AIS project #109
    pipelines shallow story +3
    
17. [STORY] Clean up containers after running Unit tests ArcanaFramework/arcana#65
    shallow +1

[github-actions]

Stories

• [STORY] Write unittest to cover listing of blocked tasks in Pydra PR #191
• [STORY] Write unittest to cover a line in the PR patch #192
• [STORY] Sort out issue with images not being cleaned up by deploy build #194
• [STORY] Clean up versioning in auto-generated docs #201
• [STORY] Update to long descriptions don't trigger rebuild of pipeline images #200
• [STORY] Sort out clean up of built docker images in package release #202
• [STORY] Add correct labels to GHCR packages #207
• [STORY] Ensure the description of the XNAT commands include tool name + version #208
• [STORY] Clean up file/directory path handling in Pydra and create PR #211
• [STORY] Port Neurodocker to use latest Click dependency #212
• [STORY] Automatically update CS images on an AIS node #163
• [STORY] Test launch of pipelines on k8s #264
• [STORY] pydra2app ext command for installing new app to xnat #317
• [STORY] Implement proposed Pydra syntax #318

[github-actions]

Features

• [FEATURE] Add ability to deploy pipelines to Flywheel Gears #209
• [FEATURE] Create deployment testing framework #50
• [FEATURE] Specify resources required by pipelines #229