Implement a subset of the CWL Draft 3 tool format.

[jmchilton]

CWL Support:

• Implemented integer, long, float, double, boolean, and File parameters, and arrays thereof as well ["null", <simple_type>] union parameters and Any-type parameters. More complex unions of datatypes are stil unsupported (unions of two or more non-null parameters, unions of ["null", Any], etc...).
• Draft 3 CreateFileRequirements are supported (see the test_rename test case).
• Draft 3 InlineJavascriptRequirement are support to define output files (see test_cat3 test case).
• EnvVarRequirements are supported (see the test_env_tool1 and test_env_tool2 test cases).
• Secondary files are supported at least partially, see the index1 and showindex1 CWL tools created to verify this as well as the test_index1 test case.
• Docker integration is only partial (simple docker pull is supported) - so cat3-tool.cwl works for example. Full semantics of CWL docker support has yet to be implemented. The remaining work is straight-forward and trackd in the meta-issue More Refined Docker Support for Tools galaxyproject/galaxy#1684.
• Expression tools are supported (see parseInt-tool test case).
• Non-File CWL outputs are represented as expression.json files. Traditionally Galaxy hasn't supported non-File outputs from tools but CWL Galaxy has work in progress on bringing native Galaxy support for such outputs Add Expression Tools to Galaxy #27.

Implementation Notes:

• CWL secondary files are stored in __secondary_files__ directory in the dataset's extra_files_path directory.
• The tool execution API has been extended to add a inputs_representation parameter that can be set to "cwl" now. The cwl representation for running tools corresonding to the CWL job json format with {class: "File: path: "/path/to/file"} inputs replaced with {"src": "hda", "id": "<dataset_id>"}. Code for building these requests for CWL job json is available in the test class.
• Since the CWL <-> Galaxy parameter translation may change over time, for instance if Galaxy develops or refines parameter classes - CWL state and CWL state version is tracked in the database and hopefully for reruns, etc... we could update the Galaxy state from an older version to a new one.
• CWL allows output parameters to be either File or non-File and determined at runtime, so galaxy.json is used to dynamically adjust output extension as needed for non-File parameters.

Implementation Description:

The reference implementation Python library (mainly developed by Peter Amstutz - https://github.com/common-workflow-language/common-workflow-language/tree/master/reference) is used to load tool files ending with .json or .cwl and proxy objects are created to adapt these tools to Galaxy representations. In particular input and output descriptions are loaded from the tool.

When the tool is submitted, a special specialized tool class is used to build a cwltool compatible job description from the supplied Galaxy inputs and the CWL reference implementation is used to generate a CWL reference implementation Job object. A command-line is generated from this Job object.

As a result of this - Galaxy largely does not need to worry about the details of command-line adapters, expressions, etc....

Galaxy writes a description of the CWL job that it can reload to the job working directory. After the process is complete (on the Galaxy compute server, but outside the Docker container) this representation is reloaded and the dynamic outputs are discovered and moved to fixed locations as expected by Galaxy. CWL allows for much more expressive output locations than Galaxy, for better or worse, and this step uses cwltool to adapt CWL to Galaxy outputs.

Currently all File outputs are sniffed to determined a Galaxy datatype, CWL draft 3 allows refinement on this and this remains work to be done.

1. CWL should support EDAM declaration of types and Galaxy should provide a mapping to core datasets to skip sniffing is types are found.
2. For finer grain control within Galaxy, extensions to CWL should allow setting actual Galaxy output types on outputs. (Distinction between fastq and fastqsanger in Galaxy is very important for instance.)

Testing:

% git clone https://github.com/common-workflow-language/galaxy.git
% git checkout cwl
% cd galaxy
% virtualenv .venv
% . .venv/bin/activate
% pip install cwltool

Start Galaxy.

% GALAXY_RUN_WITH_TEST_TOOLS=1 run.sh --reload

Open http://localhost:8080/ and see CWL test tools (along with all Galaxy test tools) in left hand tool panel.

To go a step further and actually run CWL jobs within their designated Docker containers, copy the following minimal Galaxy job configuration file to config/job_conf.xml. (Adjust the docker_sudo parameter based on how you execute Docker).

https://gist.github.com/jmchilton/3997fa471d1b4c556966

Run API tests demonstrating the various CWL demo tools with the following command.

./run_tests.sh -api test/api/test_tools_cwl.py

Issues

Work remaining on CWL support for Galaxy is tracked at https://github.com/common-workflow-language/galaxy/issues.

[jmchilton]

This will be rebased frequently but this standing PR should represent the current progress of the cwl branch. Any help would be appreciated, will work on building a TODO list of what needs to be done.

[jmchilton]

This PR has be superseded by #47 - work will now be tracked in the cwl-1.0 branch.