3.24.0

This is the last version of Toil that supports python2.7 and allows the command word "cwltoil" (now "toil-cwl-runner").

• Update ssh and docker setup files. #2958
• Update EC2 instance types. #2962
• Source toil environment as a configuration file. #2960
• Scan for the latest caching database instead of using a hard link. #2948
• Specify py3.6 or py3.x instead of just py3 in appliance. #2966

Bug Fixes

• Fix database being locked when trying to write in the CachingFileStore. #2955
• Fix py3 spot market scaling. #2971

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @jeffrey856!

3.23.1

Functionally the same as Toil Version 3.22.0.

3.22.0

• Make an appliance for Python 3. #2937
• Use status to properly get the HTTP status code from boto. #2934
• Use atomic file copy when writing to posix file system. #2912
• Improve Kubernetes job timing. #2922
• Increase SQLite DB timeout to 60s. #2913
• Consider pods that are very close to their memory limits to be stuck. #2895
• Constantly reload Kubernetes config. #2869
• Up the disk to the default as well as the memory. #2880
• Tolerate unnecessary bucket cleanup. #2894
• Add checksum verification for AWS job store. #2889
• Change python3.6 ppa to deadsnakes. #2888
• Change Kubernetes job sizing logic. #2877
• Added env var TOIL_KUBERNETES_OWNER. #2872
• Raise job limits to be more than their requests. #2868
• Remove all Azure support from toil. #2860

Bug Fixes

• Exclude .nfs files from deferred function search. #2930
• Fix cpu_count() to actually read quotas. #2933
• Compile http-parser so that it doesn't break pip dependencies. #2940
• Use an http-parser for pymesos that builds on 3.7+. #2931
• Fix botocredAdapter. #2904
• Make sure to commit when giving away files. #2914
• Actually call the implemented size polling method. #2916
• Set Docker vars back to ?=. #2908
• LSF: Don't check for Status , as that breaks the error code finding. #2863

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @jeffrey856, @diekhans!

3.21.0

• Add kubernetes watch. #2848
• Add service account configuration for Kubernetes. #2851
• Head object 404 with encryption. #2853
• Tests for Kubernetes. #2838
• Turn usernames into non-spooky Kubernetes name components. #2836
• Make examples a symlink to make examples discoverable. #2834
• Kubernetes BatchSystem Support. #2786
• Specify GCE coreos image by family name instead of specific version. #2829
• Add toil-cwl-runner --provenance. #2785
• Revise caching more. #2770
• Bump psutil >= 3.0.1, <6. #2790
• Add "failed_" prefix to failed logs. #2792
• Add --writeLogsFromAllJobs option. #2781
• Expose --default-container to toil-cwl-runner. #2784
• Update cwltool from 1.0.20190228155703 -> 1.0.20190815141648. #2764
• Pass jobID to Torque Wrapper. #2762
• Limit Length of Log Names. #2755
• Add CWL options --strict-memory-limit, --relax-path-checks, and --leave-container. #2718
• Update EC2 nodes and script. #2709
• Redesign File Job Store layout. #2669

Bug Fixes

• Fix for passing default values between Workflows. #2831
• Ignore checking versioning consistency when cleaning s3 jobstore. #2826
• Ensure terminated ignored nodes are cleared from batch system. #2789
• Fix Slurm sbatch command. #2777
• Typo in gridengine.py. #2778
• CWL fixes. #2783
• Clarify /data mountpoint for dockerCall. #2747
• Fix CWL tmpdir to be unique per job. #2745
• Protect against potential AWS truncated downloads. #2731
• Fix runningOnEC2 to work with all instances. #2734
• Prevent Toil from zipping entire dirs. #2728
• Changes to lsf.py to make it compatible on lsf9 and lsf10. #2726
• Remove X86_64 type requirement from lsf support. #2723
• Clean up temporary directories cwltoil uses to hold output streams. #2690

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @jeffrey856, @tthyer, @kaushik-work, @dleehr, @ionox0, @psafont, @mb1069, @tobiaszjarosiewicz, @gconcepcion, @byee4, @smoe, @TMiguelT, @jvivian-atreca!

3.20.0

• Deprecate subprocessDockerCall(). #2653
• Move wallTime to a label so we can use exact timestamps. #2664
• Capture and log standard output/error from batch systems. #2662
• Add an option to set custom AWS IAM roles. #2656
• Add a ToilNodeType tag to distinguish leader vs. workers. #2658
• Slurm: include both exit code and terminating signal in job return status. #2646
• Update requests package. #2610
• HTCondorBatchSystem(): fixes and features. #2647
• Update cwltool from 1.0.20180820141117 -> 1.0.20190228155703 #2609
• Add an option to set disableCaching to false. #2584
• Speed up S3 bucket deletion. #2597
• Tolerate missing jobs/workers in stats expando. #2533
• Add private IP support to node operations. #2569
• Encode logs for gzip. #2595

Bug Fixes

• Improve NoSuchFileException usage and robustness. #2593
• Make sure we always have current package lists. #2675
• Make sure to use --tmpdir-prefix to place workdir and jobstore. #2587
• Subprocess dockercall bytestring compatibility for py2/3. #2677
• Address python3 deprecation warnings. #2668
• Fix python3.7 break on new static method. #2659
• Get value of taskId in Mesos' killTask method. #2661
• Stop NamedTemporaryFile tampering. #2638
• StatsAndLogging(): ensure json.load() receives only string input. #2641
• Reduce max size of files allowed to be inlined in SDB. #2440
• Increase hardcoded encapsulate follow-on resources. #2591
• Ensure TOIL_DOCKER_REGISTRY variable is in version_template. #2583
• Use current python executable. #2599

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @kwwette, @liinus, @saimeCS, @cdunn2001, @Fifan31, @jeffrey856!

3.19.0

• Support for CWL s3 imports. #2522
• Support for running custom init commands in docker (allowing auth in private dockers). #2561
• Add --no-read-only option to the CWL runner. #2531
• Add support for assuming roles. #2527
• Added an adapter to allow Boto 2 to use credentials obtained via Boto 3. #2527
• Add a flag for running CWL internal jobs on workers. #2524
• Added CWL tests for http, https, and s3 file imports. #2234
• Begin testing python3 in gitlab. #2518
• Relegate pytest dependency to the dev reqs. #2508
• WDL changes to enable resource usage. #2495
• Sphinx and pathlib2 dependencies. #2490 #2493
• Upgrade appliance to Ubuntu 16.04. #2480
• Update cwltool from 1.0.20180820141117 -> 1.0.20181118133959 #2469
• Add CWL's --preserve-entire-environment option. #2481
• Bump minimum version of galaxy-lib (17.9.3 -> 17.9.9). #2476
• Added all tutorial scripts to unit testing. #2426
• More python3 support added. #2405
• Update the CoreOS AMI. #2430
• Simplify log creation in leader. #2423
• Support for hardlinks and greater testing for toil status. #2402
• Pymesos port. #2366 #2338
• Use displayName for CWLJob to show stats in toil --stats. #2397

Bug Fixes

• Import errors. #2558
• Fix boto3 credentials for permanent credentials. #2549
• Fix bug in Toil stats. #2513
• Replace some str objects as real FileIDs. #2500
• Remove linear-time bottleneck from Mesos batch system. #2498
• Slurm and py3 fixes. #2494
• Safeguard unintentional boto imports. #2489
• Fix broken readthedocs build. #2485
• Specify region in prometheus. #2437
• Fixed bytestring-string tagging. #2472
• LSF log job id on error. #2450
• Fix massive slowdown when many jobs are running. #2439
• Make ServiceManager stop blocking while (most) services are established. #2441
• Default to bash on the leader node rather than sh. #2435
• resource.getusage returns KB already, so dividing by 1024 yields MB. #2425
• Recognize conda virtual envs. #2422

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @nikhil, @lamby, @gijzelaerr, @joelarmstrong, @markasbach, @davidjsherman, @psafont, @jsmedmar, @bencvdb, @emlys, @ionox0, @ibebio, @MarkFilus!

3.18.0

• Refactor stream writes in the fileJobStore to minimize holding open file handles
• Large python 2/3 compatibility changes (+ python 3 compatible mesos library: pymesos)
• Get status/PID API added
• CWL:fix support for user space overrides
• Update cwltool from 1.0.20180518123035 to 1.0.20180820141117
• support lsf 10 per core reservation
• WDL bucket outputs with orig names
• Grid engine job creation fix
• Add Mesos port forwarding and the ability to use a SOCKS proxy with toil ssh-cluster
• Require a jobstore to be specified when a provisioner is.
• Retry HTTP connections on BadStatusLine errors
• Quote() all Docker arguments
• log LSF job id to error on failure
• Improve test file cleanup
• retries for batch commands
• Update CoreOS AMI
• CWL workDir defaults to tmpdir_prefix now if not set by the user
• Set the default duration for batch systems with getWaitDuration()
• Coverage Evaluation

Bug Fixes

• AMI Fetch Timeout
• Stop logging every message twice in certain cases
• Fix htcondor in wheel breaking on Apple
• Fix toil kill
• Fix invalid qsub in torque.py

Thank you to our contributors: @mr-c (big thanks for all of the CWL support!), @MarkFilus, @xzhub, @ionox0, @jsmedmar, @bencvdb, @tom-dyar, @joelarmstrong, @mwiens91, @jessebrennan, and @adamnovak!

3.17.0

• Add CWL destBucket runtime option to deposit CWL outputs into cloud buckets
• Update boto3 and boto
• Better Error messages when docker images are unavailable
• Broader checking for docker image existence prior to launching blindly
• Incremental Python2/3 compatibility improvements
• Enable support for specifying zone in all environment variables
• Expose --force-docker-pull and --no-match-user options from cwltool
• Add mem free option to UGE
• Convert most log messages to debug level
• Update cwltool from 1.0.20180518123035 to 1.0.20180521150620
• Incremental WDL Support Improvements (docker permissions, parsing, and improved logging)
• Documentation Housekeeping and Code Cleanup

Bug Fixes

• Stop FileJobStore DEBUG messages from sneaking out once for every job
• WDL Parsing Error
• Fix docker daemon detection depending on terminal size
• Checking virtualenv
• Properly filtering AWS security groups
• Fix a crash if job is larger than all available node types
• Avoid overwriting NVMe root volumes on current-gen AWS instances
• Don't call addNodes if the delta was met by unignoring nodes
• Fix AWSJobStore updates not working if values are shrunk
• Prevent worker from checking for nonexistant file

Thank you to our contributors: @mr-c (big thanks for all of the CWL support!), @jsmedmar, @bencvdb, @tom-dyar, @joelarmstrong, @mwiens91, @jessebrennan, and @adamnovak!

Release 3.16.0

• More Robust WDL Support
• Reworked the Cluster Scaler Algorithm
• Updated subprocess32 to version 3.5.1
• Eliminated bd2k-lib dependencies
• apiDockerCall allows blocking now for both detached/attached images
• Util to fetch the latest AWS instance types
• Check if docker images specified by the Toil appliance are valid before running
• Reduce toil clean memory usage for AWS jobStores

Bug Fixes

• CWL fixes for handling promises / followOn jobs
• Fix occasional cPickle deadlocks
• Increased Google and Azure stability
• Google jobstore fixes
• fileJobStore deletion made more robust for NFS-like systems

Thank you to our external contributors: @mr-c (big thanks for all of the CWL support!), @chapmanb, @ryanking, @jmchilton, @jorvis, @evan-wehi, @brainstorm, @jsmedmar

Release 3.15.0

• Azure Support (Beta)
• Google Support (Beta)

Bug Fixes

• CWL fixes.
• LSF cpu count fix.

Thank you to our external contributors: @mr-c (big thanks for all of the CWL support!), @chapmanb, @ryanking, @jmchilton, @jorvis, @evan-wehi, @brainstorm, @jsmedmar