Caching is now turned on by default.
CWL v1.1 is also now fully supported and passing all conformance tests.
The following batch system names are now deprecated (#3225) and replaced by:
- singleMachine -> single_machine
- gridEngine -> grid_engine
- LSF -> lsf
- Mesos -> mesos
- Slurm -> slurm
- Torque -> torque
- HTCondor -> htcondor
- Kubernetes -> kubernetes
- k8s -> kubernetes
Changelog
CWL
- Upgrade to a CWL v1.2.0 capable cwltool (3.0.20200709181526 -> 3.0.20200807132242). #3137
- Export to a bucket if cwl's CreateFile is specified. #3124
- CWL v1.1 Support: loadContents if True for StepValueFrom objects. #3266
- CWL: Fix workdir permissions. #3229
- Test CWL secondary file from s3. #3311
- CWL v1.1 Support: Inplace update has side effect on directory content. #3280
- More CWL v1.2 conformance tests. #3336
- CWL: Reformat w/ Black and better support for tmpdir_prefix with suffix. #3333
WDL
- Add ceil() and floor() WDL functions. #3168
- Create a stdout and stderr file for each WDL task. #3181
- Add WDL write functions and builtin tests. #3236
- write_lines
- write_tsv
- write_json
- write_map
- Add WDL read functions and builtin tests. #3244
- read_lines
- read_tsv
- read_json
- read_map
- read_int
- read_string
- read_float
- read_boolean
- Add transpose() WDL function. #3273
- Add length() WDL function. #3307
- Add sub() WDL function. #3252
- Add range() WDL function. #3277
- Add size() WDL function. #3255
- Add zip() WDL function. #3355
- Implement WDL Pair type. #3304
- Refactor WDL types #3335
Kubernetes
- Adding Labels to Pods/Jobs. #3233
- Make kubernetes & botocore packages fully optional. #3259 #3261
- KubeWatch optimization. #3227
- Catch other kubernetes imports. #3261
- Automatically infer singularity for kubernetes. #3279
Misc
- Allow Caching By Default. #3111
- Update dependencies and loosen restrictions slightly. #3139
- Ensure that falsey (but non-null) values are considered. #3143
- Remove pathlib. #3169
- Move AWS job store uploads/downloads to boto3. #3153
- Update prom/node-exporter to use quay.io. #3186
- Add support for TOIL_CUSTOM_INIT_COMMAND. #3183
- Update docker module version to 4.3.1. #3222
- Add 'stream' and 'demux' options to apiDockerCall. #3224
- Use boto3 to create our leader node. #3145
- Prevent job kind filenames from getting too long. #3230
- Add a retry decorator. #3144
- Remove mesos offer messages. #3308
- Job Concept Unification. #3250
- Add reporting for accessed files of failed jobs. #3309
- Add coverage and refactor. #3314
- Expose --statePollingWait param through toil. #3321
- Record the maximum memory used by LSF jobs. #3327
- Remove custom math functions and replace with built ins. #3338
- Define a changelog process. #3316
- SLURM: getting job details from
scontrol show jobs
was non-functional and has now been fixed. This behavior in the case of sacct
not being configured now works. #3346
- LSF: handle unicode configuration files #3354
- Replace abssympath w/ builtin. #3356
Bug Fixes
- Turn off watch code in Kubernetes. #3175
- Check against joining the current thread during CachingFileStore destructor. #3187
- Slurm.py error, undefined stdout - stdout should be stderr in code. #3194
- Fix restart flag error (issue #3094). #3142
- Batch System generates a key error instead of a usage message. #3225
- Remove Python 2 imports in WDL. #3228
- Move checkForDeadlocks() in the leader. #3234
- Fix boto3 migration bugs. #3243
- Fix None check when parsing WDL JSON file. #3272
- Fix broken stdout/stderr in torque job. #3306
- Add doubleMem for LSF jobs that die due to imposed memory limit. #3313
- Add 500 status to the list of retriable SDB BotoServerErrors. #3329
- Add a retry logic for BucketNotEmpty error. #3341
- Fix None check when parsing WDL JSON file. #3272
- Warn when trying to clean non-existent job store #3288
- Only rm a statefile that exists. #3348
- CWL: Remove unprocessed secondaryFiles from both Dict and List types.
Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @diekhans, @w-gao, @jonathanxu18, @jeffrey856, @davidlougheed, @altairwei, @mberacochea, @drkennetz, @ionox0!