Home / Python / Miscellaneous

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

Toil is a scalable, efficient, cross-platform (Linux & macOS) pipeline management system, written entirely in Python, and designed around the principles of functional programming. Check the website for a description of Toil and i
Information
Category: Python / Miscellaneous
Watchers: 55
Star: 813
Fork: 230
Last update: Aug 4, 2022
Resource links
https://github.com/DataBiosphere/toil
Join the chat at https://gitter.im/bd2k-genomics-toil/Lobby

Toil is a scalable, efficient, cross-platform (Linux & macOS) pipeline management system, written entirely in Python, and designed around the principles of functional programming.

  • Check the website for a description of Toil and its features.
  • Full documentation for the latest stable release can be found at Read the Docs.
  • Please subscribe to low-volume announce mailing list so we keep you informed
  • Google Groups discussion forum
  • See our occasional blog for tutorials.

Notes:

396 Open More issues
Closed

Continue the pymesos port

I'm PR-ing this against the other PR to see if I can convince the tests to run, because Courtyard can't run the tests properly due to the absence of Docker availability.

opened Aug 31, 2018 by adamnovak 75
Closed

WDL basic support. Alpha testing phase.

Support for .wdl files currently working for the GATK tutorials 1-4 on the WDL webpage. Have not tested on any other files, so currently this is still in alpha.

in progress 
opened Aug 14, 2017 by DailyDreaming 74
Closed

Initial PBS/Torque support and AbstractBatchSystem refactoring

This is a preliminary run at adding PBS/Torque support, designed off the SLURM and SGE classes. Tests have also been added.

Resolves #1163

opened Oct 6, 2016 by cjfields 68
Closed

Add Spark cluster service (resolves #774)

Resolves #774.

Ping @benedictpaten

spark/adam firecloud services 
opened Apr 13, 2016 by fnothaft 63
Closed

Support for Slurm batchSystem (resolves #682)

Implements support for slurm scheduler

  • SlurmBatchSystem class is modeled after GridengineBatchSystem
  • Maps AbstractBatchSystem methods to sbatch, squeue, scancel, sacct, sinfo subprocess calls
  • Includes SlurmBatchSystemTest for testing system
opened Mar 17, 2016 by dleehr 49
Closed

Set workdir to default to the executor sandbox in mesos (resolves #1143)

resolves #1143

The hierarchy of workdir is now: User-specified workdir > (sandox if mesos) > ENV TOIL_WORKDIR > ENV TMPDIR

opened Sep 7, 2016 by arkal 45
Open

Update setuptools requirement from <63,>=51.3.3 to >=51.3.3,<65

Updates the requirements on setuptools to permit the latest version.

Changelog

Sourced from setuptools's changelog.

v64.0.0

Deprecations ^^^^^^^^^^^^

  • #3380: Passing some types of parameters via --global-option to setuptools PEP 517/PEP 660 backend is now considered deprecated. The user can pass the same arbitrary parameter via --build-option (--global-option is now reserved for flags like --verbose or --quiet).

    Both --build-option and --global-option are supported as a transitional effort (a.k.a. "escape hatch"). In the future a proper list of allowed config_settings may be created.

Breaking Changes ^^^^^^^^^^^^^^^^

  • #3265: Added implementation for editable install hooks (PEP 660).

    By default the users will experience a lenient behavior which prioritises the ability of the users of changing the distributed packages (e.g. adding new files or removing old ones). But they can also opt into a strict mode, which will try to replicate as much as possible the behavior of the package as if it would be normally installed by end users. The strict editable installation is not able to detect if files are added or removed from the project (a new installation is required).

    .. important:: The editable aspect of the editable install supported this implementation is restricted to the Python modules contained in the distributed package. Changes in binary extensions (e.g. C/C++), entry-point definitions, dependencies, metadata, datafiles, etc may require a new installation.

Changes ^^^^^^^

  • #3380: Improved the handling of the config_settings parameter in both PEP 517 and PEP 660 interfaces:

    • It is possible now to pass both --global-option and --build-option. As discussed in #1928, arbitrary arguments passed via --global-option should be placed before the name of the setuptools' internal command, while --build-option should come after.

    • Users can pass editable-mode=strict to select a strict behaviour for the editable installation.

  • #3392: Exposed get_output_mapping() from build_py and build_ext subcommands. This interface is reserved for the use of setuptools Extensions and third part packages are explicitly disallowed to calling it. However, any implementation overwriting build_py or build_ext are required to honour this interface.

  • #3412: Added ability of collecting source files from custom build sub-commands to

... (truncated)

Commits

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
dependencies 
opened Aug 11, 2022 by dependabot[bot] 0
Open

Create Toil release checklist

┆Issue is synchronized with this Jira Task ┆friendlyId: TOIL-1202

opened Aug 10, 2022 by unito-bot 0
Open

Could toil also store requested cores data for stats purposes?

I'm not aware if recent versions do this already, but it would be nice to store stats about the number of cpus requested by jobs. Doing clock / cores could help calculate cpu efficiency.

Thoughts?

┆Issue is synchronized with this Jira Story ┆friendlyId: TOIL-1201

opened Aug 4, 2022 by jsmedmar 0
Open

Do not submit empty jobs

Thanks again for such a great tool.

Here I'm proposing a change, which I don't think its production ready, for a leader.py TODO item. When Toil finishes jobs, it walks up the DAG submitting empty jobs (jobs that actually do no work) requesting the original resources. Provisioning cloud machines for these jobs is not great.

Here I'm proposing to delete these jobs directly in leader.py. Full disclaimer, my knowledge of Toil's inner workings is limited, so this should be considered as a starting point.

All the best

I had created an issue about this before: https://github.com/DataBiosphere/toil/issues/3131

opened Aug 3, 2022 by jsmedmar 3
Open

toil stucks in recursive loop when s3 object key is a folder (ends in "/")

It seems that toil would get stuck in a recursive loop if from the AWS s3 file structure, there is a "folder" as one of the keys (which can happen if you create a folder on AWS, and then upload files into it). The error message is below,

Traceback (most recent call last):
File "/usr/local/bin/toil-cwl-runner", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/toil/cwl/cwltoil.py", line 3583, in main
builder = tool._init_job(initialized_job_order, runtime_context)
File "/usr/local/lib/python3.8/dist-packages/cwltool/process.py", line 808, in _init_job
get_listing(fs_access, job, recursive=(load_listing == "deep_listing"))
File "/usr/local/lib/python3.8/dist-packages/cwltool/utils.py", line 312, in get_listing
get_listing(fs_access, f, recursive=recursive)
File "/usr/local/lib/python3.8/dist-packages/cwltool/utils.py", line 328, in get_listing
get_listing(fs_access, ent, recursive)
File "/usr/local/lib/python3.8/dist-packages/cwltool/utils.py", line 328, in get_listing
get_listing(fs_access, ent, recursive)
File "/usr/local/lib/python3.8/dist-packages/cwltool/utils.py", line 328, in get_listing
get_listing(fs_access, ent, recursive)
[Previous line repeated 953 more times]
File "/usr/local/lib/python3.8/dist-packages/cwltool/utils.py", line 318, in get_listing
for ld in fs_access.listdir(loc):
File "/usr/local/lib/python3.8/dist-packages/toil/cwl/cwltoil.py", line 1287, in listdir
os.path.join(fn, entry.rstrip('/')) for entry in AbstractJobStore.list_url(fn)
File "/usr/local/lib/python3.8/dist-packages/toil/jobStores/abstractJobStore.py", line 544, in list_url
return otherCls._list_url(parseResult)
File "/usr/local/lib/python3.8/dist-packages/toil/jobStores/aws/jobStore.py", line 497, in _list_url
return list_objects_for_url(url)
File "/usr/local/lib/python3.8/dist-packages/toil/lib/retry.py", line 293, in call
return func(*args, kwargs)
File "/usr/local/lib/python3.8/dist-packages/toil/lib/aws/utils.py", line 379, in list_objects_for_url
for page in result:
File "/usr/local/lib/python3.8/dist-packages/botocore/paginate.py", line 269, in iter
response = self._make_request(current_kwargs)
File "/usr/local/lib/python3.8/dist-packages/botocore/paginate.py", line 357, in _make_request
return self._method(current_kwargs)
File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 508, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 898, in _make_api_call
http, parsed_response = self._make_request(
File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 921, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 119, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 199, in _send_request
success_response, exception = self._get_response(
File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 241, in _get_response
success_response, exception = self._do_get_response(
File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 308, in _do_get_response
parsed_response = parser.parse(
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 251, in parse
parsed = self._do_parse(response, shape)
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 871, in _do_parse
self._add_modeled_parse(response, shape, final_parsed)
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 881, in _add_modeled_parse
self._parse_payload(response, shape, member_shapes, final_parsed)
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 923, in _parse_payload
body_parsed = self._parse_shape(shape, original_parsed)
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 331, in _parse_shape
return handler(shape, node)
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 436, in _handle_structure
parsed[member_name] = self._parse_shape(
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 331, in _parse_shape
return handler(shape, node)
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 982, in _handle_list
return super()._handle_list(shape, node)
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 413, in _handle_list
return super()._handle_list(shape, node)
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 339, in _handle_list
parsed.append(self._parse_shape(member_shape, item))
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 331, in _parse_shape
return handler(shape, node)
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 436, in _handle_structure
parsed[member_name] = self._parse_shape(
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 331, in _parse_shape
return handler(shape, node)
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 177, in _get_text_content
return func(self, shape, text)
File "/usr/local/lib/python3.8/dist-packages/botocore/parsers.py", line 534, in _handle_timestamp
return self._timestamp_parser(text)
File "/usr/local/lib/python3.8/dist-packages/botocore/utils.py", line 938, in parse_timestamp
return _parse_timestamp_with_tzinfo(value, tzinfo)
File "/usr/local/lib/python3.8/dist-packages/botocore/utils.py", line 919, in _parse_timestamp_with_tzinfo
return dateutil.parser.parse(value, tzinfos={'GMT': tzutc()})
File "/usr/local/lib/python3.8/dist-packages/dateutil/parser/_parser.py", line 1368, in parse
return DEFAULTPARSER.parse(timestr, kwargs)
File "/usr/local/lib/python3.8/dist-packages/dateutil/parser/_parser.py", line 640, in parse
res, skipped_tokens = self._parse(timestr, kwargs)
File "/usr/local/lib/python3.8/dist-packages/dateutil/parser/_parser.py", line 719, in _parse
l = _timelex.split(timestr) # Splits the timestr into tokens
File "/usr/local/lib/python3.8/dist-packages/dateutil/parser/_parser.py", line 201, in split
return list(cls(s))
File "/usr/local/lib/python3.8/dist-packages/dateutil/parser/_parser.py", line 190, in next
token = self.get_token()
File "/usr/local/lib/python3.8/dist-packages/dateutil/parser/_parser.py", line 117, in get_token
if self.isword(nextchar):
File "/usr/local/lib/python3.8/dist-packages/dateutil/parser/_parser.py", line 206, in isword
return nextchar.isalpha()
RecursionError: maximum recursion depth exceeded while calling a Python object

From the logs, in normal instances, you'd see toil loop through the folder structure, as examples below,

Found in ParseResult(scheme='s3', netloc='foldername', path='/path/to/file', params='', query='', fragment='') items: []
Found in ParseResult(scheme='s3', netloc='foldername', path='/path/to/folder', params='', query='', fragment='') items: ['file1', 'file2']

But when there is the error with recursion, you would see the items array would contain an item with ''

Found in ParseResult(scheme='s3', netloc='foldername', path='/path/to/folder/with/issue', params='', query='', fragment='') items: ['']

If I do a query with boto3 to list all the objects in the bucket, this occurs in cases where an object with key is pointing toward to a folder object instead of a file object, and I think somehow toil parses it as a ''.

s3.ObjectSummary(bucket_name='bucket', key='folder/')
s3.ObjectSummary(bucket_name='bucket', key='folder/filename.ext')

┆Issue is synchronized with this Jira Story ┆friendlyId: TOIL-1200

opened Aug 2, 2022 by boyangzhao 0
Open

Toil uses `/var/run/user/$UID` if it sees it, whether or not it is actually available to Toil's current session

Hi,

I noticed that u are using /var/run/user/$UID to put locally locks on files.

It creates problems on our cluster because the way our session are managed by PAM does initialize pam_systemd. I thinks it's a normal behaviour for non interactive session.

Indeed pam_systemd is the pam module responsible for creating the /run/user/$UID as you can read here: https://manpages.debian.org/unstable/libpam-systemd/pam_systemd.8.en.htmt

/var/run appears to be a symlink to /run however I think it's something that can vary across linux distributions.

There is an interesting discussion about using or not /var/run/user/$UID here: https://superuser.com/questions/1056926/is-var-run-user-uid-the-new-var-run-for-pid-files

As far as I understand this is not created by default on debian buster for non-interactive session.

Below are the file contents that I think I got from : https://packages.debian.org/fr/buster/libpam-runtime

$ grep 'systemd' /etc/pam.d/common-session
session optional        pam_systemd.so
$ grep 'systemd' /etc/pam.d/common-session-noninteractive
$

As far as I understand, beyond pam service, /var/run/user/<uid> can be created by any demon managed by systemd with root privileges, which is not the case of toil workers.

Given those elements, I would not recommand using /var/run/user/$UID by default. I would not even recommand using it after checking that the folder exists because it might be a collision with an interactive session that is running but may stop at any time.

┆Issue is synchronized with this Jira Story ┆friendlyId: TOIL-1199

opened Jul 29, 2022 by jfouret 4

releases/5.7.1(Jul 19, 2022)

Changelog

Highlighted Features Added

AWS Batch Batch System (#3956) AGC Integration (#4039) + More AGC integration (#4067) + AGC megabranch (#4113) Scale TES to be able to run reasonably-sized workflows on Funnel on Kubernetes with the AWS job store (#3927)

CWL

Run CWL conformance tests via WES (#4052) Implement and test CWL loadContents from URLs to fix #4125 (#4126) Add CWL tests under ARM (#4038) Cache results of cwltool version lookup (#4141)

Misc

SGE batch system change to support serial jobs. (#4022) Performance testing for Graviton instances (#4123) Stop waiting on hostpath volumes to exist (#4146) Catch and warn about jobs going away too slowly on FileJobStore (#4149) Add documentation for the type-checking hooks (#4117) Pod murder bot (#4060) Contrib hook scripts (#4105) Allow newer google-cloud-storage (#4114) Use environment variable to set parallel partition name (#4096) Register pytest markers (#4103) Mention --export=ALL for SLURM environments (#4100) (#4102) Allow persisting workflow state in WES server across container recreation (#4082) Change toil kill to use the job store shared file API to find pig.log (#4075) Bring back kill loop in the single_machine batch system but with a timeout (#4070) Reorganize Locking (#4059) Add and test preemptability constraints (#4044) Enhanced types (#3975) Use an init process that reaps zombies on toil clusters (#3974) Add launch cluster support for ARM (#3971) Feat: square bracket to period separator (#4008) Add AGC health check endpoint (#3997) Tolerate and require typed Werkzeug (#4011) Add more static URLs for Singularity debs (#4007)

Bug Fixes

Update WES set up docs (#4027) Add real time logs (#4031) Fail fast if Docker builder is missing (#4001) Make Toil version be reported as a string in WES (#4013) Fix assorted typos within assorted comments (#4023) Make file store case insensitive (#4153) Pre-lex commands for qsub (#4150) Update Cactus and exclude broken networkx (#4107) Make toil kill work when the leader is on another machine (#4084) Wrong filename in output (#4139) Tolerate a missing VersionID key to fix #4129 (#4130) Only import from typing_extensions on old Python where we install it (#4090) Allow missing username and fix Docker build (#4077) Leave more time for concurrency measurement to fix #4012 (#4068) Stop people asking for ARM Mesos clusters to fix #4057 (#4058)

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @Hexotical, @gmloose, @kannon92, @douglowe, @gcapes, and @pmiddend!

releases/5.6.0(Jan 10, 2022)

Changelog

Highlighted Features Added

  • Integrate ARM Docker builds to make multi-arch images. (#3802)
  • WES support and server mode. (#3779)
  • TES batch system prototype. (#3821)
  • Support for new resource syntax in PBSPro. (#3048)
  • Toil now looks for lost jobs every minute instead of every hour (#3948)

Breaking Changes

  • Remove --disableCaching's true/false argument. (#3869)

CWL

  • CWL helper jobs: better disk & memory requirements. (#3834)
  • CWL: safer test path generation for post-install testing. (#3818)
  • More detailed names, which show up in job names sent to BatchSystem schedulers. #3941 (was #3893)
  • If you use scatter and collect files, duplicates are correctly dealt with and renamed. (#3968)
  • CWL: at the end of a job, ask cwltool to cleanup (#3965)

Kubernetes

  • Assign Kubernetes jobs an explicit TTL. (#3936)
  • Wait for node creation. (#3934)

Dependencies

  • Enable Dependabot updates. (#3827)
  • Multiple consolidated dependabot updates. (#3851)
  • Update addict requirement from <2.3,>=2.2.1 to >=2.2.1,<2.5. (#3861)
  • Bump cwltool to 3.1.20211107152837. (#3833 #3866 #3909)
  • Bump cwltest from 2.1.20210626101542 to 2.2.20210901154959. (#3848)
  • Bump flake8 from 3.8.4 to 4.0.1. (#3847)
  • Allow more docker-py versions. (#3860)
  • Remove pyyaml dependency. (#3858)

Misc

  • Update cleanup script. (#3937)
  • Remove use of sys.maxsize. (#3824)
  • Spelling fixes. (#3814)
  • Add codeql-analysis for Python. (#3825)
  • Update jobstore function names. (#3809)
  • Move AMI functions to lib. (#3810)
  • Add "make pyupgrade" (py36-plus). (#3805)
  • Type hints. (#3930)
  • Coalesce status calls in slurm. (#3822)
  • Python logging takes format values as *args. (#3852)
  • Change quick test to a 10 minute timeout. (#3843)
  • Add make uninstall to makefile. (#3883)
  • Fix toil kill to find shared pid.log file (with unit test). #3941 (was #3932)
  • Update documentation. (#3947)
  • Remove remains of Travis. (#3976)

Bug Fixes

  • Stop checkpoints from being reissued multiple times. (#3931)
  • Don't consult LSF config when explicitly defining memory units. (#3820)
  • Robustly remove state dirs. (#3836)
  • Fix exception checking for exit_code. (#3830)
  • Use exitStatus instead of exitReason for batch exit type comparison. (#3839)
  • Update cwltest to improve K8 runs. (#3935)
  • Consolidated CI Fixes. (#3887)
  • Fix CWL conformance tests. (#3891)
  • Toil-managed cluster scaling should work again with --metrics. (#3943)

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @Hexotical, @tmooney, @nikhil, @kannon92, @douglowe, @mhpopescu, @Phhere, and @gmloose!

releases/5.5.0(Sep 25, 2021)

Changelog

CWL

  • Add podman support; and other fixes from recent cwltool #3799
  • Add streaming feature for cwltoil #3694
  • Warn users if a different cwltool version is installed #3686
  • Turn on all Kubernetes CWL tests that are expected to work on Singularity #3720
  • Fix CWL in toil docs jobstore usage #3728
  • DOC: update versions of CWL support
  • Allow filestore bypass #3652

Misc

  • Numerous Type Hints. #3705 #3701 #3693 #3691 #3663 #3688 #3642 #3684 #3682 #3680 #3666 #3675
  • Single source of truth for job state #3776
  • Do not set default for statePollingWait #3774
  • Use absolute local paths when exportFile/importFile do not detect a schema #3767
  • Multi-zone balancing within regions for AWS autoscaling groups #3746
  • Migrate cloud-config to ignition #3488
  • 🎡 Wheel Of Issues 🎰 #3760
  • Add back addBatchSystemFactory function #3754
  • Redirect stderr to /dev/null of lsf conf queries #3751
  • Set number of cores based on job.cores for OpenMP applications #3739
  • Google jobstore batching #3740
  • Locations of CLI option docs
  • Add AWS provisioner storage system #3727
  • Set cls.bucket #3726
  • Stream vs dowload jobs #3722
  • Update Toil's main python test version to 3.8. #3669
  • Move Travis tests to Gitlab #3675

Bug Fixes

  • Don't leak symlinks #3795
  • Prevent exception from being raised when modifying dir permissions for clean up #3778
  • Fix scontrol output parsing #3793
  • Fix AttributeError #3742
  • Workaround for S3 in us-east-1 #3710
  • Time data format #3708
  • Fix leader.py batch system std files prefix glob #3679

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @Hexotical, @ionox0, @gmloose, @juanesarango, @mhpopescu, @mberacochea, @nikhil!

releases/5.4.0(Jun 25, 2021)

Changelog

CWL

  • Fix cwl and wdl dependency bleed and add stand alone tests. #3582
  • Use MpiConfig.load() to handle MPI config file #3574
  • Add support for MPI with CWL. #3525

Misc

  • Numerous Type Hints. #3571 #3634 #3626 #3625 #3616 #3614 #3601 #3592 #3590 #3581
  • Configurable Grafana port #3597
  • Handle streaming reads from the cache when the data isn't written back yet #3595
  • Balance over pools using AWS ASGs #3490
  • Change how Kubernetes schedules and scales for hopefully better scale-down behavior #3587
  • Add Owner tags to our AWS buckets when testing. #3577
  • Allow workdir override #3586
  • Additional info for "Permission denied" error #3579
  • Consolidate memory functions #3529
  • Pen children #3482
  • Sniff raid better #3526
  • Add a propagation policy to batch delete #3522
  • Cleanup script for buckets, sdb domains, instance profiles, and roles. #3373
  • Add decorator for flaky tests. #3510

Bug Fixes

  • Enforce valid uuid from getNodeId() #3611
  • Removed extra memory multiplication #3608
  • Fix non integers lsf memory requested #3609
  • Fixes jobCommand unpack #3610
  • Fix CWL test 20 on Kubernetes #3572
  • Fix #3579 breaking AWS docs #3596
  • Update cactus test to fix broken bucket links. #3594
  • Allow cleaning up a job whose overlargeID fell off #3584
  • Only deploy a user script if we can deploy user scripts #3518
  • Reduce path name lengths. #3438
  • Insist on credentials for testing AMI finding #3524

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @thiagogenez, @julian-klode, @darafferty, @nikhil, @mhpopescu!

releases/5.3.0(Mar 25, 2021)

Changelog

CWL

  • Run CWL conformance tests on Kubernetes. #3323
  • CWL symlinking files into work directory #3445
  • set resource reqs for all CWL_INTERNAL_JOBS #3442

WDL

  • Add WDL flatten() function + tests. #3485
  • Add WDL collect_by_key() function + tests. #3476
  • Add WDL keys() function + tests. #3460
  • Add WDL as_pairs() function + tests. #3364
  • Add as_map() WDL function #3448
  • Add basic WDL development support #3434
  • Add basic WDL 1.0 support #3421
  • Update dictionary structure in AnalyzeWDL. #3416

Misc

  • Add extra context for disk usage warning. #3495
  • Update developing.rst #3479
  • Allow text/encode options for read and write functions in file store #3428
  • Type hinting and mypy checking additions #3470 # #3458 #3456
  • Quality of life improvements for cactus. #3463
  • Add checklists for reviewing and merging PRs #3432
  • Ensure that python3 and not python (2) is used #3446
  • Support Kubernetes clusters in toil launch-cluster #3357
  • Additional check for predecessors. #3417
  • Allow setOptions to accept an argparse group. #3426

Bug Fixes

  • AWS: Handle socket timeout during AWS discovery #3503
  • Use back-up flatcar AMI if not found. #3513
  • Lsf command parser #3475
  • Retry on s3 throttling. #3504
  • Stop deleting all ASGs with tags #3474
  • Only determine execute permissions on files we copy/move. #3437
  • cope with an invalid HTTP_PROXY #3447
  • Fix failing Google jobstore test #3420
  • Update Google job store #3412

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, Arthur Rand, @mberacochea, @Jessime, @thiagogenez, @julian-klode!

releases/5.2.0(Jan 8, 2021)

Changelog

CWL

  • Update to the latest cwltool (3.0.20201121085451 -> 3.0.20201203173111). #3375
  • Add better handling for potentially mis-ordered CWL args. #3395
  • Confirm CommandInputParameter expression can receive a File object. #3350

WDL

  • Make AnalyzeWDL abstract to allow implementation for different WDL versions #3391

Misc

  • Clean up memoize.py. #3374
  • Refactor bioio library, simplify logging, and small adjustments to utils. #3351
  • Use regular division for memory calculation on LSF. #3387
  • Remove unused --nodeOptions arg. #3397
  • Get boto S3 args for minio from environment variables. (#3370)
  • Update EC2 lists. #3376
  • Add a retry to AWS destroy. #3379
  • Regular division for memory calculation in LSF. #3387
  • Add option to blank mem allocation for SLURM. #3399
  • Preserve file permissions on imported/exported files #3322

Bug Fixes

  • Preserve file permissions on imported/exported files. #3322
  • Make building a flattened list of live jobs non-recursive. #3394
  • Move AWS job store uploads/downloads to boto3 (reopened) #3400
  • Use a Docker Hub mirror when pulling Docker images. #3411
  • Corrected gridengine maxMEM check to use memorystring #3410

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @douglowe, @stevekm , @thiagogenez!

Contents

    sartography A powerful workflow engine implemented in pure Python
    hooopo Petri Net Workflow Engine for Ruby.
    osmedeus Community Workflow for the Osmedeus Engine that describes basic reconnaissance methodology for you to build your own
    godotengine Godot Engine – Multi-platform 2D and 3D game engine
    danielgerlag Lightweight workflow engine for .NET Standard
    ScriptArtist Makes the UI elements scalable to fit inside the container area.
    enigo-rs enigo - Enigo lets you control your mouse and keyboard in an abstract way on different operating systems (currently only Linux, macOS, Win – Redox and *BSD planned)
    tenacityteam Tenacity is an easy-to-use, cross-platform multi-track audio editor/recorder for Windows, MacOS, GNU/Linux
    insitro Yet another redundant workflow engine
    cormiertyshawn895 Run Aperture, iPhoto, and iTunes on macOS Catalina. Xcode 11.4.1 on macOS Mojave. Final Cut Pro 7, Logic Pro 9, and iWork ’09 on macOS Mojave or macOS High Sierra.
    winfsp Cross-platform FUSE library for Go - Works on Windows, macOS, Linux, FreeBSD, NetBSD, OpenBSD
    dthree Cross-platform Linux commands in ES6
    uber Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
    redstrate Cross-platform 3D game engine featuring real-time physically based rendering
    scottcgi A cross-platform, open-source, pure C game engine for mobile game.
    flowable A compact and highly efficient workflow and Business Process Management (BPM) platform for developers, system admins and business users.
    42coders The Workflow Package add Drag & Drop Workflows to your Laravel Application.
    tillkruss An ultra-fast Laravel docs search workflow for Alfred 3
    devxoul Pure DI in Swift
    Famous Famous Engine a free and open source JavaScript rendering engine.
    google Cross platform Go logging library.
    Share this repo

    Related Repos


    Miscellaneous
    100
    Playpen is a set of modern, gas optimized staking pool contracts.
    ZeframLou Playpen Playpen is a set of modern, gas optimized staking pool contracts. Features Support for both ERC20 staking and ERC721 staking Can start new rew
     
    Miscellaneous
    47
    DIAL(Did I Alert Lambda?) is a centralised security misconfiguration detection framework w...
    CRED-CLUB DIAL Workloads on cloud provide equal opportunities for hackers as much as they do for internal teams. Cloud-native companies are open to attacks from
     
    Miscellaneous
    42
    Code for "Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation"
    alvinliu0 Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation Code for "Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation"
     
    Miscellaneous
    40
    IDA Frida Plugin for tracing something interesting.
    P4nda0s IDAFrida A simple IDA plugin to generate FRIDA script. Edit template for functions or you can use the default template. Select functions you want to t
     
    Miscellaneous
    15
    https://t.me/master_thyself
    evmn Know Thyself 中文版 Chinese Title Authors Mind Hacks 刘未鹏 编程随想: 2009 ~ 2021 Blog Archives Program Think 晚点 LatePost 宋玮…… English Blogs Title Authors Essay
     
    Miscellaneous
    717
    a Jupyter kernel for Clojure
    clojupyter A Jupyter kernel for Clojure - run Clojure code in Jupyter Lab, Notebook and Console. Table of Contents Getting Started Installation Usage Scenarios -
     
    Miscellaneous
    132
    using your namespace as a notebook
    scicloj notespace Notebook experience in your Clojure namespace What is it? This tool is an attempt to answer the following question: can we have a notebook-l
     
    Miscellaneous
    794
    Python bindings for Clojure
    clj-python Deep Clojure/Python Integration Version Info New Versions Are HOT! Huge New Features! You Can't Afford To Miss Out! API Documentation Java API - you c
     
    Miscellaneous
    78
    Turn data into functions! A simple and functional machine learning library written in elix...
    mrdimosthenis emel Turn data into functions! A simple and functional machine learning library written in elixir. Installation The package can be installed by adding
     
    Miscellaneous
    6
    An optional component handler for hikari, inspired by discord.py's views.
    HyperGH hikari-miru An optional component handler for hikari, inspired by discord.py's views. Installation pip install git+https://github.com/HyperGH/hikari-m
     
    Miscellaneous
    5
    TikTok 4L and 4C checker that doesn't count banned usernames as available
    cliphd TikTok 4L and 4C checker that doesn't count banned usernames as available. Once a username is available, it will send it to your Discord Webhook
     
    Miscellaneous
    4
    TBOT Binance Smart Chain token
    TBOT-Project tbot-bsc-token TBOT Binance Smart Chain token Minting transaction: https://bscscan.com/tx/0xd387a96ac15a616f9b17bf9af84d63fd54de98e176a55bf11bb4f08855
     
    Miscellaneous
    4
    notes on reusable p5.js coding patterns
    M0nica p5.js Notes on random p5.js coding patterns Table of Contents Positioning Elements Responding to User Input Positioning Elements centering elements fu
     
    Miscellaneous
    3
    smart contracts for the implementation of a networked money stream using the ERC1155 stand...
    phydy MoneyStreamNetwork #Short Description A Tradable Cash-Flow DeFi-NFT that gives a holder the power to leverage their superfluid flow to get loans or fu
     
    Miscellaneous
    3
    Fb token and cookie making tools
    niloy0 🥀 🍁 ASSALAMU WALAYKUM 🥰 🥀 🥀 😻 🌺 HI I AM N!LOY 🥀 😘 😻 ! 🥀 😻 WELCOME TO MY PROFILE 🥰 MY PROFILE VISITORS : GITHUB STATES : CONTACT WITH ME :