Home / Python / Miscellaneous

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

Toil is a scalable, efficient, cross-platform (Linux & macOS) pipeline management system, written entirely in Python, and designed around the principles of functional programming. Check the website for a description of Toil and i
Information
Category: Python / Miscellaneous
Watchers: 55
Star: 821
Fork: 227
Last update: Jan 26, 2023
Resource links
https://github.com/DataBiosphere/toil
Join the chat at https://gitter.im/bd2k-genomics-toil/Lobby

Toil is a scalable, efficient, cross-platform (Linux & macOS) pipeline management system, written entirely in Python, and designed around the principles of functional programming.

  • Check the website for a description of Toil and its features.
  • Full documentation for the latest stable release can be found at Read the Docs.
  • Please subscribe to low-volume announce mailing list so we keep you informed
  • Google Groups discussion forum
  • See our occasional blog for tutorials.

Notes:

releases/5.8.0(Jan 4, 2023)

Changelog

Highlighted Features Added

  • Toil server now exposes workflow tasks via WES (#4046).
  • Toil server now has a --wes_dialect agc option that will hide any tasks that don't have Amazon Batch job IDs, and put the IDs in the task names for those that do (#4047).
  • Toil jobs now accept an accelerators requirement, like accelerators=1 or accelerators={'kind': 'gpu', 'brand': 'nvidia', 'count': 2} (#4163)
  • Include total requested cores for each job type in toil stats (#4173)
  • Toil jobs now expose job.accelerators to workflow
  • Add prefix suffix params to AbstractFileStore.getLocalTempFile and AbstractFileStore.getLocalTempFileName (#4273)
  • CWL: --no-compute-checksum, --strict-cpu-limit, --disable-validate, and --fast-parser are now available

Breaking Changes

  • Toil's built-in autoscaler now guesses that some memory and disk space on nodes will not actually be available for jobs; pass --assumeZeroOverhead to revert to the old behavior (#2103)

CWL

  • CWL job unit and display names have been changed to make more sense as task names, and management of them has been unified into a CWLNamedJob. (#4046/#4047)
  • CWL CUDARequirement is parsed by cwltool and turned into a requirement for the minimum requested number of nvidia GPU accelerators (#3982)
  • fix false warning when outputSource contains only one None value (#4300)

Kubernetes

  • KubernetesBatchSystem can add nvidia.com/gpu and amd.com/gpu resource requests for jobs that request those accelerators (#4163)
  • KubernetesBatchSystem can request GPUs by model key, if nodes are labeled appropriately (#4163)

Dependencies

Misc

  • Toil WES server now accepts requests that leave out workflow_params. (#4037)
  • The MessageBus has been expanded to use pypubsub, and now has MessageInbox and MessageOutbox objects to represent connections to it. (#4046/#4047)
  • ToilMetrics now rides on the MessageBus rails. (#4046/#4047)
  • Toil workflows now have a --writeMessages option, which takes a file to which a line-oriented stream of MessageBus messages will be written. Reading this file will allow you to recover the current state of the workflow. (#4046/#4047)
  • Add code for warning check to be used when launching cluster with AWS. (#3514)
  • Use a CI prebake image for gitlab testing. (#4185)
  • Toil clusters now have /var/tmp as the default temporary directory, since they often make large temporary files (#4148)
  • Adds basic testing for slurm using a slurm docker cluster by running sample workflows. (#3856)
  • Add message bus documentation (#4239)
  • SingleMachineBatchSystem can schedule nvidia GPU accelerators, limiting the concurrent jobs to no more than there are accelerators to support, and setting CUDA_VISIBLE_DEVICES in the tasks' environments to tell them which nvidia GPU(s) to use. (#4163)
  • AWSBatchBatchSystem can use AWS Batch's GPU resource to provide nvidia GPU accelerators (#4163)
  • Toil jobs no longer need to re-run after their child/followOn/service jobs in order to delete themselves. (#3188)
  • Message bus is now thread safe (#4276)
  • Docker build has been updated with new Aventer Mesos deb URL (fixes #4290)
  • docker binary in the container has been updated to that included in the Ubuntu repos (fixes #4282)
  • Singularity in the appliance has been updated to 3.10 which is >=3.9, for cgroups v2 support.
  • Base Ubuntu container image for the appliance has been updated to 22.04, which has a new enough libc for Debian's Singularity 3.10 debs.
  • Safer type usage checking for systems without boto3 installed
  • Tests are now more runnable post-installation. Temporary paths are not selected based upon the location of the tests themselves. (#4287)

Bug Fixes

  • Only use /var/run/user if XDG tells us we have it in our session. Otherwise we will try other places, including /run/lock/toil. (#4170)
  • toil destroy-cluster: terminate stopped instances when destroying the cluster (#4271)
  • fileJobStore: handle arbitrary os.link errors to work on some filesystems (#2232)

Thank you to our contributors!

releases/5.7.1(Jul 19, 2022)

Changelog

Highlighted Features Added

AWS Batch Batch System (#3956) AGC Integration (#4039) + More AGC integration (#4067) + AGC megabranch (#4113) Scale TES to be able to run reasonably-sized workflows on Funnel on Kubernetes with the AWS job store (#3927)

CWL

Run CWL conformance tests via WES (#4052) Implement and test CWL loadContents from URLs to fix #4125 (#4126) Add CWL tests under ARM (#4038) Cache results of cwltool version lookup (#4141)

Misc

SGE batch system change to support serial jobs. (#4022) Performance testing for Graviton instances (#4123) Stop waiting on hostpath volumes to exist (#4146) Catch and warn about jobs going away too slowly on FileJobStore (#4149) Add documentation for the type-checking hooks (#4117) Pod murder bot (#4060) Contrib hook scripts (#4105) Allow newer google-cloud-storage (#4114) Use environment variable to set parallel partition name (#4096) Register pytest markers (#4103) Mention --export=ALL for SLURM environments (#4100) (#4102) Allow persisting workflow state in WES server across container recreation (#4082) Change toil kill to use the job store shared file API to find pig.log (#4075) Bring back kill loop in the single_machine batch system but with a timeout (#4070) Reorganize Locking (#4059) Add and test preemptability constraints (#4044) Enhanced types (#3975) Use an init process that reaps zombies on toil clusters (#3974) Add launch cluster support for ARM (#3971) Feat: square bracket to period separator (#4008) Add AGC health check endpoint (#3997) Tolerate and require typed Werkzeug (#4011) Add more static URLs for Singularity debs (#4007)

Bug Fixes

Update WES set up docs (#4027) Add real time logs (#4031) Fail fast if Docker builder is missing (#4001) Make Toil version be reported as a string in WES (#4013) Fix assorted typos within assorted comments (#4023) Make file store case insensitive (#4153) Pre-lex commands for qsub (#4150) Update Cactus and exclude broken networkx (#4107) Make toil kill work when the leader is on another machine (#4084) Wrong filename in output (#4139) Tolerate a missing VersionID key to fix #4129 (#4130) Only import from typing_extensions on old Python where we install it (#4090) Allow missing username and fix Docker build (#4077) Leave more time for concurrency measurement to fix #4012 (#4068) Stop people asking for ARM Mesos clusters to fix #4057 (#4058)

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @Hexotical, @gmloose, @kannon92, @douglowe, @gcapes, and @pmiddend!

releases/5.6.0(Jan 10, 2022)

Changelog

Highlighted Features Added

  • Integrate ARM Docker builds to make multi-arch images. (#3802)
  • WES support and server mode. (#3779)
  • TES batch system prototype. (#3821)
  • Support for new resource syntax in PBSPro. (#3048)
  • Toil now looks for lost jobs every minute instead of every hour (#3948)

Breaking Changes

  • Remove --disableCaching's true/false argument. (#3869)

CWL

  • CWL helper jobs: better disk & memory requirements. (#3834)
  • CWL: safer test path generation for post-install testing. (#3818)
  • More detailed names, which show up in job names sent to BatchSystem schedulers. #3941 (was #3893)
  • If you use scatter and collect files, duplicates are correctly dealt with and renamed. (#3968)
  • CWL: at the end of a job, ask cwltool to cleanup (#3965)

Kubernetes

  • Assign Kubernetes jobs an explicit TTL. (#3936)
  • Wait for node creation. (#3934)

Dependencies

  • Enable Dependabot updates. (#3827)
  • Multiple consolidated dependabot updates. (#3851)
  • Update addict requirement from <2.3,>=2.2.1 to >=2.2.1,<2.5. (#3861)
  • Bump cwltool to 3.1.20211107152837. (#3833 #3866 #3909)
  • Bump cwltest from 2.1.20210626101542 to 2.2.20210901154959. (#3848)
  • Bump flake8 from 3.8.4 to 4.0.1. (#3847)
  • Allow more docker-py versions. (#3860)
  • Remove pyyaml dependency. (#3858)

Misc

  • Update cleanup script. (#3937)
  • Remove use of sys.maxsize. (#3824)
  • Spelling fixes. (#3814)
  • Add codeql-analysis for Python. (#3825)
  • Update jobstore function names. (#3809)
  • Move AMI functions to lib. (#3810)
  • Add "make pyupgrade" (py36-plus). (#3805)
  • Type hints. (#3930)
  • Coalesce status calls in slurm. (#3822)
  • Python logging takes format values as *args. (#3852)
  • Change quick test to a 10 minute timeout. (#3843)
  • Add make uninstall to makefile. (#3883)
  • Fix toil kill to find shared pid.log file (with unit test). #3941 (was #3932)
  • Update documentation. (#3947)
  • Remove remains of Travis. (#3976)

Bug Fixes

  • Stop checkpoints from being reissued multiple times. (#3931)
  • Don't consult LSF config when explicitly defining memory units. (#3820)
  • Robustly remove state dirs. (#3836)
  • Fix exception checking for exit_code. (#3830)
  • Use exitStatus instead of exitReason for batch exit type comparison. (#3839)
  • Update cwltest to improve K8 runs. (#3935)
  • Consolidated CI Fixes. (#3887)
  • Fix CWL conformance tests. (#3891)
  • Toil-managed cluster scaling should work again with --metrics. (#3943)

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @Hexotical, @tmooney, @nikhil, @kannon92, @douglowe, @mhpopescu, @Phhere, and @gmloose!

releases/5.5.0(Sep 25, 2021)

Changelog

CWL

  • Add podman support; and other fixes from recent cwltool #3799
  • Add streaming feature for cwltoil #3694
  • Warn users if a different cwltool version is installed #3686
  • Turn on all Kubernetes CWL tests that are expected to work on Singularity #3720
  • Fix CWL in toil docs jobstore usage #3728
  • DOC: update versions of CWL support
  • Allow filestore bypass #3652

Misc

  • Numerous Type Hints. #3705 #3701 #3693 #3691 #3663 #3688 #3642 #3684 #3682 #3680 #3666 #3675
  • Single source of truth for job state #3776
  • Do not set default for statePollingWait #3774
  • Use absolute local paths when exportFile/importFile do not detect a schema #3767
  • Multi-zone balancing within regions for AWS autoscaling groups #3746
  • Migrate cloud-config to ignition #3488
  • 🎡 Wheel Of Issues 🎰 #3760
  • Add back addBatchSystemFactory function #3754
  • Redirect stderr to /dev/null of lsf conf queries #3751
  • Set number of cores based on job.cores for OpenMP applications #3739
  • Google jobstore batching #3740
  • Locations of CLI option docs
  • Add AWS provisioner storage system #3727
  • Set cls.bucket #3726
  • Stream vs dowload jobs #3722
  • Update Toil's main python test version to 3.8. #3669
  • Move Travis tests to Gitlab #3675

Bug Fixes

  • Don't leak symlinks #3795
  • Prevent exception from being raised when modifying dir permissions for clean up #3778
  • Fix scontrol output parsing #3793
  • Fix AttributeError #3742
  • Workaround for S3 in us-east-1 #3710
  • Time data format #3708
  • Fix leader.py batch system std files prefix glob #3679

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @Hexotical, @ionox0, @gmloose, @juanesarango, @mhpopescu, @mberacochea, @nikhil!

releases/5.4.0(Jun 25, 2021)

Changelog

CWL

  • Fix cwl and wdl dependency bleed and add stand alone tests. #3582
  • Use MpiConfig.load() to handle MPI config file #3574
  • Add support for MPI with CWL. #3525

Misc

  • Numerous Type Hints. #3571 #3634 #3626 #3625 #3616 #3614 #3601 #3592 #3590 #3581
  • Configurable Grafana port #3597
  • Handle streaming reads from the cache when the data isn't written back yet #3595
  • Balance over pools using AWS ASGs #3490
  • Change how Kubernetes schedules and scales for hopefully better scale-down behavior #3587
  • Add Owner tags to our AWS buckets when testing. #3577
  • Allow workdir override #3586
  • Additional info for "Permission denied" error #3579
  • Consolidate memory functions #3529
  • Pen children #3482
  • Sniff raid better #3526
  • Add a propagation policy to batch delete #3522
  • Cleanup script for buckets, sdb domains, instance profiles, and roles. #3373
  • Add decorator for flaky tests. #3510

Bug Fixes

  • Enforce valid uuid from getNodeId() #3611
  • Removed extra memory multiplication #3608
  • Fix non integers lsf memory requested #3609
  • Fixes jobCommand unpack #3610
  • Fix CWL test 20 on Kubernetes #3572
  • Fix #3579 breaking AWS docs #3596
  • Update cactus test to fix broken bucket links. #3594
  • Allow cleaning up a job whose overlargeID fell off #3584
  • Only deploy a user script if we can deploy user scripts #3518
  • Reduce path name lengths. #3438
  • Insist on credentials for testing AMI finding #3524

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @thiagogenez, @julian-klode, @darafferty, @nikhil, @mhpopescu!

releases/5.3.0(Mar 25, 2021)

Changelog

CWL

  • Run CWL conformance tests on Kubernetes. #3323
  • CWL symlinking files into work directory #3445
  • set resource reqs for all CWL_INTERNAL_JOBS #3442

WDL

  • Add WDL flatten() function + tests. #3485
  • Add WDL collect_by_key() function + tests. #3476
  • Add WDL keys() function + tests. #3460
  • Add WDL as_pairs() function + tests. #3364
  • Add as_map() WDL function #3448
  • Add basic WDL development support #3434
  • Add basic WDL 1.0 support #3421
  • Update dictionary structure in AnalyzeWDL. #3416

Misc

  • Add extra context for disk usage warning. #3495
  • Update developing.rst #3479
  • Allow text/encode options for read and write functions in file store #3428
  • Type hinting and mypy checking additions #3470 # #3458 #3456
  • Quality of life improvements for cactus. #3463
  • Add checklists for reviewing and merging PRs #3432
  • Ensure that python3 and not python (2) is used #3446
  • Support Kubernetes clusters in toil launch-cluster #3357
  • Additional check for predecessors. #3417
  • Allow setOptions to accept an argparse group. #3426

Bug Fixes

  • AWS: Handle socket timeout during AWS discovery #3503
  • Use back-up flatcar AMI if not found. #3513
  • Lsf command parser #3475
  • Retry on s3 throttling. #3504
  • Stop deleting all ASGs with tags #3474
  • Only determine execute permissions on files we copy/move. #3437
  • cope with an invalid HTTP_PROXY #3447
  • Fix failing Google jobstore test #3420
  • Update Google job store #3412

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, Arthur Rand, @mberacochea, @Jessime, @thiagogenez, @julian-klode!

Contents

    sartography A powerful workflow engine implemented in pure Python
    hooopo Petri Net Workflow Engine for Ruby.
    osmedeus Community Workflow for the Osmedeus Engine that describes basic reconnaissance methodology for you to build your own
    godotengine Godot Engine – Multi-platform 2D and 3D game engine
    danielgerlag Lightweight workflow engine for .NET Standard
    ScriptArtist Makes the UI elements scalable to fit inside the container area.
    enigo-rs enigo - Enigo lets you control your mouse and keyboard in an abstract way on different operating systems (currently only Linux, macOS, Win – Redox and *BSD planned)
    tenacityteam Tenacity is an easy-to-use, cross-platform multi-track audio editor/recorder for Windows, MacOS, GNU/Linux
    insitro Yet another redundant workflow engine
    cormiertyshawn895 Run Aperture, iPhoto, and iTunes on macOS Catalina. Xcode 11.4.1 on macOS Mojave. Final Cut Pro 7, Logic Pro 9, and iWork ’09 on macOS Mojave or macOS High Sierra.
    winfsp Cross-platform FUSE library for Go - Works on Windows, macOS, Linux, FreeBSD, NetBSD, OpenBSD
    dthree Cross-platform Linux commands in ES6
    uber Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
    redstrate Cross-platform 3D game engine featuring real-time physically based rendering
    scottcgi A cross-platform, open-source, pure C game engine for mobile game.
    flowable A compact and highly efficient workflow and Business Process Management (BPM) platform for developers, system admins and business users.
    42coders The Workflow Package add Drag & Drop Workflows to your Laravel Application.
    tillkruss An ultra-fast Laravel docs search workflow for Alfred 3
    devxoul Pure DI in Swift
    Famous Famous Engine a free and open source JavaScript rendering engine.
    google Cross platform Go logging library.
    Share this repo

    Related Repos


    Miscellaneous
    100
    Playpen is a set of modern, gas optimized staking pool contracts.
    ZeframLou Playpen Playpen is a set of modern, gas optimized staking pool contracts. Features Support for both ERC20 staking and ERC721 staking Can start new rew
     
    Miscellaneous
    47
    DIAL(Did I Alert Lambda?) is a centralised security misconfiguration detection framework w...
    CRED-CLUB DIAL Workloads on cloud provide equal opportunities for hackers as much as they do for internal teams. Cloud-native companies are open to attacks from
     
    Miscellaneous
    42
    Code for "Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation"
    alvinliu0 Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation Code for "Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation"
     
    Miscellaneous
    40
    IDA Frida Plugin for tracing something interesting.
    P4nda0s IDAFrida A simple IDA plugin to generate FRIDA script. Edit template for functions or you can use the default template. Select functions you want to t
     
    Miscellaneous
    15
    https://t.me/master_thyself
    evmn Know Thyself 中文版 Chinese Title Authors Mind Hacks 刘未鹏 编程随想: 2009 ~ 2021 Blog Archives Program Think 晚点 LatePost 宋玮…… English Blogs Title Authors Essay
     
    Miscellaneous
    717
    a Jupyter kernel for Clojure
    clojupyter A Jupyter kernel for Clojure - run Clojure code in Jupyter Lab, Notebook and Console. Table of Contents Getting Started Installation Usage Scenarios -
     
    Miscellaneous
    132
    using your namespace as a notebook
    scicloj notespace Notebook experience in your Clojure namespace What is it? This tool is an attempt to answer the following question: can we have a notebook-l
     
    Miscellaneous
    794
    Python bindings for Clojure
    clj-python Deep Clojure/Python Integration Version Info New Versions Are HOT! Huge New Features! You Can't Afford To Miss Out! API Documentation Java API - you c
     
    Miscellaneous
    78
    Turn data into functions! A simple and functional machine learning library written in elix...
    mrdimosthenis emel Turn data into functions! A simple and functional machine learning library written in elixir. Installation The package can be installed by adding
     
    Miscellaneous
    6
    An optional component handler for hikari, inspired by discord.py's views.
    HyperGH hikari-miru An optional component handler for hikari, inspired by discord.py's views. Installation pip install git+https://github.com/HyperGH/hikari-m
     
    Miscellaneous
    5
    TikTok 4L and 4C checker that doesn't count banned usernames as available
    cliphd TikTok 4L and 4C checker that doesn't count banned usernames as available. Once a username is available, it will send it to your Discord Webhook
     
    Miscellaneous
    4
    TBOT Binance Smart Chain token
    TBOT-Project tbot-bsc-token TBOT Binance Smart Chain token Minting transaction: https://bscscan.com/tx/0xd387a96ac15a616f9b17bf9af84d63fd54de98e176a55bf11bb4f08855
     
    Miscellaneous
    4
    notes on reusable p5.js coding patterns
    M0nica p5.js Notes on random p5.js coding patterns Table of Contents Positioning Elements Responding to User Input Positioning Elements centering elements fu
     
    Miscellaneous
    3
    smart contracts for the implementation of a networked money stream using the ERC1155 stand...
    phydy MoneyStreamNetwork #Short Description A Tradable Cash-Flow DeFi-NFT that gives a holder the power to leverage their superfluid flow to get loans or fu
     
    Miscellaneous
    3
    Fb token and cookie making tools
    niloy0 🥀 🍁 ASSALAMU WALAYKUM 🥰 🥀 🥀 😻 🌺 HI I AM N!LOY 🥀 😘 😻 ! 🥀 😻 WELCOME TO MY PROFILE 🥰 MY PROFILE VISITORS : GITHUB STATES : CONTACT WITH ME :