Home / Python / Miscellaneous

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

Toil is a scalable, efficient, cross-platform (Linux & macOS) pipeline management system, written entirely in Python, and designed around the principles of functional programming. Check the website for a description of Toil and i
Information
Category: Python / Miscellaneous
Watchers: 54
Star: 761
Fork: 216
Last update: Oct 4, 2021
Join the chat at https://gitter.im/bd2k-genomics-toil/Lobby

Toil is a scalable, efficient, cross-platform (Linux & macOS) pipeline management system, written entirely in Python, and designed around the principles of functional programming.

  • Check the website for a description of Toil and its features.
  • Full documentation for the latest stable release can be found at Read the Docs.
  • Please subscribe to low-volume announce mailing list so we keep you informed
  • Google Groups discussion forum
  • See our occasional blog for tutorials.

Notes:

releases/5.5.0(Sep 25, 2021)

Changelog

CWL

  • Add podman support; and other fixes from recent cwltool #3799
  • Add streaming feature for cwltoil #3694
  • Warn users if a different cwltool version is installed #3686
  • Turn on all Kubernetes CWL tests that are expected to work on Singularity #3720
  • Fix CWL in toil docs jobstore usage #3728
  • DOC: update versions of CWL support
  • Allow filestore bypass #3652

Misc

  • Numerous Type Hints. #3705 #3701 #3693 #3691 #3663 #3688 #3642 #3684 #3682 #3680 #3666 #3675
  • Single source of truth for job state #3776
  • Do not set default for statePollingWait #3774
  • Use absolute local paths when exportFile/importFile do not detect a schema #3767
  • Multi-zone balancing within regions for AWS autoscaling groups #3746
  • Migrate cloud-config to ignition #3488
  • 🎡 Wheel Of Issues 🎰 #3760
  • Add back addBatchSystemFactory function #3754
  • Redirect stderr to /dev/null of lsf conf queries #3751
  • Set number of cores based on job.cores for OpenMP applications #3739
  • Google jobstore batching #3740
  • Locations of CLI option docs
  • Add AWS provisioner storage system #3727
  • Set cls.bucket #3726
  • Stream vs dowload jobs #3722
  • Update Toil's main python test version to 3.8. #3669
  • Move Travis tests to Gitlab #3675

Bug Fixes

  • Don't leak symlinks #3795
  • Prevent exception from being raised when modifying dir permissions for clean up #3778
  • Fix scontrol output parsing #3793
  • Fix AttributeError #3742
  • Workaround for S3 in us-east-1 #3710
  • Time data format #3708
  • Fix leader.py batch system std files prefix glob #3679

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @Hexotical, @ionox0, @gmloose, @juanesarango, @mhpopescu, @mberacochea, @nikhil!

releases/5.4.0(Jun 25, 2021)

Changelog

CWL

  • Fix cwl and wdl dependency bleed and add stand alone tests. #3582
  • Use MpiConfig.load() to handle MPI config file #3574
  • Add support for MPI with CWL. #3525

Misc

  • Numerous Type Hints. #3571 #3634 #3626 #3625 #3616 #3614 #3601 #3592 #3590 #3581
  • Configurable Grafana port #3597
  • Handle streaming reads from the cache when the data isn't written back yet #3595
  • Balance over pools using AWS ASGs #3490
  • Change how Kubernetes schedules and scales for hopefully better scale-down behavior #3587
  • Add Owner tags to our AWS buckets when testing. #3577
  • Allow workdir override #3586
  • Additional info for "Permission denied" error #3579
  • Consolidate memory functions #3529
  • Pen children #3482
  • Sniff raid better #3526
  • Add a propagation policy to batch delete #3522
  • Cleanup script for buckets, sdb domains, instance profiles, and roles. #3373
  • Add decorator for flaky tests. #3510

Bug Fixes

  • Enforce valid uuid from getNodeId() #3611
  • Removed extra memory multiplication #3608
  • Fix non integers lsf memory requested #3609
  • Fixes jobCommand unpack #3610
  • Fix CWL test 20 on Kubernetes #3572
  • Fix #3579 breaking AWS docs #3596
  • Update cactus test to fix broken bucket links. #3594
  • Allow cleaning up a job whose overlargeID fell off #3584
  • Only deploy a user script if we can deploy user scripts #3518
  • Reduce path name lengths. #3438
  • Insist on credentials for testing AMI finding #3524

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @thiagogenez, @julian-klode, @darafferty, @nikhil, @mhpopescu!

releases/5.3.0(Mar 25, 2021)

Changelog

CWL

  • Run CWL conformance tests on Kubernetes. #3323
  • CWL symlinking files into work directory #3445
  • set resource reqs for all CWL_INTERNAL_JOBS #3442

WDL

  • Add WDL flatten() function + tests. #3485
  • Add WDL collect_by_key() function + tests. #3476
  • Add WDL keys() function + tests. #3460
  • Add WDL as_pairs() function + tests. #3364
  • Add as_map() WDL function #3448
  • Add basic WDL development support #3434
  • Add basic WDL 1.0 support #3421
  • Update dictionary structure in AnalyzeWDL. #3416

Misc

  • Add extra context for disk usage warning. #3495
  • Update developing.rst #3479
  • Allow text/encode options for read and write functions in file store #3428
  • Type hinting and mypy checking additions #3470 # #3458 #3456
  • Quality of life improvements for cactus. #3463
  • Add checklists for reviewing and merging PRs #3432
  • Ensure that python3 and not python (2) is used #3446
  • Support Kubernetes clusters in toil launch-cluster #3357
  • Additional check for predecessors. #3417
  • Allow setOptions to accept an argparse group. #3426

Bug Fixes

  • AWS: Handle socket timeout during AWS discovery #3503
  • Use back-up flatcar AMI if not found. #3513
  • Lsf command parser #3475
  • Retry on s3 throttling. #3504
  • Stop deleting all ASGs with tags #3474
  • Only determine execute permissions on files we copy/move. #3437
  • cope with an invalid HTTP_PROXY #3447
  • Fix failing Google jobstore test #3420
  • Update Google job store #3412

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, Arthur Rand, @mberacochea, @Jessime, @thiagogenez, @julian-klode!

releases/5.2.0(Jan 8, 2021)

Changelog

CWL

  • Update to the latest cwltool (3.0.20201121085451 -> 3.0.20201203173111). #3375
  • Add better handling for potentially mis-ordered CWL args. #3395
  • Confirm CommandInputParameter expression can receive a File object. #3350

WDL

  • Make AnalyzeWDL abstract to allow implementation for different WDL versions #3391

Misc

  • Clean up memoize.py. #3374
  • Refactor bioio library, simplify logging, and small adjustments to utils. #3351
  • Use regular division for memory calculation on LSF. #3387
  • Remove unused --nodeOptions arg. #3397
  • Get boto S3 args for minio from environment variables. (#3370)
  • Update EC2 lists. #3376
  • Add a retry to AWS destroy. #3379
  • Regular division for memory calculation in LSF. #3387
  • Add option to blank mem allocation for SLURM. #3399
  • Preserve file permissions on imported/exported files #3322

Bug Fixes

  • Preserve file permissions on imported/exported files. #3322
  • Make building a flattened list of live jobs non-recursive. #3394
  • Move AWS job store uploads/downloads to boto3 (reopened) #3400
  • Use a Docker Hub mirror when pulling Docker images. #3411
  • Corrected gridengine maxMEM check to use memorystring #3410

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @douglowe, @stevekm , @thiagogenez!

releases/5.1.0(Dec 16, 2020)

Version 5.1.0

Changelog

WDL

  • Add WDL cross function. #3360

Bug Fixes

  • Better error message for insufficient disk. #3362
  • Fix maxMEM comparison in LSF. #3369

Thank you to our contributors: @mr-c , @adamnovak , @w-gao , @jonathanxu18 !

releases/5.0.0(Dec 3, 2020)

Caching is now turned on by default.

CWL v1.1 is also now fully supported and passing all conformance tests.

The following batch system names are now deprecated (#3225) and replaced by:

  • singleMachine -> single_machine
  • gridEngine -> grid_engine
  • LSF -> lsf
  • Mesos -> mesos
  • Slurm -> slurm
  • Torque -> torque
  • HTCondor -> htcondor
  • Kubernetes -> kubernetes
  • k8s -> kubernetes

Changelog

CWL

  • Upgrade to a CWL v1.2.0 capable cwltool (3.0.20200709181526 -> 3.0.20200807132242). #3137
  • Export to a bucket if cwl's CreateFile is specified. #3124
  • CWL v1.1 Support: loadContents if True for StepValueFrom objects. #3266
  • CWL: Fix workdir permissions. #3229
  • Test CWL secondary file from s3. #3311
  • CWL v1.1 Support: Inplace update has side effect on directory content. #3280
  • More CWL v1.2 conformance tests. #3336
  • CWL: Reformat w/ Black and better support for tmpdir_prefix with suffix. #3333

WDL

  • Add ceil() and floor() WDL functions. #3168
  • Create a stdout and stderr file for each WDL task. #3181
  • Add WDL write functions and builtin tests. #3236
    • write_lines
    • write_tsv
    • write_json
    • write_map
  • Add WDL read functions and builtin tests. #3244
    • read_lines
    • read_tsv
    • read_json
    • read_map
    • read_int
    • read_string
    • read_float
    • read_boolean
  • Add transpose() WDL function. #3273
  • Add length() WDL function. #3307
  • Add sub() WDL function. #3252
  • Add range() WDL function. #3277
  • Add size() WDL function. #3255
  • Add zip() WDL function. #3355
  • Implement WDL Pair type. #3304
  • Refactor WDL types #3335

Kubernetes

  • Adding Labels to Pods/Jobs. #3233
  • Make kubernetes & botocore packages fully optional. #3259 #3261
  • KubeWatch optimization. #3227
  • Catch other kubernetes imports. #3261
  • Automatically infer singularity for kubernetes. #3279

Misc

  • Allow Caching By Default. #3111
  • Update dependencies and loosen restrictions slightly. #3139
  • Ensure that falsey (but non-null) values are considered. #3143
  • Remove pathlib. #3169
  • Move AWS job store uploads/downloads to boto3. #3153
  • Update prom/node-exporter to use quay.io. #3186
  • Add support for TOIL_CUSTOM_INIT_COMMAND. #3183
  • Update docker module version to 4.3.1. #3222
  • Add 'stream' and 'demux' options to apiDockerCall. #3224
  • Use boto3 to create our leader node. #3145
  • Prevent job kind filenames from getting too long. #3230
  • Add a retry decorator. #3144
  • Remove mesos offer messages. #3308
  • Job Concept Unification. #3250
  • Add reporting for accessed files of failed jobs. #3309
  • Add coverage and refactor. #3314
  • Expose --statePollingWait param through toil. #3321
  • Record the maximum memory used by LSF jobs. #3327
  • Remove custom math functions and replace with built ins. #3338
  • Define a changelog process. #3316
  • SLURM: getting job details from scontrol show jobs was non-functional and has now been fixed. This behavior in the case of sacct not being configured now works. #3346
  • LSF: handle unicode configuration files #3354
  • Replace abssympath w/ builtin. #3356

Bug Fixes

  • Turn off watch code in Kubernetes. #3175
  • Check against joining the current thread during CachingFileStore destructor. #3187
  • Slurm.py error, undefined stdout - stdout should be stderr in code. #3194
  • Fix restart flag error (issue #3094). #3142
  • Batch System generates a key error instead of a usage message. #3225
  • Remove Python 2 imports in WDL. #3228
  • Move checkForDeadlocks() in the leader. #3234
  • Fix boto3 migration bugs. #3243
  • Fix None check when parsing WDL JSON file. #3272
  • Fix broken stdout/stderr in torque job. #3306
  • Add doubleMem for LSF jobs that die due to imposed memory limit. #3313
  • Add 500 status to the list of retriable SDB BotoServerErrors. #3329
  • Add a retry logic for BucketNotEmpty error. #3341
  • Fix None check when parsing WDL JSON file. #3272
  • Warn when trying to clean non-existent job store #3288
  • Only rm a statefile that exists. #3348
  • CWL: Remove unprocessed secondaryFiles from both Dict and List types.

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @diekhans, @w-gao, @jonathanxu18, @jeffrey856, @davidlougheed, @altairwei, @mberacochea, @drkennetz, @ionox0!

Contents
    sartography A powerful workflow engine implemented in pure Python
    hooopo Petri Net Workflow Engine for Ruby.
    godotengine Godot Engine – Multi-platform 2D and 3D game engine
    danielgerlag Lightweight workflow engine for .NET Standard
    enigo-rs enigo - Enigo lets you control your mouse and keyboard in an abstract way on different operating systems (currently only Linux, macOS, Win – Redox and *BSD planned)
    ScriptArtist Makes the UI elements scalable to fit inside the container area.
    tenacityteam Tenacity is an easy-to-use, cross-platform multi-track audio editor/recorder for Windows, MacOS, GNU/Linux
    billziss-gh Cross-platform FUSE library for Go - Works on Windows, macOS, Linux, FreeBSD, NetBSD, OpenBSD
    cormiertyshawn895 Run Aperture, iPhoto, and iTunes on macOS Catalina. Xcode 11.4.1 on macOS Mojave. Final Cut Pro 7, Logic Pro 9, and iWork ’09 on macOS Mojave or macOS High Sierra.
    dthree Cross-platform Linux commands in ES6
    uber Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
    redstrate Cross-platform 3D game engine featuring real-time physically based rendering
    42coders The Workflow Package add Drag & Drop Workflows to your Laravel Application.
    tillkruss An ultra-fast Laravel docs search workflow for Alfred 3
    flowable A compact and highly efficient workflow and Business Process Management (BPM) platform for developers, system admins and business users.
    PierreQuentel A fast, pure Python in-memory database engine
    scottcgi A cross-platform, open-source, pure C game engine for mobile game.
    Famous Famous Engine a free and open source JavaScript rendering engine.
    clangen a cross-platform, terminal-based music player, audio engine, metadata indexer, and server in c++
    google Cross platform Go logging library.
    ahmadawais [✔️ ℹ️ ⚠️ ✖] Cross platform CLI Alerts with colors & colored symbols for success, info, warning, error. Works on macOS, Linux, and Windows.
    Resource links
    https://github.com/DataBiosphere/toil
    Share this repo

    Related Repos


    Miscellaneous
    12
    Python FFI bindings for libsecp256k1 (maintained)
    rustyrussell secp256k1-py Python FFI bindings for libsecp256k1 (an experimental and optimized C library for EC operations on curve secp256k1). Previously maintaine
     
    Miscellaneous
    11
    Shiny Wechat Pay SDK for Python
    Obrisk WeChat third-party Python SDK master: Read the Documentation Features Common public platforms passively respond and actively call APIs WeChat Pay API
     
    Miscellaneous
    10
    RSS reader for EAF.
    emacs-eaf EAF RSS Reader RSS Reader application for the Emacs Application Framework. Load application (add-to-list 'load-path "~/.emacs.d/site-lisp/eaf-rss-read
     
    Miscellaneous
    1k
    an awesome list of internet services that don't require logins or registrations
    fiatjaf an awesome list of internet services that don't require logins or registrations
     
    Miscellaneous
    19
    Peer2Peer Nix-Binary-Cache
    cid-chan Peerix Peerix is a peer-to-peer binary cache for nix derivations. Every participating node can pull derivations from each other instances' respective
     
    Miscellaneous
    16
    A httpx token generator for discord [ hcaptcha bypass ]
    Aced1337 Discord-Token-Generator-Yazato A httpx token generator for discord This generator was developed by Aced#0001, Dreamy Tos Follower#0001, Scripted#0131
     
    Miscellaneous
    10
    Python packages for M1 users
    coaxsoft ARM python packages This repository contains pre-compiled packages for arm64. The idea of this repo is to have one storage for the packages (while the
     
    Miscellaneous
    161
    This program writes READMEs for you.
    tom-doerr 🤖 codex-readme 📜 The gif shows how the readme below was generated. To generate your own readmes you need to get access to the Codex API (https://ope
     
    Miscellaneous
    128
    Monitor creation, deletion and changes to LDAP objects live during your pentest or system ...
    p0dalirius LDAP Monitor Monitor creation, deletion and changes to LDAP objects live during your pentest or system administration! With this tool you can quickly
     
    Miscellaneous
    35
    Programming Language and Framework Flash Cards as Study Material
    vandadnp Flash Cards Programming Language and Framework Flash Cards as Study Material I recommend that you download the PDF versions since they have active lin
     
    Miscellaneous
    31
    Find the best repos to contribute to, right from Discord!
    savioxavier repo-finder-bot Find the best repos to contribute to, right from Discord! Add to your server FAQs Hmm. What's this? This is the Repo Finder Bot, a bot
     
    Miscellaneous
    27
    This is a crypto trading bot that scans the Binance Annoucements page for new coins, and p...
    CyberPunkMetalHead gateio-trading-bot-binance-announcements This Gateio x Binance cryptocurrency trading bot scans the Binance Announcements page and picks up on new coi
     
    Miscellaneous
    12
    Discord Token Creator :hot_face:
    notdropout Discord Token Creator A discord token creator that uses the service capmonster for captcha solving! Report Bug · Request Feature Features Autojoin dis
     
    Miscellaneous
    20
    The 0 config meme you all were asking for.
    mjlbach The 0 config meme you all were asking for. Usage Install: local use = require('packer').use
     
    Miscellaneous
    242
    ElevenClock: A secondary clock for secondary taskbars on Windows 11
    martinet101 ElevenClock A taskbar clock for secondary taskbars on Windows 11. When microsoft's engineers were creating Windows 11, they forgot to add a clock on t