Home / Python / Miscellaneous

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

Toil is a scalable, efficient, cross-platform (Linux & macOS) pipeline management system, written entirely in Python, and designed around the principles of functional programming. Check the website for a description of Toil and i
Join the chat at https://gitter.im/bd2k-genomics-toil/Lobby

Toil is a scalable, efficient, cross-platform (Linux & macOS) pipeline management system, written entirely in Python, and designed around the principles of functional programming.

  • Check the website for a description of Toil and its features.
  • Full documentation for the latest stable release can be found at Read the Docs.
  • Please subscribe to low-volume announce mailing list so we keep you informed
  • Google Groups discussion forum
  • See our occasional blog for tutorials.

Notes:

374 Open More issues
Closed

Continue the pymesos port

I'm PR-ing this against the other PR to see if I can convince the tests to run, because Courtyard can't run the tests properly due to the absence of Docker availability.

opened Aug 31, 2018 by adamnovak 75
Closed

WDL basic support. Alpha testing phase.

Support for .wdl files currently working for the GATK tutorials 1-4 on the WDL webpage. Have not tested on any other files, so currently this is still in alpha.

in progress 
opened Aug 14, 2017 by DailyDreaming 74
Closed

Initial PBS/Torque support and AbstractBatchSystem refactoring

This is a preliminary run at adding PBS/Torque support, designed off the SLURM and SGE classes. Tests have also been added.

Resolves #1163

opened Oct 6, 2016 by cjfields 68
Closed

Add Spark cluster service (resolves #774)

Resolves #774.

Ping @benedictpaten

firecloud services spark/adam 
opened Apr 13, 2016 by fnothaft 63
Closed

Support for Slurm batchSystem (resolves #682)

Implements support for slurm scheduler

  • SlurmBatchSystem class is modeled after GridengineBatchSystem
  • Maps AbstractBatchSystem methods to sbatch, squeue, scancel, sacct, sinfo subprocess calls
  • Includes SlurmBatchSystemTest for testing system
opened Mar 17, 2016 by dleehr 49
Closed

Set workdir to default to the executor sandbox in mesos (resolves #1143)

resolves #1143

The hierarchy of workdir is now: User-specified workdir > (sandox if mesos) > ENV TOIL_WORKDIR > ENV TMPDIR

opened Sep 7, 2016 by arkal 45
Open

Research Graviton and Arm

┆Issue is synchronized with this Jira Task ┆Epic: Graviton Summer 2021 Project ┆Issue Number: TOIL-969

opened Jul 23, 2021 by unito-bot 0
Open

Initial setup for AWS provisioner storage system

Closes #3714.

Changelog Entry

To be copied to the draft changelog by merger:

  • Add storage system to the AWS provisioner

Reviewer Checklist

  • [ ] Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
    • [ ] If it is coming from an external repo, make sure to pull it in for CI with: 
      contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    • [ ] If there is no associated issue, create one.
  • [ ] Read through the code changes. Make sure that it doesn't have:
    • [ ] Addition of trailing whitespace.
    • [ ] New variable or member names in camelCase that want to be in snake_case.
    • [ ] New functions without type hints.
    • [ ] New functions or classes without informative docstrings.
    • [ ] Changes to semantics not reflected in the relevant docstrings.
    • [ ] New or changed command line options for Toil workflows that are not reflected in docs/running/cliOptions.rst
    • [ ] New features without tests.
  • [ ] Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
  • [ ] Finish the review with an overall description of your opinion.

Merger Checklist

  • [ ] Make sure the PR passes tests.
  • [ ] Make sure the PR has been reviewed since its last modification. If not, review it.
  • [ ] Merge with the Github "Squash and merge" feature.
    • [ ] If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
  • [ ] Copy its recommended changelog entry to the Draft Changelog.
  • [ ] Append the issue number in parentheses to the changelog entry.
opened Jul 22, 2021 by w-gao 0
Open

Create pytest marker for internet-needing tests

Should fix #3444.

Changelog Entry

To be copied to the draft changelog by merger:

  • PR submitter writes their recommendation for a changelog entry here

Reviewer Checklist

  • [ ] Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
    • [ ] If it is coming from an external repo, make sure to pull it in for CI with: 
      contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    • [ ] If there is no associated issue, create one.
  • [ ] Read through the code changes. Make sure that it doesn't have:
    • [ ] Addition of trailing whitespace.
    • [ ] New variable or member names in camelCase that want to be in snake_case.
    • [ ] New functions without type hints.
    • [ ] New functions or classes without informative docstrings.
    • [ ] Changes to semantics not reflected in the relevant docstrings.
    • [ ] New or changed command line options for Toil workflows that are not reflected in docs/running/cliOptions.rst
    • [ ] New features without tests.
  • [ ] Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
  • [ ] Finish the review with an overall description of your opinion.

Merger Checklist

  • [ ] Make sure the PR passes tests.
  • [ ] Make sure the PR has been reviewed since its last modification. If not, review it.
  • [ ] Merge with the Github "Squash and merge" feature.
    • [ ] If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
  • [ ] Copy its recommended changelog entry to the Draft Changelog.
  • [ ] Append the issue number in parentheses to the changelog entry.
opened Jul 22, 2021 by jonathanxu18 0
Open

Summer 2021 work project- Jon

┆Issue is synchronized with this Jira Task ┆Issue Number: TOIL-968

intern 
opened Jul 21, 2021 by unito-bot 0
Open

Documents/SOPs that need to be updated or created

┆Issue is synchronized with this Jira Epic ┆Issue Number: TOIL-967

opened Jul 21, 2021 by unito-bot 0
Open

CachingFileStoreTestWithAwsJobStore.testMultipleJobsReadSameCacheHitGlobalFile is flakey

In https://ucsc-ci.com/databiosphere/toil/-/jobs/90501 I observed a failure that the PR cannot have caused:

_ CachingFileStoreTestWithAwsJobStore.testMultipleJobsReadSameCacheHitGlobalFile _
Traceback (most recent call last):
  File "/builds/databiosphere/toil/src/toil/test/src/fileStoreTest.py", line 799, in testMultipleJobsReadSameCacheHitGlobalFile
    self._testMultipleJobsReadGlobalFileFunction(cacheHit=True)
  File "/builds/databiosphere/toil/src/toil/test/src/fileStoreTest.py", line 835, in _testMultipleJobsReadGlobalFileFunction
    assert int(y.read()) > 2
AssertionError

We should figure out why this test can sometimes fail, and fix it so that it can never fail.

┆Issue is synchronized with this Jira Task ┆Issue Number: TOIL-966

opened Jul 20, 2021 by adamnovak 0

releases/5.4.0(Jun 25, 2021)

releases/5.3.0(Mar 25, 2021)

releases/5.2.0(Jan 8, 2021)

Changelog

CWL

  • Update to the latest cwltool (3.0.20201121085451 -> 3.0.20201203173111). #3375
  • Add better handling for potentially mis-ordered CWL args. #3395
  • Confirm CommandInputParameter expression can receive a File object. #3350

WDL

  • Make AnalyzeWDL abstract to allow implementation for different WDL versions #3391

Misc

  • Clean up memoize.py. #3374
  • Refactor bioio library, simplify logging, and small adjustments to utils. #3351
  • Use regular division for memory calculation on LSF. #3387
  • Remove unused --nodeOptions arg. #3397
  • Get boto S3 args for minio from environment variables. (#3370)
  • Update EC2 lists. #3376
  • Add a retry to AWS destroy. #3379
  • Regular division for memory calculation in LSF. #3387
  • Add option to blank mem allocation for SLURM. #3399
  • Preserve file permissions on imported/exported files #3322

Bug Fixes

  • Preserve file permissions on imported/exported files. #3322
  • Make building a flattened list of live jobs non-recursive. #3394
  • Move AWS job store uploads/downloads to boto3 (reopened) #3400
  • Use a Docker Hub mirror when pulling Docker images. #3411
  • Corrected gridengine maxMEM check to use memorystring #3410

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @douglowe, @stevekm , @thiagogenez!

releases/5.1.0(Dec 16, 2020)

Version 5.1.0

Changelog

WDL

  • Add WDL cross function. #3360

Bug Fixes

  • Better error message for insufficient disk. #3362
  • Fix maxMEM comparison in LSF. #3369

Thank you to our contributors: @mr-c , @adamnovak , @w-gao , @jonathanxu18 !

releases/5.0.0(Dec 3, 2020)

Caching is now turned on by default.

CWL v1.1 is also now fully supported and passing all conformance tests.

The following batch system names are now deprecated (#3225) and replaced by:

  • singleMachine -> single_machine
  • gridEngine -> grid_engine
  • LSF -> lsf
  • Mesos -> mesos
  • Slurm -> slurm
  • Torque -> torque
  • HTCondor -> htcondor
  • Kubernetes -> kubernetes
  • k8s -> kubernetes

Changelog

CWL

  • Upgrade to a CWL v1.2.0 capable cwltool (3.0.20200709181526 -> 3.0.20200807132242). #3137
  • Export to a bucket if cwl's CreateFile is specified. #3124
  • CWL v1.1 Support: loadContents if True for StepValueFrom objects. #3266
  • CWL: Fix workdir permissions. #3229
  • Test CWL secondary file from s3. #3311
  • CWL v1.1 Support: Inplace update has side effect on directory content. #3280
  • More CWL v1.2 conformance tests. #3336
  • CWL: Reformat w/ Black and better support for tmpdir_prefix with suffix. #3333

WDL

  • Add ceil() and floor() WDL functions. #3168
  • Create a stdout and stderr file for each WDL task. #3181
  • Add WDL write functions and builtin tests. #3236
    • write_lines
    • write_tsv
    • write_json
    • write_map
  • Add WDL read functions and builtin tests. #3244
    • read_lines
    • read_tsv
    • read_json
    • read_map
    • read_int
    • read_string
    • read_float
    • read_boolean
  • Add transpose() WDL function. #3273
  • Add length() WDL function. #3307
  • Add sub() WDL function. #3252
  • Add range() WDL function. #3277
  • Add size() WDL function. #3255
  • Add zip() WDL function. #3355
  • Implement WDL Pair type. #3304
  • Refactor WDL types #3335

Kubernetes

  • Adding Labels to Pods/Jobs. #3233
  • Make kubernetes & botocore packages fully optional. #3259 #3261
  • KubeWatch optimization. #3227
  • Catch other kubernetes imports. #3261
  • Automatically infer singularity for kubernetes. #3279

Misc

  • Allow Caching By Default. #3111
  • Update dependencies and loosen restrictions slightly. #3139
  • Ensure that falsey (but non-null) values are considered. #3143
  • Remove pathlib. #3169
  • Move AWS job store uploads/downloads to boto3. #3153
  • Update prom/node-exporter to use quay.io. #3186
  • Add support for TOIL_CUSTOM_INIT_COMMAND. #3183
  • Update docker module version to 4.3.1. #3222
  • Add 'stream' and 'demux' options to apiDockerCall. #3224
  • Use boto3 to create our leader node. #3145
  • Prevent job kind filenames from getting too long. #3230
  • Add a retry decorator. #3144
  • Remove mesos offer messages. #3308
  • Job Concept Unification. #3250
  • Add reporting for accessed files of failed jobs. #3309
  • Add coverage and refactor. #3314
  • Expose --statePollingWait param through toil. #3321
  • Record the maximum memory used by LSF jobs. #3327
  • Remove custom math functions and replace with built ins. #3338
  • Define a changelog process. #3316
  • SLURM: getting job details from scontrol show jobs was non-functional and has now been fixed. This behavior in the case of sacct not being configured now works. #3346
  • LSF: handle unicode configuration files #3354
  • Replace abssympath w/ builtin. #3356

Bug Fixes

  • Turn off watch code in Kubernetes. #3175
  • Check against joining the current thread during CachingFileStore destructor. #3187
  • Slurm.py error, undefined stdout - stdout should be stderr in code. #3194
  • Fix restart flag error (issue #3094). #3142
  • Batch System generates a key error instead of a usage message. #3225
  • Remove Python 2 imports in WDL. #3228
  • Move checkForDeadlocks() in the leader. #3234
  • Fix boto3 migration bugs. #3243
  • Fix None check when parsing WDL JSON file. #3272
  • Fix broken stdout/stderr in torque job. #3306
  • Add doubleMem for LSF jobs that die due to imposed memory limit. #3313
  • Add 500 status to the list of retriable SDB BotoServerErrors. #3329
  • Add a retry logic for BucketNotEmpty error. #3341
  • Fix None check when parsing WDL JSON file. #3272
  • Warn when trying to clean non-existent job store #3288
  • Only rm a statefile that exists. #3348
  • CWL: Remove unprocessed secondaryFiles from both Dict and List types.

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @diekhans, @w-gao, @jonathanxu18, @jeffrey856, @davidlougheed, @altairwei, @mberacochea, @drkennetz, @ionox0!

releases/4.2.0(Aug 12, 2020)

  • Don't log in the inner scheduling loop. #3065
  • Handle anticipated 404 errors without retrying. #3067
  • Test and reimplement AMI finding via Flatcar JSON feed. #3061
  • Bump Enlighten version to partly fix #3069. #3070
  • Get wait duration is far too short of a rest for lsf. #3076
  • Unify logging format. #3073
  • Support CWL 1.1 Listing. #3058
  • Enable CWL v1.2.0-dev3 (3.0.20200324120055 -> 3.0.20200530110633). #3092
  • Enable CWL v1.2.0-dev4 (3.0.20200530110633 -> 3.0.20200709181526). #3105
  • Add --nodeStorageOverrides option. #3096
  • More robust CWL filepaths and secondary files. #3114
  • Replace custom mkdir_p with the built in. #3123
  • Move conditional execution test to before fill_in_defaults. #3117

Bug Fixes

  • Stop CWL from trying to copy downloaded directories from somewhere local. #3053
  • Exit the worker with 1 when jobs fail so the batch system can see. #3052
  • Eliminate recursion in root finding to fix #3080. #3081
  • Add a Cactus Kubernetes test to the integration tests on Gitlab. #3077
  • Chaining doesn't make sense for CWL. #3091
  • Fix failed job accounting. #3103
  • Improve lsf batchsystem stability. #3101
  • Deduplicate updated jobs. #3108
  • Redundant virtualenv check. #3135
  • Change Slurm command output processing. #3133
  • Fix for log file being linked more than once. #3130
  • Wrap Kubernetes watches to fix #3125. #3126
  • Create dirs that do not exist when bind-mounting to docker. #3120
  • Fix database state when file upload fails. #3122

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @diekhans, @nikhil, @drkennetz, @jonathanxu18, @tobiaszjarosiewicz, @ionox0!

Information
Category: Python / Miscellaneous
Watchers: 54
Star: 750
Fork: 214
Last update: Jul 25, 2021

Contents
    Resource links
    https://github.com/DataBiosphere/toil
    Share this repo
    Most popular
    github A collection of useful .gitignore templates
    tonsky Free monospaced font with programming ligatures
    kelseyhightower The best way to write secure and reliable applications. Write nothing; deploy nowhere.
    awesomedata A topic-centric list of HQ open datasets.
    mtdvio A collection of (mostly) technical things every software developer should know
    florinpop17 A Collection of application ideas which can be used to improve your coding skills.
    DopplerHQ :octocat: A curated awesome list of lists of interview questions. Feel free to contribute! :mortar_board:
    sdmg15 :link: Some useful websites for programmers.
    minimaxir The Big List of Naughty Strings is a list of strings which have a high probability of causing issues when used as user-input data.
    prettier Prettier is an opinionated code formatter.

    Related Repos


    Miscellaneous
    88
    This is the code for "Simple Blockchain in 5 Minutes" By Siraj Raval on Youtube
    llSourcell Simple_Blockchain_in_5_Minutes Overview This is the code for this video on Youtube by Siraj Raval on building a simple blockchain. Here is the colab. Dependencies hashlib datetime Dependencies can b
     
    Miscellaneous
    1.3k
    Safe code refactoring for modern Python.
    facebookincubator Safe code refactoring for modern Python projects. Overview Bowler is a refactoring tool for manipulating Python at the syntax tree level. It enables safe, large scale code modifications while guaranteeing that the
     
    Miscellaneous
    1.2k
    Geometry Processing Library for Python
    PyMesh About PyMesh PyMesh is a code base developed by Qingnan Zhou for his PhD research at New York University. It is a rapid prototyping platform focused on geometry processing. PyMesh is written with both C++ and Python, where co
     
    Popular
    Miscellaneous
    6.1k
    Ready-to-run Docker images containing Jupyter applications
    jupyter Jupyter Docker Stacks Jupyter Docker Stacks are a set of ready-to-run Docker images containing Jupyter applications and interactive computing tools. Maintainer Help Wanted We value all positive contributions to t
     
    Miscellaneous
    1.2k
    Use Golang to implement PHP's common built-in functions.
    syyongx PHP2Go Use Golang to implement PHP's common built-in functions. About 140+ functions have been implemented. Install go get github.com/syyongx/php2go Requirements Go 1.10 or above. PHP Fun
     
    Miscellaneous
    219
    A no-nonsense c-like structure parsing library for Python
    fox-it dissect.cstruct Structure parsing in Python made easy. With cstruct, you can write C-like structures and use them to parse binary data, either as file-like objects or bytestrings. Parsing binary data with cstruct feels familiar
     
    Miscellaneous
    102
    KiCAD to Boardview exporter reads KiCAD PCB layout files and writes ASCII Boardview files
    whitequark KiCAD to Boardview exporter The KiCAD to Boardview exporter reads KiCAD PCB layout files and writes Boardview files. Boardview is a set of file formats used in the industry for communicating important data about PCB in post-produ
     
    Miscellaneous
    376
    A Wattman-like GTK3+ GUI
    BoukeHaarsma23 WattmanGTK This is a Python3 program which uses a simple GTK gui to view, monitor and in the future overclock a Radeon GPU on Linux. What can it do? View memory and GPU P-states including voltages. Ability to mo