Home / Python / Miscellaneous

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

Toil is a scalable, efficient, cross-platform (Linux & macOS) pipeline management system, written entirely in Python, and designed around the principles of functional programming. Check the website for a description of Toil and i
Join the chat at https://gitter.im/bd2k-genomics-toil/Lobby

Toil is a scalable, efficient, cross-platform (Linux & macOS) pipeline management system, written entirely in Python, and designed around the principles of functional programming.

  • Check the website for a description of Toil and its features.
  • Full documentation for the latest stable release can be found at Read the Docs.
  • Please subscribe to low-volume announce mailing list so we keep you informed
  • Google Groups discussion forum
  • See our occasional blog for tutorials.

Notes:

355 Open More issues
Closed

Continue the pymesos port

I'm PR-ing this against the other PR to see if I can convince the tests to run, because Courtyard can't run the tests properly due to the absence of Docker availability.

opened Aug 31, 2018 by adamnovak 75
Closed

WDL basic support. Alpha testing phase.

Support for .wdl files currently working for the GATK tutorials 1-4 on the WDL webpage. Have not tested on any other files, so currently this is still in alpha.

in progress 
opened Aug 14, 2017 by DailyDreaming 74
Closed

Initial PBS/Torque support and AbstractBatchSystem refactoring

This is a preliminary run at adding PBS/Torque support, designed off the SLURM and SGE classes. Tests have also been added.

Resolves #1163

opened Oct 6, 2016 by cjfields 68
Closed

Add Spark cluster service (resolves #774)

Resolves #774.

Ping @benedictpaten

firecloud services spark/adam 
opened Apr 13, 2016 by fnothaft 63
Closed

Support for Slurm batchSystem (resolves #682)

Implements support for slurm scheduler

  • SlurmBatchSystem class is modeled after GridengineBatchSystem
  • Maps AbstractBatchSystem methods to sbatch, squeue, scancel, sacct, sinfo subprocess calls
  • Includes SlurmBatchSystemTest for testing system
opened Mar 17, 2016 by dleehr 49
Closed

Set workdir to default to the executor sandbox in mesos (resolves #1143)

resolves #1143

The hierarchy of workdir is now: User-specified workdir > (sandox if mesos) > ENV TOIL_WORKDIR > ENV TMPDIR

opened Sep 7, 2016 by arkal 45
Open

Handle streaming reads from the cache when the data isn't written back yet

We were seeing some failures in our file job store tests because the caching file store was actually inconsistent: you wouldn't have read your own writes consistency when writing a file and then trying to stream it back, because the file would be uploaded asynchronously but the streaming read would always come from the backing job store.

This fixes the problem by making sure to do streaming reads from the cache when the file being read is there and in an uploading/uploadable state.

I ran the offending src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStore test in a loop, and this change stopped me being able to reproduce the error:

FAILED=0
for TRY_NUM in {1..5} ; do
    for ITER in {1..32} ; do
        make test_offline cov="" tests=src/toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStore >iter${ITER}.log 2>&1 &
    done
    
    for ITER in {1..32} ; do
        wait
    done
    
    for ITER in {1..32} ; do
        grep "failed job" "iter${ITER}.log"
        if [[ "${?}" == "0" ]] ; then
            echo "Failure observed in iter${ITER}.log!"
            FAILED=1
            break
        fi
    done
    
    if [[ "${FAILED}" == "1" ]] ; then
        break
    fi
    echo "No failures observed in try ${TRY_NUM}"
done

Changelog Entry

To be copied to the draft changelog by merger:

  • Streaming reads with the caching file store can now read your own writes consistently

Reviewer Checklist

  • [ ] Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
    • [ ] If it is coming from an external repo, make sure to pull it in for CI with: 
      contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    • [ ] If there is no associated issue, create one.
  • [ ] Read through the code changes. Make sure that it doesn't have:
    • [ ] Addition of trailing whitespace.
    • [ ] New variable or member names in camelCase that want to be in snake_case.
    • [ ] New functions without type hints.
    • [ ] New functions or classes without informative docstrings.
    • [ ] Changes to semantics not reflected in the relevant docstrings.
    • [ ] New or changed command line options for Toil workflows that are not reflected in docs/running/cliOptions.rst
    • [ ] New features without tests.
  • [ ] Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
  • [ ] Finish the review with an overall description of your opinion.

Merger Checklist

  • [ ] Make sure the PR passes tests.
  • [ ] Make sure the PR has been reviewed since its last modification. If not, review it.
  • [ ] Merge with the Github "Squash and merge" feature.
    • [ ] If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
  • [ ] Copy its recommended changelog entry to the Draft Changelog.
  • [ ] Append the issue number in parentheses to the changelog entry.
opened May 5, 2021 by adamnovak 0
Open

Update cactus test to fix broken bucket links.

Changelog Entry

To be copied to the draft changelog by merger:

  • Update cactus test to fix broken bucket links.

Reviewer Checklist

  • [x] Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
    • [ ] If it is coming from an external repo, make sure to pull it in for CI with: 
      contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    • [ ] If there is no associated issue, create one.
  • [x] Read through the code changes. Make sure that it doesn't have:
    • [x] Addition of trailing whitespace.
    • [x] New variable or member names in camelCase that want to be in snake_case.
    • [x] New functions without type hints.
    • [x] New functions or classes without informative docstrings.
    • [x] Changes to semantics not reflected in the relevant docstrings.
    • [x] New or changed command line options for Toil workflows that are not reflected in docs/running/cliOptions.rst
    • [x] New features without tests.
  • [x] Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
  • [x] Finish the review with an overall description of your opinion.

Merger Checklist

  • [ ] Make sure the PR passes tests.
  • [ ] Make sure the PR has been reviewed since its last modification. If not, review it.
  • [ ] Merge with the Github "Squash and merge" feature.
    • [ ] If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
  • [ ] Copy its recommended changelog entry to the Draft Changelog.
  • [ ] Append the issue number in parentheses to the changelog entry.
opened May 5, 2021 by DailyDreaming 0
Open

Update cactus to fix broken bucket links.

PR: #3594

The cactus test references a bucket of test files that was deleted and reconstituted in a new bucket. Since this is in cactus itself, we need to pull in the new commit hash so that cactus's tests have reference files that actually exist.

┆Issue is synchronized with this Jira Task ┆Issue Number: TOIL-890

opened May 5, 2021 by DailyDreaming 0
Open

toil + MPI thoughts

Follow on from https://github.com/DataBiosphere/toil/issues/3517#issuecomment-825609742

Should the Toil batch backends have a concept of requesting multiple nodes? Or shall we keep the current hack mentioned above?

┆Issue is synchronized with this Jira Task ┆Issue Number: TOIL-884

opened Apr 27, 2021 by mr-c 1
Open

Kubernetes autoscaler will scale away nodes without draining them, leading to job failures and restarts

The Kubernetes autoscaler we use is configured to scale away nodes even if they have pods running on them, even if those pods are running non-preemptable Toil jobs. This will cause those jobs to appear to fail with something like: [2021-04-24T14:05:29+0000] [MainThread] [W] [toil.batchSystems.kubernetes] Exit code and runtime unavailable; pod stopped without container terminating

If the jobs take a long time to run, then having to restart them will waste a lot of time and be inefficient. Also, if this happens to the same job too often it will run out of restarts and the workflow will fail.

We should work out how to reconfigure the Kubernetes autoscaler to not scale down nodes with jobs running on them.

┆Issue is synchronized with this Jira Task ┆Issue Number: TOIL-883

opened Apr 24, 2021 by adamnovak 1
Open

Flaky test: JobServiceTest.testServiceRecursive

Source: https://ucsc-ci.com/databiosphere/toil/-/jobs/82743

_____________________ JobServiceTest.testServiceRecursive ______________________
Traceback (most recent call last):
  File "/builds/databiosphere/toil/src/toil/test/src/jobServiceTest.py", line 177, in runToil
    Job.Runner.startToil(rootJob, options)
  File "/builds/databiosphere/toil/src/toil/job.py", line 1743, in startToil
    return toil.restart()
  File "/builds/databiosphere/toil/src/toil/common.py", line 862, in restart
    return self._runMainLoop(rootJobDescription)
  File "/builds/databiosphere/toil/src/toil/common.py", line 1120, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/builds/databiosphere/toil/src/toil/leader.py", line 262, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/tmp/toil-test-toil.test.src.jobServiceTest.JobServiceTest-testServiceRecursive-jobstore-8zv8d88g' contains 1 failed jobs: 'JobFunctionWrappingJob' kind-JobFunctionWrappingJob/instance-acedx9bf
Log from job "'JobFunctionWrappingJob' kind-JobFunctionWrappingJob/instance-acedx9bf" follows:
=========>
	[2021-04-23T15:13:45+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
	[2021-04-23T15:13:45+0000] [MainThread] [I] [toil] Running Toil version 5.4.0a1-9fe8a768da5602f2353811cf286a59e24135b596 on host runner-hyeg35g-project-3-concurrent-04sp2d.
	[2021-04-23T15:13:45+0000] [MainThread] [D] [toil] Configuration: {'workflowID': 'f0467ef6-e095-41cb-a790-73de6e343878', 'workflowAttemptNumber': 51, 'jobStore': 'file:/tmp/toil-test-toil.test.src.jobServiceTest.JobServiceTest-testServiceRecursive-jobstore-8zv8d88g', 'logLevel': 'DEBUG', 'workDir': None, 'noStdOutErr': False, 'stats': False, 'clean': 'onSuccess', 'cleanWorkDir': 'always', 'clusterStats': None, 'restart': True, 'batchSystem': 'single_machine', 'disableAutoDeployment': False, 'environment': {}, 'statePollingWait': 1, 'maxLocalJobs': 4, 'manualMemArgs': False, 'parasolCommand': 'parasol', 'parasolMaxBatches': 1000, 'scale': 1.0, 'linkImports': True, 'moveExports': False, 'mesosMasterAddress': '10.244.2.152:5050', 'allocate_mem': True, 'kubernetesHostPath': None, 'provisioner': None, 'nodeTypes': [], 'minNodes': None, 'maxNodes': [10], 'targetTime': 1800, 'betaInertia': 0.1, 'scaleInterval': 60, 'preemptableCompensation': 0.0, 'nodeStorage': 50, 'nodeStorageOverrides': [], 'metrics': False, 'maxPreemptableServiceJobs': 9223372036854775807, 'maxServiceJobs': 9223372036854775807, 'deadlockWait': 60, 'deadlockCheckInterval': 30, 'defaultMemory': 2147483648, 'defaultCores': 1, 'defaultDisk': 2147483648, 'readGlobalFileMutableByDefault': False, 'defaultPreemptable': False, 'maxCores': 9223372036854775807, 'maxMemory': 9223372036854775807, 'maxDisk': 9223372036854775807, 'retryCount': 1, 'enableUnlimitedPreemptableRetries': False, 'doubleMem': False, 'maxJobDuration': 9223372036854775807, 'rescueJobsFrequency': 3600, 'disableCaching': False, 'disableChaining': False, 'disableJobStoreChecksumVerification': False, 'maxLogFileSize': 64000, 'writeLogs': None, 'writeLogsGzip': None, 'writeLogsFromAllJobs': False, 'sseKey': None, 'servicePollingInterval': 1.0, 'useAsync': True, 'forceDockerAppliance': False, 'runCwlInternalJobsOnWorkers': False, 'statusWait': 3600, 'disableProgress': False, 'debugWorker': False, 'disableWorkerOutputCapture': False, 'badWorker': 0.5, 'badWorkerFailInterval': 0.1, 'cwl': False}
	[2021-04-23T15:13:45+0000] [MainThread] [D] [toil.deferred] Running for file /tmp/87dc614e0bb758e29408c7feecff8ab9/deferred/func6y8nwvei
	[2021-04-23T15:13:45+0000] [MainThread] [D] [toil.worker] Parsed job description
	[2021-04-23T15:13:45+0000] [MainThread] [D] [toil.worker] Job is a checkpoint
	[2021-04-23T15:13:45+0000] [MainThread] [D] [toil.worker] The checkpoint jobs seems to have completed okay, removing any checkpoint files to delete.
	[2021-04-23T15:13:45+0000] [MainThread] [I] [toil.worker] Working on job 'JobFunctionWrappingJob' kind-JobFunctionWrappingJob/instance-acedx9bf
	[2021-04-23T15:13:45+0000] [MainThread] [D] [toil.worker] Got a command to run: _toil files/for-job/kind-JobFunctionWrappingJob/instance-acedx9bf/cleanup/file-f7a8dcb0d153450f813d1c273d33c698/stream /builds/databiosphere/toil/src toil.test.src.jobServiceTest False
	[2021-04-23T15:13:45+0000] [MainThread] [D] [toil.job] Loading user module ModuleDescriptor(dirPath='/builds/databiosphere/toil/src', name='toil.test.src.jobServiceTest', fromVirtualEnv=False).
	Traceback (most recent call last):
	  File "/builds/databiosphere/toil/src/toil/worker.py", line 367, in workerScript
	    job = Job.loadJob(jobStore, jobDesc)
	  File "/builds/databiosphere/toil/src/toil/job.py", line 2238, in loadJob
	    jobStore.readFile(pickleFile, filename)
	  File "/builds/databiosphere/toil/src/toil/jobStores/fileJobStore.py", line 440, in readFile
	    self._checkJobStoreFileID(jobStoreFileID)
	  File "/builds/databiosphere/toil/src/toil/jobStores/fileJobStore.py", line 724, in _checkJobStoreFileID
	    raise NoSuchFileException(jobStoreFileID)
	toil.jobStores.abstractJobStore.NoSuchFileException: File 'files/for-job/kind-JobFunctionWrappingJob/instance-acedx9bf/cleanup/file-f7a8dcb0d153450f813d1c273d33c698/stream' does not exist.
	[2021-04-23T15:13:45+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host runner-hyeg35g-project-3-concurrent-04sp2d
<=========

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builds/databiosphere/toil/src/toil/test/src/jobServiceTest.py", line 129, in testServiceRecursive
    self.runToil(t)
  File "/builds/databiosphere/toil/src/toil/test/src/jobServiceTest.py", line 182, in runToil
    self.fail() #Exceeded a reasonable number of restarts
  File "/usr/lib/python3.6/unittest/case.py", line 670, in fail
    raise self.failureException(msg)
AssertionError: None

┆Issue is synchronized with this Jira Task ┆Issue Number: TOIL-882

opened Apr 23, 2021 by DailyDreaming 0

releases/5.3.0(Mar 25, 2021)

releases/5.2.0(Jan 8, 2021)

Changelog

CWL

  • Update to the latest cwltool (3.0.20201121085451 -> 3.0.20201203173111). #3375
  • Add better handling for potentially mis-ordered CWL args. #3395
  • Confirm CommandInputParameter expression can receive a File object. #3350

WDL

  • Make AnalyzeWDL abstract to allow implementation for different WDL versions #3391

Misc

  • Clean up memoize.py. #3374
  • Refactor bioio library, simplify logging, and small adjustments to utils. #3351
  • Use regular division for memory calculation on LSF. #3387
  • Remove unused --nodeOptions arg. #3397
  • Get boto S3 args for minio from environment variables. (#3370)
  • Update EC2 lists. #3376
  • Add a retry to AWS destroy. #3379
  • Regular division for memory calculation in LSF. #3387
  • Add option to blank mem allocation for SLURM. #3399
  • Preserve file permissions on imported/exported files #3322

Bug Fixes

  • Preserve file permissions on imported/exported files. #3322
  • Make building a flattened list of live jobs non-recursive. #3394
  • Move AWS job store uploads/downloads to boto3 (reopened) #3400
  • Use a Docker Hub mirror when pulling Docker images. #3411
  • Corrected gridengine maxMEM check to use memorystring #3410

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @douglowe, @stevekm , @thiagogenez!

releases/5.1.0(Dec 16, 2020)

Version 5.1.0

Changelog

WDL

  • Add WDL cross function. #3360

Bug Fixes

  • Better error message for insufficient disk. #3362
  • Fix maxMEM comparison in LSF. #3369

Thank you to our contributors: @mr-c , @adamnovak , @w-gao , @jonathanxu18 !

releases/5.0.0(Dec 3, 2020)

Caching is now turned on by default.

CWL v1.1 is also now fully supported and passing all conformance tests.

The following batch system names are now deprecated (#3225) and replaced by:

  • singleMachine -> single_machine
  • gridEngine -> grid_engine
  • LSF -> lsf
  • Mesos -> mesos
  • Slurm -> slurm
  • Torque -> torque
  • HTCondor -> htcondor
  • Kubernetes -> kubernetes
  • k8s -> kubernetes

Changelog

CWL

  • Upgrade to a CWL v1.2.0 capable cwltool (3.0.20200709181526 -> 3.0.20200807132242). #3137
  • Export to a bucket if cwl's CreateFile is specified. #3124
  • CWL v1.1 Support: loadContents if True for StepValueFrom objects. #3266
  • CWL: Fix workdir permissions. #3229
  • Test CWL secondary file from s3. #3311
  • CWL v1.1 Support: Inplace update has side effect on directory content. #3280
  • More CWL v1.2 conformance tests. #3336
  • CWL: Reformat w/ Black and better support for tmpdir_prefix with suffix. #3333

WDL

  • Add ceil() and floor() WDL functions. #3168
  • Create a stdout and stderr file for each WDL task. #3181
  • Add WDL write functions and builtin tests. #3236
    • write_lines
    • write_tsv
    • write_json
    • write_map
  • Add WDL read functions and builtin tests. #3244
    • read_lines
    • read_tsv
    • read_json
    • read_map
    • read_int
    • read_string
    • read_float
    • read_boolean
  • Add transpose() WDL function. #3273
  • Add length() WDL function. #3307
  • Add sub() WDL function. #3252
  • Add range() WDL function. #3277
  • Add size() WDL function. #3255
  • Add zip() WDL function. #3355
  • Implement WDL Pair type. #3304
  • Refactor WDL types #3335

Kubernetes

  • Adding Labels to Pods/Jobs. #3233
  • Make kubernetes & botocore packages fully optional. #3259 #3261
  • KubeWatch optimization. #3227
  • Catch other kubernetes imports. #3261
  • Automatically infer singularity for kubernetes. #3279

Misc

  • Allow Caching By Default. #3111
  • Update dependencies and loosen restrictions slightly. #3139
  • Ensure that falsey (but non-null) values are considered. #3143
  • Remove pathlib. #3169
  • Move AWS job store uploads/downloads to boto3. #3153
  • Update prom/node-exporter to use quay.io. #3186
  • Add support for TOIL_CUSTOM_INIT_COMMAND. #3183
  • Update docker module version to 4.3.1. #3222
  • Add 'stream' and 'demux' options to apiDockerCall. #3224
  • Use boto3 to create our leader node. #3145
  • Prevent job kind filenames from getting too long. #3230
  • Add a retry decorator. #3144
  • Remove mesos offer messages. #3308
  • Job Concept Unification. #3250
  • Add reporting for accessed files of failed jobs. #3309
  • Add coverage and refactor. #3314
  • Expose --statePollingWait param through toil. #3321
  • Record the maximum memory used by LSF jobs. #3327
  • Remove custom math functions and replace with built ins. #3338
  • Define a changelog process. #3316
  • SLURM: getting job details from scontrol show jobs was non-functional and has now been fixed. This behavior in the case of sacct not being configured now works. #3346
  • LSF: handle unicode configuration files #3354
  • Replace abssympath w/ builtin. #3356

Bug Fixes

  • Turn off watch code in Kubernetes. #3175
  • Check against joining the current thread during CachingFileStore destructor. #3187
  • Slurm.py error, undefined stdout - stdout should be stderr in code. #3194
  • Fix restart flag error (issue #3094). #3142
  • Batch System generates a key error instead of a usage message. #3225
  • Remove Python 2 imports in WDL. #3228
  • Move checkForDeadlocks() in the leader. #3234
  • Fix boto3 migration bugs. #3243
  • Fix None check when parsing WDL JSON file. #3272
  • Fix broken stdout/stderr in torque job. #3306
  • Add doubleMem for LSF jobs that die due to imposed memory limit. #3313
  • Add 500 status to the list of retriable SDB BotoServerErrors. #3329
  • Add a retry logic for BucketNotEmpty error. #3341
  • Fix None check when parsing WDL JSON file. #3272
  • Warn when trying to clean non-existent job store #3288
  • Only rm a statefile that exists. #3348
  • CWL: Remove unprocessed secondaryFiles from both Dict and List types.

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @diekhans, @w-gao, @jonathanxu18, @jeffrey856, @davidlougheed, @altairwei, @mberacochea, @drkennetz, @ionox0!

releases/4.2.0(Aug 12, 2020)

  • Don't log in the inner scheduling loop. #3065
  • Handle anticipated 404 errors without retrying. #3067
  • Test and reimplement AMI finding via Flatcar JSON feed. #3061
  • Bump Enlighten version to partly fix #3069. #3070
  • Get wait duration is far too short of a rest for lsf. #3076
  • Unify logging format. #3073
  • Support CWL 1.1 Listing. #3058
  • Enable CWL v1.2.0-dev3 (3.0.20200324120055 -> 3.0.20200530110633). #3092
  • Enable CWL v1.2.0-dev4 (3.0.20200530110633 -> 3.0.20200709181526). #3105
  • Add --nodeStorageOverrides option. #3096
  • More robust CWL filepaths and secondary files. #3114
  • Replace custom mkdir_p with the built in. #3123
  • Move conditional execution test to before fill_in_defaults. #3117

Bug Fixes

  • Stop CWL from trying to copy downloaded directories from somewhere local. #3053
  • Exit the worker with 1 when jobs fail so the batch system can see. #3052
  • Eliminate recursion in root finding to fix #3080. #3081
  • Add a Cactus Kubernetes test to the integration tests on Gitlab. #3077
  • Chaining doesn't make sense for CWL. #3091
  • Fix failed job accounting. #3103
  • Improve lsf batchsystem stability. #3101
  • Deduplicate updated jobs. #3108
  • Redundant virtualenv check. #3135
  • Change Slurm command output processing. #3133
  • Fix for log file being linked more than once. #3130
  • Wrap Kubernetes watches to fix #3125. #3126
  • Create dirs that do not exist when bind-mounting to docker. #3120
  • Fix database state when file upload fails. #3122

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @diekhans, @nikhil, @drkennetz, @jonathanxu18, @tobiaszjarosiewicz, @ionox0!

releases/4.1.0(May 7, 2020)

  • Attempt to make batch system more robust and debuggable. #2959
  • Kubernetes Shared Caching. #3012
  • Test python3.8 and build a python3.8 appliance. #3028
  • Use the original pymesos. #3036
  • Detect more kinds of virtual environment and normalize prefixes. #3035
  • Do More Hashing. Can't have enough. #3025
  • Replace reissued jobs message with better progress indicators. #3044
  • Detect insufficient resources and deadlocks more usefully. #3043

Bug Fixes

  • Don't list specific CWL features, try them all (and fix some CWL running bugs). #3015
  • Always honor TOIL_GRIDENGINE_PE and never assume a site-dependent default. #3041

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @diekhans!

Information
Category: Python / Miscellaneous
Watchers: 53
Star: 740
Fork: 208
Last update: May 5, 2021

Contents
    Resource links
    https://github.com/DataBiosphere/toil
    Share this repo
    Most popular
    github A collection of useful .gitignore templates
    tonsky Free monospaced font with programming ligatures
    kelseyhightower The best way to write secure and reliable applications. Write nothing; deploy nowhere.
    awesomedata A topic-centric list of HQ open datasets.
    mtdvio A collection of (mostly) technical things every software developer should know
    minimaxir The Big List of Naughty Strings is a list of strings which have a high probability of causing issues when used as user-input data.
    florinpop17 A Collection of application ideas which can be used to improve your coding skills.
    prettier Prettier is an opinionated code formatter.
    DopplerHQ :octocat: A curated awesome list of lists of interview questions. Feel free to contribute! :mortar_board:
    python The Python programming language

    Related Repos


    Miscellaneous
    244
    DonaldTrumpStockMonitor
    semaaJ This project originally started when I noticed that if Trump mentioned a company within his tweets, their shares would either go up in price if the tweet was positive or drop if the tweet was negative.
     
    Miscellaneous
    383
    A curated list of the best in class SaaS services for developers and business owners.
    GetStream A curated list of the best in class SaaS services for developers and business owners.
     
    Miscellaneous
    2.7k
    A curated list of awesome reversing resources
    tylerha97 awesome-reversing A curated list of awesome reversing resources Awesome Reversing Books Courses Channels Practice Hex Editors Binary Format Disassemblers Binary Analysis Bytecode Analys
     
    Miscellaneous
    409
    Text UI colors for Python.
    MasterOdin Crayons: Text UI colors for Python. This module is really simple, it gives you colored strings for terminal usage. Included colors are red, green, yellow, blue, black, magenta, cyan, white, and normal ( as well as c
     
    Miscellaneous
    123
    A tool to run periodic speedtests and save them in InfluxDB for graphing in Grafana
    barrycarey Speedtest.net Collector For InfluxDB and Grafana This tool is a wrapper for speedtest-cli which allows you to run periodic speedtets and save the results to Influxdb Configuration within config.ini GENERAL
     
    Miscellaneous
    43
    A small forth-like language that targets the CPython VM.
    llllllllll phorth phorth is a bootstrapped forth-like language where instead of writing our primitive words in some assembler, we have chosen to write them in a mix of CPython bytecode and C++*. By using a superset of CPython bytecode, we m
     
    Miscellaneous
    236
    A self-contained read-only CD/USB stick with everything you need to perform highly secure ...
    bitkey BitKey is a swiss army knife of handy Bitcoin tools built on top of Debian, which we created to scratch our own itch. Backstory We're avid Bitcoin fans but after going to our first local Bitcoin meetup we discovered the elepha
     
    Miscellaneous
    725
    A library adding some anti-Pythonic syntatic sugar to Python
    czheo syntax_sugar This lib adds some anti-Pythonic "syntactic sugar" to Python. NOTE: This is merely an experimental prototype to show some potential of operator overloading in Python. Only tested under Python 3.6.0. Anything may e