Highlighted Features Added

Added new unified plugin system (#5302)
Toil now supports URL plugins. To provide a plugin, add toil_url_access_ at beginning of python package name. To register a new URL scheme use register_plugin("url_access", scheme, implementation_type_factory). (#5302)
Updated real world example in quick start (#3534)
Toil Docker image now supports Mesos on Pythons newer than 3.10 (#5344)

Breaking Changes

Refactor URL access functions to use new URLAccess class instead of AbstractJobStore (#5301)
Toil Docker container no longer ships s3am by default (#5321)
Deleted deprecated addBatchSystemFactory, BATCH_SYSTEM_FACTORY_REGISTRY, and BATCH_SYSTEMS. Use add_batch_system_factory instead. (#5323)
--importWorkersThreshold has been renamed to --importWorkersBatchsize (#5325)

CWL

Fix a bug in Toil server mode CWL YAML parsing (#5252)
Fix missing CWL badges (#5304)
CWL worker import works again (#5256)
CWL worker import test (and other non-conformance CWL tests) now run on CI again (#5256)
Fixed a regression with tmpdir_prefix and nonexistent diectories (#5310)
Mesos CI tests no longer reference CWL conformance test class by old name (#5329)
Remove redundant listings when loading inputs to a job (#5099)
Added --cwl-min-ram to CWL. This specifies a minimum memory allocation for all jobs. This is different than --defaultMemory, which only applies to jobs without explicitly set memory requirements. (#2862)

WDL

Toil now deletes files that go out of scope at the ends of workflows and tasks. (#5222)
toil-wdl-runner can again successfully finish a workflow and emit outputs when using --restart (#5247)
wdl tests: use pytest fixtures to support parallel execution (#5235)
Added lint checking to toil-wdl-runner (#4630)
- Use --strict=True to error on lint warnings
Toil can now handle coercing input Files to Strings and back in WDL workflows. (#5265)
Allow relaxing quantifier checks with --quantCheck (#5297)
toil-wdl-runner will bump container memory if below 4MiB (#5325)

Kubernetes

Added command line options --kubernetesPodSecurityContext and --kubernetesSecurityContext to change the security context of pods and containers (#5179)

Dependencies

Update to connexion 3 (#4927)

Misc

Toil metrics published to Dockstore now are marked as coming from TOIL and not OTHER. (#5237)
Toil metrics published to Dockstore are now "authenticated" with a public, non-secret token. (#5239)
Dockstore publication tkinter dialog now opens over other windows on Mac. (#5239)
Publishing metrics for the current workflow will no longer send an empty list of tasks also. (#5239)
needsawss3: also skip tests if there is a proxy error (#5250)
Speed up CWL badge generation on CI (#5246)
gitlab-ci: add collapsable sections (with recorded durations) (#5235)
gitlab-ci: increase test threads to 4; standalone tests only need one test thread; use worksteal more often; upgrade cwltest, run conformance tests in parallel; enable caching of pip & mypy (#5235)
Toil can now build correctly for pip installs from Git again.
Toil now tests linting on the oldest supported Python for every PR (#5276)
Added a way to turn Toil history database on and off with TOIL_HISTORY. (#5254)
Added TOIL_JOB_HISTORY to force-enable job history tracking even though it's not ready for production use. (#5258)
Toil tests no longer record history to the Toil history database. (#5254)
Suppress spurious nvidia warnings (#5306)
Use different Google Cloud example file since the Broad data is now requester-pays (#5307)
Improve Google cloud download speed by polling for metadata once (#5311)
Toil Docker build is now willing to hide system Python packages (#5320)
Remove separate plugin test job since quicktestoffline runs them (#5334)
Toil server mode CI tests now run on PRs. Toil server mode integration tests now run on PRs that touch the server code. Toil server mode CI tests are now updated for Connexion 3. (#5332)
Toil web URL tests now run against local servers instead of httpstat.us (#5347)
Prepare for https://github.com/adamchainz/ec2-metadata/pull/562 and in the meantime hide SSH key name in instance tags. (#5349)
Deprecated the Toil built-in autoscaler for Mesos (#4623)
Toil now supports not having lsof installed (#5236)
Make cleaning more idempotent (#5352)

Thank you to our contributors: * @DailyDreaming * @haileyhemmingsen * @stxue1
* @mr-c * @adamnovak

- Python
Published by adamnovak 7 months ago

toil - 8.2.0

- Python
Published by DailyDreaming 10 months ago

toil - 8.1.0b1

Note: this is a beta release with new functionality immediately needed by Cactus.

Highlighted Features Added

Toil can now publish workflow execution metrics to Dockstore for workflows where it knows the TRS ID (i.e. workflows run by TRS ID or Dockstore URL) (#5159).
- Toil now saves workflow execution history in ~/.toil/history.sqlite
- Toil will prompt the user to decide whether they want to publish workflow metrics, and, if it thinks it can get ahold of the user, will not proceed until they decide or a timeout elapses.

CWL

Add caching to toil-cwl-runner. Use --cachedir [dir] to enable, and avoid rerunning previously cached jobs. (#4298)
Add CWL badges to documentation (#5183)
Improve error message when a file is not found in CWL (#5174)

Misc

Toil will no longer select GPU or high-priority Slurm partitions when it does not need to. (#5223)
Allow handling unlimited number of jobs on Slurm (#5064)
Add hack to automatically un-stick jobs that can be un-stuck by lsof. (#5214)
Re-added support for --time in TOIL_SLURM_ARGS (#5230)
Added --slurmPartition and --slurmGPUPartition options for manual Slurm partition selection. (#5231)

Thank you to our contributors: @stxue1 @mr-c @adamnovak

- Python
Published by adamnovak 12 months ago

Highlighted Features Added

toil debug-job now has --retrieveTaskDirectory <dir> which will set up a job's downloaded files under <dir> and try to stop the job after doing the downloads. Jobs can call self.files_downloaded_hook() to provide a stopping point for this mode. (#4815)
toil debug-job can now reconstruct the inside-the-container environment for CWL and WDL tasks. (#4815)
Added support for caching on Slurm and other HPC schedulers (#4775)
Replace all instances of boto2 with boto3 for all Toil AWS code (#4718)
Add support for Python 3.12 (#4718)
Add support for Python 3.13 (#5145)
Ceph input/output errors from file locking functions are now tolerated. (#4874)
Toil now uses flock to enable directory locks to work properly (#4924)
Added support to get Slurm partitions and automatically send jobs to GPUs on Slurm (#4833) (supports both CWL and WDL)
New --symlinkJobStoreReads=False option lets you force local-node copies (possibly in the cache) even when reading directly from a FileJobStore is possible, potentially reducing shared filesystem IO. (#4673)
Toil now supports reading and writing MiniWDL's call cache. (#4797)
Toil now supports running CWL and WDL workflows from Dockstore, by using either a Dockstore page URL or TRS ID as the URL/filename of the workflow to run. Since these often contain ? or #, remember to quote them on the command line! (#5049)
Add support for parallel file imports (#5114)
- New argument --importWorkersThreshold. This specifies the threshold where files will begin to be imported on individual jobs. Small files will be batched into the same import job up to this threshold.
- --importWorkersDisk defaults to 1 MiB. Should be increased when download streaming is not possible on a worker.

Breaking Changes

CWL

Prevent simultaneous Singularity container pulls in toil-cwl-runner (#4990)
Added support to import files on workers for toil-cwl-runner (#5025)
- --runImportsOnWorkers to enable importing files on workers
- --importWorkersDisk to control how much disk space the import worker will use
Don't error when passing through input as the output (#5138)
CWL jobs with dynamic requirements now have input type checking properly protected by their conditionals. (#4930)
Fixed a LoadListing bug with CWL workflows (#5149)
Fix CWL Workflow Slurm memory test (#5151)
workDir and jobStore defaults to tmp-outdir-prefix (#5154)
CWL container prepull: no reason to if extensions are enabled, they are now supported by cwl-utils 0.36+ (#5188)
CWL container prepull: skip if --no-containers is specified (#5188)

WDL

Update WDL conformance tests on CI (#4875)
Added support to run task only WDL files (#4960)
Added support for the gpu field in WDL (#4949)
Support passing inputs into toil-wdl-runner for task only WDLs (#4977)
toil-wdl-runner will now carry through task exit codes (#4978)
toil-wdl-runner will respect explicit null values for optional inputs (#4981)
toil-wdl-runner will not immediately error on nonexistent coerced files until outputted (#4994)
- File? type for string to file coercion is now supported (will be nullified)
WDL output files will now live in directories named after their tasks instead of UUID directories (#5008)
Fixed a bug with conditional statements inside a WDL scatter (#5055)
toil-wdl-runner now correctly finds and returns outputs from tasks in scatters and conditionals when a WDL workflow lacks an output section. (#5094)
toil-wdl-runner has a new --allCallOutputs option to allow including all calls' outputs in a workflow's output. (#5093)
toil-wdl-runner can now detect and try not to delete the outputs of a workflow that is meant to use the Cromwell Output Organizer (croo). Note that croo still can't actually work on the output of toil-wdl-runner. (#5093)
--allCallOutputs no longer discards WDL workflow outputs section outputs. (#5106)
File virtualization in toil-wdl-runner now only happens at task boundaries (#5028)
- File to String coercion should be supported
Added support to import files on workers for toil-wdl-runner (#5103)
Support WDL 1.1 disk specification as per spec (#5001)
Fixed a bug with WDL file imports (#5121)

Kubernetes

Dependencies

Toil can now use connexion 4 (#5196)
Toil now uses htcondor 23.6 or 24, which are still on PyPI

Misc

Makefile: use isolated builds, add dist target (sdist+wheel) and deprecate the sdist target. (#4820) (#4826)
Toil will now wait --jobStoreTimeout seconds (default: 30) to see an update to/removal of a job that was run, and will not let the job succeed unless it is seen to make progress. (#3814)
Toil job descriptions no longer have a command field, and we track the link to the job body and the command to invoke the Toil worker separately. (#4811)
Several typos in the docs were fixed (#4889)
Add a test to ensure batchsystem plugins are installable (#4879)
Fix Toil utils to work without the AWS extra (#4953)
Print commit hash with toil --version when installed from source. Before: 7.1.0a1. After: 7.1.0a1-ccf57e6071e32675daabdcbacb91988e871745a9 (#4954)
Fixed a broken URL and an omitted variable in CI tests (#4974)
Generate default config correctly (#5014)
Use the latest setuptools when running cactus. (#5017)
Toil will refuse to proceed if it detects that its coordination directory or a Singularity cache directory it needs to lock is on Ceph, to prevent hanging the Ceph MDS (#4972)
Fix a NotImplementedError in the Grid Engine batchsystem (#5061)
- Added basic Grid Engine CI tests
Update Cactus on CI to 2.9.0 (#5062)
Separate out create/delete iam role functions into lib.
Remove deprecated pipes module (#5122)
New --slurmTime/TOIL_SLURM_TIME setting to set the time limit on Slurm jobs in a way Toil itself understands. (#5010)
New --slurmPE argument to allow setting a parallel-job Slurm partition without using TOIL_SLURM_PE (#5010)
New --slurmArgs argument to allow specifying extra Slurm submission arguments without using TOIL_SLURM_ARGS (#5010)
For non-GPU jobs on Slurm, Toil will submit the job to a partition with a time limit long enough to accommodate the configured runtime (from --slurmTime). (For GPU jobs, the lowest-priority GPU partition is still always used.) (#5010)
Toil now has a --slurmDefaultAllMem option to run jobs lacking their own memory requirements with Slurm's --mem=0, so they get a whole node's memory. (#4971)
toil-cwl-runner now has --no-cwl-default-ram (and --cwl-default-ram) to control whether the CWL spec's default ramMin is applied, or Toil's own default memory logic is used. (#4971)
The --dont_allocate_mem and --allocate_mem options have been deprecated and replaced with --slurmAllocateMem, which can be True or False. (#4971)
Added WDL unit tests to CI (#5110)
Mesos build updated. (#5049)
CWL and WDL argument parsing revised for Python 3.12. (#5049)
Organize stats and logging files into stats/inbox and stats/archive and avoid a circular rename. (#1727)
Added proper FTP support for jobstores (#5134)
URL existence and size gets/checks are now done with HEAD requests (#5134)
Dependabot configuration should now pass schema validation and is itself under CI (#5175)
Toil now tests a version of Cactus that ought to run on Python 3.13. (#5184)
WDL conformance tests on Kubernetes may now run for 30 minutes. (#5185)
When importing files on workers, fall back to importing on the leader when file sizes are not obtainable (#5135)

Thank you to our contributors: @stxue1, @DailyDreaming, @adamnovak, @mr-c, @gmloose, @davidjsherman!

- Python
Published by stxue1 about 1 year ago

toil - 7.0.0

- Python
Published by DailyDreaming almost 2 years ago

Highlighted Features Added

WDL and CWL task standard output and standard error logs that are not captured by the workflow will now be logged at INFO level and stored in the --writeLogs/--writeLogsGzip directory. (#4657)
- Use a default log limit of 100MiB (#4788)

Breaking Changes

Stats and logging system again uses job display name (#4755)
--disableProgress is once again a flag that doesn't take an argument (#4758)

CWL

Don't clear out user-provided values for the --default-container option (#4730)

WDL

WDL job names now include numbers for scatters (#4755)
Multi-line WDL placeholder substitutions no longer interfere with de-indenting WDL command blocks (https://github.com/chanzuckerberg/miniwdl/issues/665)
Standard error for failed tasks is now always logged to the worker log somewhere (#4781)

Kubernetes

Dependencies

Deps: removed the ruaml.yaml.string plugin dependency for a simpler solution (#4760)

Misc

Toil will no longer warn about a missing XDGRUNTIMEDIR (#4769)
Read the Docs and CI docs builds should have Graphviz installed (pending CI image rebuild) (#4734)
Add more Python3.12 compatibility by replacing the one function from distutils that we use, strtobool(). (#4765)
Set default cache folders to be accessible between toil-wdl-runner workflows (Same as MiniWDL/Singularity defaults) (#4761)
Set toil-wdl-runner cache folders on Toil managed clusters to be at /var/lib/toil (#4761)
Fall back to assuming machine has 1 core when CPU count is unavailable. (#4545)
FileJobStore now supports filenames that get modified when percent-encoded (#4779)

Thank you to our contributors:

@DailyDreaming @mr-c @stxue1 @adamnovak @app/dependabot

Full Changelog: https://github.com/DataBiosphere/toil/compare/releases/6.0.0...releases/6.1.0

- Python
Published by adamnovak almost 2 years ago

toil - 6.0.0

add pure Python fallback for getDirSizeRecursively() (#4753)
Update flake8-bugbear requirement from <24,>=20.11.1 to >=20.11.1,<25 (#4752)
Bump cwltool from 3.1.20231207110929 to 3.1.20240112164112 (#4751)
Don't mark inputs (or outputs) executable for no reason (#4728)
Log more usefully for CWL workflows (#4736)
Update EC2 instances and EC2 update script. (#4745)
Fix scheduled CI tests (#4742)
Improve WDL documentation (#4732)
remove extraneous dependency on old 'mock' (#4739)
Fix --printJobInfo (#4709)
Update flake8 requirement from <7,>=3.8.4 to >=3.8.4,<8 (#4738)
Tolerate a failed AMI polling attempt (#4727)
Bump mypy from 1.7.1 to 1.8.0 (#4731)
Point CI at the new public URLs for stuff we host
Make cwl mutually exclusive groups exist only when cwl is not suppressed (#4725)
Add init.py to options folder (#4723)
Strip leading whitespace from WDL commands (#4720)
Implement a better config file system for CWL/WDL options (#4666)
Update docker requirement from <7,>=3.7.2 to >=3.7.2,<8 (#4713)
upgrade to cwltool 3.1.20231207110929 (#4707)
Allow working with remote files in CWL and WDL workflows (#4690)
Remove the WDL compiler. (#4679)
Support CWL 1.2.1 (#4682)
more resiliancy (#4395)
remove useage of the deprecated pkg_resources (#4701)
Make sure cwltool always knows we have an outdir to fix #4698 (#4699)
AWS jobStoreTest: re-use deletes3bucket from toil.lib.aws (#4700)
Bump mypy from 1.7.0 to 1.7.1 (#4697)
Only count output file usage when using the file store (#4692)
Reenable Cactus on Kubernetes CI test (#4604)
Remove the parasol batch system. (#4678)
Bump mypy from 1.6.1 to 1.7.0 (#4684)
Bump cwltool from 3.1.20231020140205 to 3.1.20231114134824 (#4685)
CWL Pipefish compatibility (#4636)
skip unwanted networkx version (#4450)
Move TES batch system to a plugin (#4650)
Bump miniwdl from 1.10.0 to 1.11.1 (#4669)
Turn batch system tests back on (#4649)
Move around reqs and move aws dev libraries to aws (#4664)
Bump mypy from 1.5.1 to 1.6.1 (#4660)
Fix #3867 and try to explain but not crash when bad things happen to our mutex file (#4656)
Fix CI Appliance Builds (#4655)
Update ruamel-yaml requirement from <0.17.33,>=0.15 to >=0.15,<0.18.4 (#4659)
Fix configargparse in CWL (#4618)
Support Python3.11 and drop Python 3.7 (#4646)
Report errors in WDL using MiniWDL's error location printer (#4637)
cwl: use the latest commit from the proposed CWL v1.2.1 branch (#4565)
Upgrade cwltool to avoid broken galaxy-tool-util release. (#4639)
Make sure the --batchLogsDir exists if it is set (#4635)
Ensure sibling files in toil-wdl-runner (#4610)
Make WDLOutputJob collect all task outputs (#4602)
take any nvidia-smi exception as not having gpu (#4611)
Add config file support (#4569)
Separate out integration tests to run on a schedule (#4612)
Add String to File functionality into toil-wdl-runner (#4589)
Avoid concurrent modification in cluster scaler tests (#4600)
Update docs to hide Mesos (#4413)
Remove old buckets from AWS (#4588)
Tests: only request a single core (#4572)
Reduce the number of assert statements (#4590)
Fix pickling error when jobstate file doesnt exist and fix threading error when lock file exists then disappears (#4575)
Allow the WDL output section to reference itself (#4592)
Disable Kubernetes on CI temporarily
Bump cwltool from 3.1.20230719185429 to 3.1.20230906142556
Bump actions/checkout from 3 to 4 (#4593)
Run WDL output through Toil export system to support URIs (#4579)
Add version bound for typing-extensions (#4586)
Lower redirect log level to fix #4526 (#4578)
Fix mypy from being broken by new boto types (#4577)
Fix CI on local Gitlab runners (#4571)
Bump mypy from 1.5.0 to 1.5.1 (#4568)
Banish ghost jobs (#4563)
More WDL and Slurm documentation (#4558)
Bump mypy from 1.4.1 to 1.5.0 (#4559)
Stop deleting chained-to jobs which fail as orphaned jobs (#4557)
Support per-task preemptibility in CWL (#4551)
Simplify WDL Toil job graphs (#4524)
Bump flask-cors from 3.0.10 to 4.0.0 (#4553)
Bump gunicorn from 20.1.0 to 21.2.0 (#4556)
Update ruamel-yaml requirement from <=0.17.21,>=0.15 to >=0.15,<=0.17.32 (#4555)
Bump cwltool from 3.1.20230601100705 to 3.1.20230719185429 (#4552)
Update setuptools requirement from <68,>=65.5.1 to >=65.5.1,<69 (#4547)
Bump mypy from 1.2.0 to 1.4.1 (#4548)
Update docutils requirement from <0.18,>=0.16 to >=0.16,<0.21 (#4549)

- Python
Published by DailyDreaming about 2 years ago

WDL

Virtualize filenames as in-container paths from point of view of WDL command (#4527)
Add WDL conformance tests to CI (#4530)
Use less memory in the Giraffe WDL test (#4541)

Version Upgrades

Upgrade to cwltool 3.1.20230601100705 (#4500)
Update mock requirement from <5,>=4.0.3 to >=4.0.3,<6 (#4366)

Misc

Anonymous access to Google Storage (#4518)
Reorder config so that default settings are applied first (#4528)
Add a way to forward accelerators to Docker containers (#4492)

Bug Fixes * Fix test failures without docker installed (#4544) * Prevent certain tests from being run twice in CI (#4529) * Drop external Docker builder (#4523) * Fix CI lint test (#4533) * Grab AWS group policies on top of user (#4505) * Grab accelerator set off the end of the list instead of by index (#4506) * Fix RtD build (#4491) * Include tests (#4499)

Thank you to our contributors: @stxue1 , @DailyDreaming , @mr-c , @adamnovak , and @tjni !

- Python
Published by DailyDreaming over 2 years ago

toil - 5.11.0

Breaking Changes

Imported files will be symlinked by default, unless the user sets --noLinkImports or the workflow imports with symlink=False. (#3949)

WDL

Toil will now stop if it encounters an error polling a possible import URL for a WDL workflow input file. (#4479)
WDL workflows will be protected against imported files with no basenames. (#4477)

Misc

Toil batch system ID numbers for issued jobs now start at 1. (#4482)
Attempts to import files from URLs when the implementing job store is missing an extra are now better reported. (#4479)
Include tests in the source distribution that gets published to PyPI (#4499)

Bug Fixes * Toil should no longer crash when a delete wins a race against a load in FileJobStore (#4484) * Prevent local root jobs (such as WDLRootJob) from being run twice. (#4482) * Slurm and other grid batch system jobs will now have more informative names (#4472) * WDL workflows can no longer import "" as a File. (#4477)

Thank you to our contributors: @stxue1, @DailyDreaming, @mr-c, @adamnovak

- Python
Published by adamnovak over 2 years ago

toil - 5.10.0

Changelog

Highlighted Features Added

Add a --caching option which explicitly states whether to use caching with a workflow. Uses a default value depending on whether or not we are using the file job store if not specified. (#4218)
New prototype WDL runner python -m toil.wdl.wdltoil using MiniWDL (#3468)
MiniWDL-based WDL implementation can now run the vg Giraffe WDL workflow ( #4353)
Toil now tests against our own tiny set of WDL conformance tests (#4351)
Toil can run the HPRC assembly WDL workflows (#4435)
Toil can now use Mesos roles (#4455)

Breaking Changes

Replace "preemptable" with "preemptible", add example of using --defaultPreemptible flag to Preemptibility documentation (#1951)

CWL

CWL: run all ExpressionTools on the Leader node, instead of submitting separate jobs (#4157)

Kubernetes

Kubernetes batch system: Delete jobs individually when batch delete fails (#3403)
Documentation for running a Toil leader for a Kubernetes workflow outside Kubernetes now covers examples and common problems for running CWL workflows (document toil-cwl-runner + "Running the Leader Outside Kubernetes" #3422)
Kubernetes batch system: support --maxCores, --maxDisk, and --maxMemory (#2864)
Add tutorial for Kubernetes launch cluster (#3743)

Dependencies

Require htcondor 10 exactly (#4315)
Toil jobs now have a local parameter which determines if they should run on the leader. (#4388)

Misc

The offline tests can now be run in parallel (#3493)
Code updated to be more idiomatic for Python3.7 (#4295)
Support for a --network for toil launch-cluster for Google cloud (#4196)
Support for a --use_private_ip for toil launch-cluster to dial nodes by private IP instead of public IP (#4196)
GPU scheduling should now be supported on Slurm (#4308)
Toil now supports a --batchLogsDir option and TOIL_BATCH_LOGS_DIR environment variable, to provide a directory other than the work dir where Toil will instruct HPC batch systems to save their captured job logs.
htcondor batch system should now work again, and will retry connections
Updated the --coalesceStatusCalls help documentation to reflect the current state of https://github.com/DataBiosphere/toil/issues/4431 (#4437)
Toil no longer trusts XDGRUNTIMEDIR under Slurm (fixes some of the issues behind #4395 when Slurm is configured not to follow the XDG spec) (#4435)
Toil now puts it lock files for Singularity cache directories for WDL in those directories (#4435)
Toil's WDL interpreter can now use local-to-the-leader jobs for evaluating WDL code that doesn't need appreciable resources (#4388)
Toil now tolerates more possible exceptions related to the panasas network file system (#4440)
Type hinting to functions in resource.py (#938)
Added return type to inVirtualEnv() in __init__.py (#938)
Added None checks to some function bodies (#938)

Bug Fixes * Stop crashing when predefined batch job exit reasons are used and need to go into the message bus log file (#4321) * Added import subprocess to restore the behavior of #588. (#4429) * Toil will no longer use the stored message bus path from an old execution of a workflow when deciding where to save the message bus log when restarting a workflow (#4438) * Fix --custom-net mutual exclusivity bug. (#4458)

Thank you to our contributors: @stxue1 , @DailyDreaming , @mr-c , @adamnovak , @jfennick , @misterbrandonwalker , @w-gao , @stephanaime , @glennhickey , @Hexotical , @manabuishii @gmloose , @boukn , and @thiagogenez !

- Python
Published by DailyDreaming almost 3 years ago

toil - 5.9.2

Changelog

Bug Fixes * Change build tag import (#4329)

Thank you to our contributors: @adamnovak , @Hexotical !

- Python
Published by DailyDreaming about 3 years ago

toil - 5.9.0

Changelog

Bug Fixes * Fix --provisioner and --metrics together (#4328) * Ignore incorrect type hint from boto3, remove json.loads (#4330) * Warn about missing --bypass-file-store with in-place update (#4337) * Replace prepareHTSubmission with prepareSubmission in HTCondor (#4319) * Merge "Google fixes" (#4293) * Support (only) current htcondor (#4320) * Delete k8s jobs individually when batch delete fails (#4306)

Misc

Update aws spot documentation (#4310)
Enable parallel testing (#3493)
Add documentation for running CWL workflows on non-Toil-managed Kubernetes clusters (#4332)
Export all slurm args by default (#4237)
Allow for subclasses of base types in messages (#4322)
Non cache default (#4299)

Dependencies

Bump mypy from 0.982 to 0.991 (#4345)
Bump schema-salad>=8.4.20230128170514,<9 to schema-salad>=8.3.20220913105718,<8.4 (#4342) (#4341)
Bump cwltool from 3.1.20221008225030 to 3.1.20221201130942 (#4338)
Bump pyupgrade to 3.7 (#4295)

Thank you to our contributors: @adamnovak , @Hexotical , @w-gao, @mr-c , @gmloose , @boukn , and @thiagogenez !

- Python
Published by DailyDreaming about 3 years ago

toil - 5.8.0

Changelog

Highlighted Features Added

Toil server now exposes workflow tasks via WES (#4046).
Toil server now has a --wes_dialect agc option that will hide any tasks that don't have Amazon Batch job IDs, and put the IDs in the task names for those that do (#4047).
Toil jobs now accept an accelerators requirement, like accelerators=1 or accelerators={'kind': 'gpu', 'brand': 'nvidia', 'count': 2} (#4163)
Include total requested cores for each job type in toil stats (#4173)
Toil jobs now expose job.accelerators to workflow
Add prefix suffix params to AbstractFileStore.getLocalTempFile and AbstractFileStore.getLocalTempFileName (#4273)
CWL: --no-compute-checksum, --strict-cpu-limit, --disable-validate, and --fast-parser are now available

Breaking Changes

Toil's built-in autoscaler now guesses that some memory and disk space on nodes will not actually be available for jobs; pass --assumeZeroOverhead to revert to the old behavior (#2103)

CWL

CWL job unit and display names have been changed to make more sense as task names, and management of them has been unified into a CWLNamedJob. (#4046/#4047)
CWL CUDARequirement is parsed by cwltool and turned into a requirement for the minimum requested number of nvidia GPU accelerators (#3982)
- fix false warning when outputSource contains only one None value (#4300)

Kubernetes

KubernetesBatchSystem can add nvidia.com/gpu and amd.com/gpu resource requests for jobs that request those accelerators (#4163)
KubernetesBatchSystem can request GPUs by model key, if nodes are labeled appropriately (#4163)

Dependencies

Misc

Toil WES server now accepts requests that leave out workflow_params. (#4037)
The MessageBus has been expanded to use pypubsub, and now has MessageInbox and MessageOutbox objects to represent connections to it. (#4046/#4047)
ToilMetrics now rides on the MessageBus rails. (#4046/#4047)
Toil workflows now have a --writeMessages option, which takes a file to which a line-oriented stream of MessageBus messages will be written. Reading this file will allow you to recover the current state of the workflow. (#4046/#4047)
Add code for warning check to be used when launching cluster with AWS. (#3514)
Use a CI prebake image for gitlab testing. (#4185)
Toil clusters now have /var/tmp as the default temporary directory, since they often make large temporary files (#4148)
Adds basic testing for slurm using a slurm docker cluster by running sample workflows. (#3856)
Add message bus documentation (#4239)
SingleMachineBatchSystem can schedule nvidia GPU accelerators, limiting the concurrent jobs to no more than there are accelerators to support, and setting CUDA_VISIBLE_DEVICES in the tasks' environments to tell them which nvidia GPU(s) to use. (#4163)
AWSBatchBatchSystem can use AWS Batch's GPU resource to provide nvidia GPU accelerators (#4163)
Toil jobs no longer need to re-run after their child/followOn/service jobs in order to delete themselves. (#3188)
Message bus is now thread safe (#4276)
Docker build has been updated with new Aventer Mesos deb URL (fixes #4290)
docker binary in the container has been updated to that included in the Ubuntu repos (fixes #4282)
Singularity in the appliance has been updated to 3.10 which is >=3.9, for cgroups v2 support.
Base Ubuntu container image for the appliance has been updated to 22.04, which has a new enough libc for Debian's Singularity 3.10 debs.
Safer type usage checking for systems without boto3 installed
Tests are now more runnable post-installation. Temporary paths are not selected based upon the location of the tests themselves. (#4287)

Bug Fixes * Only use /var/run/user if XDG tells us we have it in our session. Otherwise we will try other places, including /run/lock/toil. (#4170) * toil destroy-cluster: terminate stopped instances when destroying the cluster (#4271) * fileJobStore: handle arbitrary os.link errors to work on some filesystems (#2232)

Thank you to our contributors!

- Python
Published by DailyDreaming about 3 years ago

toil - 5.7.1

Changelog

Highlighted Features Added

AWS Batch Batch System (#3956) AGC Integration (#4039) + More AGC integration (#4067) + AGC megabranch (#4113) Scale TES to be able to run reasonably-sized workflows on Funnel on Kubernetes with the AWS job store (#3927)

CWL

Run CWL conformance tests via WES (#4052) Implement and test CWL loadContents from URLs to fix #4125 (#4126) Add CWL tests under ARM (#4038) Cache results of cwltool version lookup (#4141)

Misc

SGE batch system change to support serial jobs. (#4022) Performance testing for Graviton instances (#4123) Stop waiting on hostpath volumes to exist (#4146) Catch and warn about jobs going away too slowly on FileJobStore (#4149) Add documentation for the type-checking hooks (#4117) Pod murder bot (#4060) Contrib hook scripts (#4105) Allow newer google-cloud-storage (#4114) Use environment variable to set parallel partition name (#4096) Register pytest markers (#4103) Mention --export=ALL for SLURM environments (#4100) (#4102) Allow persisting workflow state in WES server across container recreation (#4082) Change toil kill to use the job store shared file API to find pig.log (#4075) Bring back kill loop in the single_machine batch system but with a timeout (#4070) Reorganize Locking (#4059) Add and test preemptability constraints (#4044) Enhanced types (#3975) Use an init process that reaps zombies on toil clusters (#3974) Add launch cluster support for ARM (#3971) Feat: square bracket to period separator (#4008) Add AGC health check endpoint (#3997) Tolerate and require typed Werkzeug (#4011) Add more static URLs for Singularity debs (#4007)

Bug Fixes

Update WES set up docs (#4027) Add real time logs (#4031) Fail fast if Docker builder is missing (#4001) Make Toil version be reported as a string in WES (#4013) Fix assorted typos within assorted comments (#4023) Make file store case insensitive (#4153) Pre-lex commands for qsub (#4150) Update Cactus and exclude broken networkx (#4107) Make toil kill work when the leader is on another machine (#4084) Wrong filename in output (#4139) Tolerate a missing VersionID key to fix #4129 (#4130) Only import from typing_extensions on old Python where we install it (#4090) Allow missing username and fix Docker build (#4077) Leave more time for concurrency measurement to fix #4012 (#4068) Stop people asking for ARM Mesos clusters to fix #4057 (#4058)

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @Hexotical, @gmloose, @kannon92, @douglowe, @gcapes, and @pmiddend!

- Python
Published by DailyDreaming over 3 years ago

toil - 5.6.0

Changelog

Highlighted Features Added

Integrate ARM Docker builds to make multi-arch images. (#3802)
WES support and server mode. (#3779)
TES batch system prototype. (#3821)
Support for new resource syntax in PBSPro. (#3048)
Toil now looks for lost jobs every minute instead of every hour (#3948)

Breaking Changes

Remove --disableCaching's true/false argument. (#3869)

CWL

CWL helper jobs: better disk & memory requirements. (#3834)
CWL: safer test path generation for post-install testing. (#3818)
More detailed names, which show up in job names sent to BatchSystem schedulers. #3941 (was #3893)
If you use scatter and collect files, duplicates are correctly dealt with and renamed. (#3968)
CWL: at the end of a job, ask cwltool to cleanup (#3965)

Kubernetes

Assign Kubernetes jobs an explicit TTL. (#3936)
Wait for node creation. (#3934)

Dependencies

Enable Dependabot updates. (#3827)
Multiple consolidated dependabot updates. (#3851)
Update addict requirement from <2.3,>=2.2.1 to >=2.2.1,<2.5. (#3861)
Bump cwltool to 3.1.20211107152837. (#3833 #3866 #3909)
Bump cwltest from 2.1.20210626101542 to 2.2.20210901154959. (#3848)
Bump flake8 from 3.8.4 to 4.0.1. (#3847)
Allow more docker-py versions. (#3860)
Remove pyyaml dependency. (#3858)

Misc

Update cleanup script. (#3937)
Remove use of sys.maxsize. (#3824)
Spelling fixes. (#3814)
Add codeql-analysis for Python. (#3825)
Update jobstore function names. (#3809)
Move AMI functions to lib. (#3810)
Add "make pyupgrade" (py36-plus). (#3805)
Type hints. (#3930)
Coalesce status calls in slurm. (#3822)
Python logging takes format values as *args. (#3852)
Change quick test to a 10 minute timeout. (#3843)
Add make uninstall to makefile. (#3883)
Fix toil kill to find shared pid.log file (with unit test). #3941 (was #3932)
Update documentation. (#3947)
Remove remains of Travis. (#3976)

Bug Fixes * Stop checkpoints from being reissued multiple times. (#3931) * Don't consult LSF config when explicitly defining memory units. (#3820) * Robustly remove state dirs. (#3836) * Fix exception checking for exit_code. (#3830) * Use exitStatus instead of exitReason for batch exit type comparison. (#3839) * Update cwltest to improve K8 runs. (#3935) * Consolidated CI Fixes. (#3887) * Fix CWL conformance tests. (#3891) * Toil-managed cluster scaling should work again with --metrics. (#3943)

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @Hexotical, @tmooney, @nikhil, @kannon92, @douglowe, @mhpopescu, @Phhere, and @gmloose!

- Python
Published by DailyDreaming about 4 years ago

toil - 5.5.0

Changelog

CWL

Add podman support; and other fixes from recent cwltool #3799
Add streaming feature for cwltoil #3694
Warn users if a different cwltool version is installed #3686
Turn on all Kubernetes CWL tests that are expected to work on Singularity #3720
Fix CWL in toil docs jobstore usage #3728
DOC: update versions of CWL support
Allow filestore bypass #3652

Misc

Numerous Type Hints. #3705 #3701 #3693 #3691 #3663 #3688 #3642 #3684 #3682 #3680 #3666 #3675
Single source of truth for job state #3776
Do not set default for statePollingWait #3774
Use absolute local paths when exportFile/importFile do not detect a schema #3767
Multi-zone balancing within regions for AWS autoscaling groups #3746
Migrate cloud-config to ignition #3488
🎡 Wheel Of Issues 🎰 #3760
Add back addBatchSystemFactory function #3754
Redirect stderr to /dev/null of lsf conf queries #3751
Set number of cores based on job.cores for OpenMP applications #3739
Google jobstore batching #3740
Locations of CLI option docs
Add AWS provisioner storage system #3727
Set cls.bucket #3726
Stream vs dowload jobs #3722
Update Toil's main python test version to 3.8. #3669
Move Travis tests to Gitlab #3675

Bug Fixes * Don't leak symlinks #3795 * Prevent exception from being raised when modifying dir permissions for clean up #3778 * Fix scontrol output parsing #3793 * Fix AttributeError #3742 * Workaround for S3 in us-east-1 #3710 * Time data format #3708 * Fix leader.py batch system std files prefix glob #3679

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @Hexotical, @ionox0, @gmloose, @juanesarango, @mhpopescu, @mberacochea, @nikhil!

- Python
Published by DailyDreaming over 4 years ago

toil - 5.4.0

Changelog

CWL

Fix cwl and wdl dependency bleed and add stand alone tests. #3582
Use MpiConfig.load() to handle MPI config file #3574
Add support for MPI with CWL. #3525

Misc

Numerous Type Hints. #3571 #3634 #3626 #3625 #3616 #3614 #3601 #3592 #3590 #3581
Configurable Grafana port #3597
Handle streaming reads from the cache when the data isn't written back yet #3595
Balance over pools using AWS ASGs #3490
Change how Kubernetes schedules and scales for hopefully better scale-down behavior #3587
Add Owner tags to our AWS buckets when testing. #3577
Allow workdir override #3586
Additional info for "Permission denied" error #3579
Consolidate memory functions #3529
Pen children #3482
Sniff raid better #3526
Add a propagation policy to batch delete #3522
Cleanup script for buckets, sdb domains, instance profiles, and roles. #3373
Add decorator for flaky tests. #3510

Bug Fixes * Enforce valid uuid from getNodeId() #3611 * Removed extra memory multiplication #3608 * Fix non integers lsf memory requested #3609 * Fixes jobCommand unpack #3610 * Fix CWL test 20 on Kubernetes #3572 * Fix #3579 breaking AWS docs #3596 * Update cactus test to fix broken bucket links. #3594 * Allow cleaning up a job whose overlargeID fell off #3584 * Only deploy a user script if we can deploy user scripts #3518 * Reduce path name lengths. #3438 * Insist on credentials for testing AMI finding #3524

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @thiagogenez, @julian-klode, @darafferty, @nikhil, @mhpopescu!

- Python
Published by DailyDreaming over 4 years ago

toil - 5.3.0

Changelog

CWL

Run CWL conformance tests on Kubernetes. #3323
CWL symlinking files into work directory #3445
set resource reqs for all CWLINTERNALJOBS #3442

WDL

Add WDL flatten() function + tests. #3485
Add WDL collectbykey() function + tests. #3476
Add WDL keys() function + tests. #3460
Add WDL as_pairs() function + tests. #3364
Add as_map() WDL function #3448
Add basic WDL development support #3434
Add basic WDL 1.0 support #3421
Update dictionary structure in AnalyzeWDL. #3416

Misc

Add extra context for disk usage warning. #3495
Update developing.rst #3479
Allow text/encode options for read and write functions in file store #3428
Type hinting and mypy checking additions #3470 # #3458 #3456
Quality of life improvements for cactus. #3463
Add checklists for reviewing and merging PRs #3432
Ensure that python3 and not python (2) is used #3446
Support Kubernetes clusters in toil launch-cluster #3357
Additional check for predecessors. #3417
Allow setOptions to accept an argparse group. #3426

Bug Fixes * AWS: Handle socket timeout during AWS discovery #3503 * Use back-up flatcar AMI if not found. #3513 * Lsf command parser #3475 * Retry on s3 throttling. #3504 * Stop deleting all ASGs with tags #3474 * Only determine execute permissions on files we copy/move. #3437 * cope with an invalid HTTP_PROXY #3447 * Fix failing Google jobstore test #3420 * Update Google job store #3412

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, Arthur Rand, @mberacochea, @Jessime, @thiagogenez, @julian-klode!

- Python
Published by DailyDreaming almost 5 years ago

toil - 5.2.0

Changelog

CWL

Update to the latest cwltool (3.0.20201121085451 -> 3.0.20201203173111). #3375
Add better handling for potentially mis-ordered CWL args. #3395
Confirm CommandInputParameter expression can receive a File object. #3350

WDL

Make AnalyzeWDL abstract to allow implementation for different WDL versions #3391

Misc

Clean up memoize.py. #3374
Refactor bioio library, simplify logging, and small adjustments to utils. #3351
Use regular division for memory calculation on LSF. #3387
Remove unused --nodeOptions arg. #3397
Get boto S3 args for minio from environment variables. (#3370)
Update EC2 lists. #3376
Add a retry to AWS destroy. #3379
Regular division for memory calculation in LSF. #3387
Add option to blank mem allocation for SLURM. #3399
Preserve file permissions on imported/exported files #3322

Bug Fixes * Preserve file permissions on imported/exported files. #3322 * Make building a flattened list of live jobs non-recursive. #3394 * Move AWS job store uploads/downloads to boto3 (reopened) #3400 * Use a Docker Hub mirror when pulling Docker images. #3411 * Corrected gridengine maxMEM check to use memorystring #3410

Thank you to our contributors: @mr-c, @adamnovak, @w-gao, @jonathanxu18, @douglowe, @stevekm , @thiagogenez!

- Python
Published by DailyDreaming about 5 years ago

toil - 5.1.0

Version 5.1.0

Changelog

WDL

Add WDL cross function. #3360

Bug Fixes * Better error message for insufficient disk. #3362 * Fix maxMEM comparison in LSF. #3369

Thank you to our contributors: @mr-c , @adamnovak , @w-gao , @jonathanxu18 !

- Python
Published by DailyDreaming about 5 years ago

toil - 5.0.0

Caching is now turned on by default.

CWL v1.1 is also now fully supported and passing all conformance tests.

The following batch system names are now deprecated (#3225) and replaced by: - singleMachine -> singlemachine - gridEngine -> gridengine - LSF -> lsf - Mesos -> mesos - Slurm -> slurm - Torque -> torque - HTCondor -> htcondor - Kubernetes -> kubernetes - k8s -> kubernetes

Changelog

CWL

Upgrade to a CWL v1.2.0 capable cwltool (3.0.20200709181526 -> 3.0.20200807132242). #3137
Export to a bucket if cwl's CreateFile is specified. #3124
CWL v1.1 Support: loadContents if True for StepValueFrom objects. #3266
CWL: Fix workdir permissions. #3229
Test CWL secondary file from s3. #3311
CWL v1.1 Support: Inplace update has side effect on directory content. #3280
More CWL v1.2 conformance tests. #3336
CWL: Reformat w/ Black and better support for tmpdir_prefix with suffix. #3333

WDL

Add ceil() and floor() WDL functions. #3168
Create a stdout and stderr file for each WDL task. #3181
Add WDL write functions and builtin tests. #3236
- write_lines
- write_tsv
- write_json
- write_map
Add WDL read functions and builtin tests. #3244
- read_lines
- read_tsv
- read_json
- read_map
- read_int
- read_string
- read_float
- read_boolean
Add transpose() WDL function. #3273
Add length() WDL function. #3307
Add sub() WDL function. #3252
Add range() WDL function. #3277
Add size() WDL function. #3255
Add zip() WDL function. #3355
Implement WDL Pair type. #3304
Refactor WDL types #3335

Kubernetes

Adding Labels to Pods/Jobs. #3233
Make kubernetes & botocore packages fully optional. #3259 #3261
KubeWatch optimization. #3227
Catch other kubernetes imports. #3261
Automatically infer singularity for kubernetes. #3279

Misc

Allow Caching By Default. #3111
Update dependencies and loosen restrictions slightly. #3139
Ensure that falsey (but non-null) values are considered. #3143
Remove pathlib. #3169
Move AWS job store uploads/downloads to boto3. #3153
Update prom/node-exporter to use quay.io. #3186
Add support for TOILCUSTOMINIT_COMMAND. #3183
Update docker module version to 4.3.1. #3222
Add 'stream' and 'demux' options to apiDockerCall. #3224
Use boto3 to create our leader node. #3145
Prevent job kind filenames from getting too long. #3230
Add a retry decorator. #3144
Remove mesos offer messages. #3308
Job Concept Unification. #3250
Add reporting for accessed files of failed jobs. #3309
Add coverage and refactor. #3314
Expose --statePollingWait param through toil. #3321
Record the maximum memory used by LSF jobs. #3327
Remove custom math functions and replace with built ins. #3338
Define a changelog process. #3316
SLURM: getting job details from scontrol show jobs was non-functional and has now been fixed. This behavior in the case of sacct not being configured now works. #3346
LSF: handle unicode configuration files #3354
Replace abssympath w/ builtin. #3356

Bug Fixes * Turn off watch code in Kubernetes. #3175 * Check against joining the current thread during CachingFileStore destructor. #3187 * Slurm.py error, undefined stdout - stdout should be stderr in code. #3194 * Fix restart flag error (issue #3094). #3142 * Batch System generates a key error instead of a usage message. #3225 * Remove Python 2 imports in WDL. #3228 * Move checkForDeadlocks() in the leader. #3234 * Fix boto3 migration bugs. #3243 * Fix None check when parsing WDL JSON file. #3272 * Fix broken stdout/stderr in torque job. #3306 * Add doubleMem for LSF jobs that die due to imposed memory limit. #3313 * Add 500 status to the list of retriable SDB BotoServerErrors. #3329 * Add a retry logic for BucketNotEmpty error. #3341 * Fix None check when parsing WDL JSON file. #3272 * Warn when trying to clean non-existent job store #3288 * Only rm a statefile that exists. #3348 * CWL: Remove unprocessed secondaryFiles from both Dict and List types.

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @diekhans, @w-gao, @jonathanxu18, @jeffrey856, @davidlougheed, @altairwei, @mberacochea, @drkennetz, @ionox0!

- Python
Published by DailyDreaming about 5 years ago

toil - 4.2.0

Don't log in the inner scheduling loop. #3065
Handle anticipated 404 errors without retrying. #3067
Test and reimplement AMI finding via Flatcar JSON feed. #3061
Bump Enlighten version to partly fix #3069. #3070
Get wait duration is far too short of a rest for lsf. #3076
Unify logging format. #3073
Support CWL 1.1 Listing. #3058
Enable CWL v1.2.0-dev3 (3.0.20200324120055 -> 3.0.20200530110633). #3092
Enable CWL v1.2.0-dev4 (3.0.20200530110633 -> 3.0.20200709181526). #3105
Add --nodeStorageOverrides option. #3096
More robust CWL filepaths and secondary files. #3114
Replace custom mkdir_p with the built in. #3123
Move conditional execution test to before fillindefaults. #3117

Bug Fixes * Stop CWL from trying to copy downloaded directories from somewhere local. #3053 * Exit the worker with 1 when jobs fail so the batch system can see. #3052 * Eliminate recursion in root finding to fix #3080. #3081 * Add a Cactus Kubernetes test to the integration tests on Gitlab. #3077 * Chaining doesn't make sense for CWL. #3091 * Fix failed job accounting. #3103 * Improve lsf batchsystem stability. #3101 * Deduplicate updated jobs. #3108 * Redundant virtualenv check. #3135 * Change Slurm command output processing. #3133 * Fix for log file being linked more than once. #3130 * Wrap Kubernetes watches to fix #3125. #3126 * Create dirs that do not exist when bind-mounting to docker. #3120 * Fix database state when file upload fails. #3122

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @diekhans, @nikhil, @drkennetz, @jonathanxu18, @tobiaszjarosiewicz, @ionox0!

- Python
Published by DailyDreaming over 5 years ago

toil - 4.1.0

Attempt to make batch system more robust and debuggable. #2959
Kubernetes Shared Caching. #3012
Test python3.8 and build a python3.8 appliance. #3028
Use the original pymesos. #3036
Detect more kinds of virtual environment and normalize prefixes. #3035
Do More Hashing. Can't have enough. #3025
Replace reissued jobs message with better progress indicators. #3044
Detect insufficient resources and deadlocks more usefully. #3043

Bug Fixes * Don't list specific CWL features, try them all (and fix some CWL running bugs). #3015 * Always honor TOILGRIDENGINEPE and never assume a site-dependent default. #3041

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @diekhans!

- Python
Published by DailyDreaming almost 6 years ago

toil - 4.0.0

This release moves to python3 only, dropping all python2.7 compatibility, and also deprecates the command "cwltoil" in favor of "toil-cwl-runner".

Update cwltool version (==1.0.20190906054215 -> <=2.0.20200126090152). #2969
Drop the long-deprecated cwltoil. #2843
Dropped python2.7. #2973
Port provisioner tests and only test py3.6. #2842
CWL: Refactor of link merge + conditionals + pickValue. #2845
Add import subprocess to executor.py. #2976
Encode mtail stdin to utf-8 and flush. #2978
Encode framework message as utf-8 as well. #2981
Make the output files for grid engine batch systems not try and be in per-host directories. #2956
Use file locks instead of PID polling to see if other processes sharing the cache are alive. #2982
Size downloads during download. #2989
Add timeouts to all SQLite connect statements. #2993
Eliminate sleep time on --restart. #2990
Add support for --awsEc2ExtraSecurityGroupId. #2997
Add moveExports option. #2983
Set worker threads as dameon to prevent hanging process on error. #2998
Cut the Threads. #2999
[Part 1] CWL v1.1 Support #2985
[Part 2] CWL v1.1 Support #3000
Update Sphinx and let its version float. #3011
Add --enableUnlimitedPreemptableRetries option. #2896
Add less and vim to docker. #3017
Improve and Unify Log Dumping. #3008
Make dependency on Python3.6+ machine-readable in setup.py. #3021
Improve slurm update frequency. #3026
Make Sphinx a make prepare component and not a dependency. #3023
Use forked pymesos and http-parser dependencies. #3024

Bug Fixes * Fix python3 string/bytes error when using --printLogs. #3005 * Fix connection between Kubernetes and Leader. #3004

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @jeffrey856, @kaushik-work, @glmxndr, @johnbradley, @cmarkello, @diekhans!

- Python
Published by DailyDreaming almost 6 years ago

toil - 3.24.0

This is the last version of Toil that supports python2.7 and allows the command word "cwltoil" (now "toil-cwl-runner").

Update ssh and docker setup files. #2958
Update EC2 instance types. #2962
Source toil environment as a configuration file. #2960
Scan for the latest caching database instead of using a hard link. #2948
Specify py3.6 or py3.x instead of just py3 in appliance. #2966

Bug Fixes * Fix database being locked when trying to write in the CachingFileStore. #2955 * Fix py3 spot market scaling. #2971

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @jeffrey856!

- Python
Published by adamnovak about 6 years ago

toil - 3.23.1

Functionally the same as Toil Version 3.22.0.

- Python
Published by DailyDreaming about 6 years ago

toil - 3.22.0

Make an appliance for Python 3. #2937
Use status to properly get the HTTP status code from boto. #2934
Use atomic file copy when writing to posix file system. #2912
Improve Kubernetes job timing. #2922
Increase SQLite DB timeout to 60s. #2913
Consider pods that are very close to their memory limits to be stuck. #2895
Constantly reload Kubernetes config. #2869
Up the disk to the default as well as the memory. #2880
Tolerate unnecessary bucket cleanup. #2894
Add checksum verification for AWS job store. #2889
Change python3.6 ppa to deadsnakes. #2888
Change Kubernetes job sizing logic. #2877
Added env var TOILKUBERNETESOWNER. #2872
Raise job limits to be more than their requests. #2868
Remove all Azure support from toil. #2860

Bug Fixes * Exclude .nfs files from deferred function search. #2930 * Fix cpu_count() to actually read quotas. #2933 * Compile http-parser so that it doesn't break pip dependencies. #2940 * Use an http-parser for pymesos that builds on 3.7+. #2931 * Fix botocredAdapter. #2904 * Make sure to commit when giving away files. #2914 * Actually call the implemented size polling method. #2916 * Set Docker vars back to ?=. #2908 * LSF: Don't check for Status , as that breaks the error code finding. #2863

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @jeffrey856, @diekhans!

- Python
Published by DailyDreaming about 6 years ago

toil - 3.21.0

Add kubernetes watch. #2848
Add service account configuration for Kubernetes. #2851
Head object 404 with encryption. #2853
Tests for Kubernetes. #2838
Turn usernames into non-spooky Kubernetes name components. #2836
Make examples a symlink to make examples discoverable. #2834
Kubernetes BatchSystem Support. #2786
Specify GCE coreos image by family name instead of specific version. #2829
Add toil-cwl-runner --provenance. #2785
Revise caching more. #2770
Bump psutil >= 3.0.1, <6. #2790
Add "failed_" prefix to failed logs. #2792
Add --writeLogsFromAllJobs option. #2781
Expose --default-container to toil-cwl-runner. #2784
Update cwltool from 1.0.20190228155703 -> 1.0.20190815141648. #2764
Pass jobID to Torque Wrapper. #2762
Limit Length of Log Names. #2755
Add CWL options --strict-memory-limit, --relax-path-checks, and --leave-container. #2718
Update EC2 nodes and script. #2709
Redesign File Job Store layout. #2669

Bug Fixes * Fix for passing default values between Workflows. #2831 * Ignore checking versioning consistency when cleaning s3 jobstore. #2826 * Ensure terminated ignored nodes are cleared from batch system. #2789 * Fix Slurm sbatch command. #2777 * Typo in gridengine.py. #2778 * CWL fixes. #2783 * Clarify /data mountpoint for dockerCall. #2747 * Fix CWL tmpdir to be unique per job. #2745 * Protect against potential AWS truncated downloads. #2731 * Fix runningOnEC2 to work with all instances. #2734 * Prevent Toil from zipping entire dirs. #2728 * Changes to lsf.py to make it compatible on lsf9 and lsf10. #2726 * Remove X86_64 type requirement from lsf support. #2723 * Clean up temporary directories cwltoil uses to hold output streams. #2690

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @jeffrey856, @tthyer, @kaushik-work, @dleehr, @ionox0, @psafont, @mb1069, @tobiaszjarosiewicz, @gconcepcion, @byee4, @smoe, @TMiguelT, @jvivian-atreca!

- Python
Published by DailyDreaming over 6 years ago

toil - 3.20.0

Deprecate subprocessDockerCall(). #2653
Move wallTime to a label so we can use exact timestamps. #2664
Capture and log standard output/error from batch systems. #2662
Add an option to set custom AWS IAM roles. #2656
Add a ToilNodeType tag to distinguish leader vs. workers. #2658
Slurm: include both exit code and terminating signal in job return status. #2646
Update requests package. #2610
HTCondorBatchSystem(): fixes and features. #2647
Update cwltool from 1.0.20180820141117 -> 1.0.20190228155703 #2609
Add an option to set disableCaching to false. #2584
Speed up S3 bucket deletion. #2597
Tolerate missing jobs/workers in stats expando. #2533
Add private IP support to node operations. #2569
Encode logs for gzip. #2595

Bug Fixes * Improve NoSuchFileException usage and robustness. #2593 * Make sure we always have current package lists. #2675 * Make sure to use --tmpdir-prefix to place workdir and jobstore. #2587 * Subprocess dockercall bytestring compatibility for py2/3. #2677 * Address python3 deprecation warnings. #2668 * Fix python3.7 break on new static method. #2659 * Get value of taskId in Mesos' killTask method. #2661 * Stop NamedTemporaryFile tampering. #2638 * StatsAndLogging(): ensure json.load() receives only string input. #2641 * Reduce max size of files allowed to be inlined in SDB. #2440 * Increase hardcoded encapsulate follow-on resources. #2591 * Ensure TOILDOCKERREGISTRY variable is in version_template. #2583 * Use current python executable. #2599

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @kwwette, @liinus, @saimeCS, @cdunn2001, @Fifan31, @jeffrey856!

- Python
Published by DailyDreaming over 6 years ago

toil - 3.19.0

Support for CWL s3 imports. #2522
Support for running custom init commands in docker (allowing auth in private dockers). #2561
Add --no-read-only option to the CWL runner. #2531
Add support for assuming roles. #2527
Added an adapter to allow Boto 2 to use credentials obtained via Boto 3. #2527
Add a flag for running CWL internal jobs on workers. #2524
Added CWL tests for http, https, and s3 file imports. #2234
Begin testing python3 in gitlab. #2518
Relegate pytest dependency to the dev reqs. #2508
WDL changes to enable resource usage. #2495
Sphinx and pathlib2 dependencies. #2490 #2493
Upgrade appliance to Ubuntu 16.04. #2480
Update cwltool from 1.0.20180820141117 -> 1.0.20181118133959 #2469
Add CWL's --preserve-entire-environment option. #2481
Bump minimum version of galaxy-lib (17.9.3 -> 17.9.9). #2476
Added all tutorial scripts to unit testing. #2426
More python3 support added. #2405
Update the CoreOS AMI. #2430
Simplify log creation in leader. #2423
Support for hardlinks and greater testing for toil status. #2402
Pymesos port. #2366 #2338
Use displayName for CWLJob to show stats in toil --stats. #2397

Bug Fixes * Import errors. #2558 * Fix boto3 credentials for permanent credentials. #2549 * Fix bug in Toil stats. #2513 * Replace some str objects as real FileIDs. #2500 * Remove linear-time bottleneck from Mesos batch system. #2498 * Slurm and py3 fixes. #2494 * Safeguard unintentional boto imports. #2489 * Fix broken readthedocs build. #2485 * Specify region in prometheus. #2437 * Fixed bytestring-string tagging. #2472 * LSF log job id on error. #2450 * Fix massive slowdown when many jobs are running. #2439 * Make ServiceManager stop blocking while (most) services are established. #2441 * Default to bash on the leader node rather than sh. #2435 * resource.getusage returns KB already, so dividing by 1024 yields MB. #2425 * Recognize conda virtual envs. #2422

Thank you to our contributors: @mr-c, @arostamianfar, @adamnovak, @nikhil, @lamby, @gijzelaerr, @joelarmstrong, @markasbach, @davidjsherman, @psafont, @jsmedmar, @bencvdb, @emlys, @ionox0, @ibebio, @MarkFilus!

- Python
Published by DailyDreaming almost 7 years ago

toil - 3.18.0

Refactor stream writes in the fileJobStore to minimize holding open file handles
Large python 2/3 compatibility changes (+ python 3 compatible mesos library: pymesos)
Get status/PID API added
CWL:fix support for user space overrides
Update cwltool from 1.0.20180518123035 to 1.0.20180820141117
support lsf 10 per core reservation
WDL bucket outputs with orig names
Grid engine job creation fix
Add Mesos port forwarding and the ability to use a SOCKS proxy with toil ssh-cluster
Require a jobstore to be specified when a provisioner is.
Retry HTTP connections on BadStatusLine errors
Quote() all Docker arguments
log LSF job id to error on failure
Improve test file cleanup
retries for batch commands
Update CoreOS AMI
CWL workDir defaults to tmpdir_prefix now if not set by the user
Set the default duration for batch systems with getWaitDuration()
Coverage Evaluation

Bug Fixes * AMI Fetch Timeout * Stop logging every message twice in certain cases * Fix htcondor in wheel breaking on Apple * Fix toil kill * Fix invalid qsub in torque.py

Thank you to our contributors: @mr-c (big thanks for all of the CWL support!), @MarkFilus, @xzhub, @ionox0, @jsmedmar, @bencvdb, @tom-dyar, @joelarmstrong, @mwiens91, @jessebrennan, and @adamnovak!

- Python
Published by DailyDreaming over 7 years ago

toil - 3.17.0

Add CWL destBucket runtime option to deposit CWL outputs into cloud buckets
Update boto3 and boto
Better Error messages when docker images are unavailable
Broader checking for docker image existence prior to launching blindly
Incremental Python2/3 compatibility improvements
Enable support for specifying zone in all environment variables
Expose --force-docker-pull and --no-match-user options from cwltool
Add mem free option to UGE
Convert most log messages to debug level
Update cwltool from 1.0.20180518123035 to 1.0.20180521150620
Incremental WDL Support Improvements (docker permissions, parsing, and improved logging)
Documentation Housekeeping and Code Cleanup

Bug Fixes * Stop FileJobStore DEBUG messages from sneaking out once for every job * WDL Parsing Error * Fix docker daemon detection depending on terminal size * Checking virtualenv * Properly filtering AWS security groups * Fix a crash if job is larger than all available node types * Avoid overwriting NVMe root volumes on current-gen AWS instances * Don't call addNodes if the delta was met by unignoring nodes * Fix AWSJobStore updates not working if values are shrunk * Prevent worker from checking for nonexistant file

Thank you to our contributors: @mr-c (big thanks for all of the CWL support!), @jsmedmar, @bencvdb, @tom-dyar, @joelarmstrong, @mwiens91, @jessebrennan, and @adamnovak!

- Python
Published by DailyDreaming over 7 years ago

toil - Release 3.16.0

More Robust WDL Support
Reworked the Cluster Scaler Algorithm
Updated subprocess32 to version 3.5.1
Eliminated bd2k-lib dependencies
apiDockerCall allows blocking now for both detached/attached images
Util to fetch the latest AWS instance types
Check if docker images specified by the Toil appliance are valid before running
Reduce toil clean memory usage for AWS jobStores

Bug Fixes * CWL fixes for handling promises / followOn jobs * Fix occasional cPickle deadlocks * Increased Google and Azure stability * Google jobstore fixes * fileJobStore deletion made more robust for NFS-like systems

Thank you to our external contributors: @mr-c (big thanks for all of the CWL support!), @chapmanb, @ryanking, @jmchilton, @jorvis, @evan-wehi, @brainstorm, @jsmedmar

- Python
Published by DailyDreaming over 7 years ago

toil - Release 3.15.0

Azure Support (Beta)
Google Support (Beta)

Bug Fixes * CWL fixes. * LSF cpu count fix.

Thank you to our external contributors: @mr-c (big thanks for all of the CWL support!), @chapmanb, @ryanking, @jmchilton, @jorvis, @evan-wehi, @brainstorm, @jsmedmar

- Python
Published by DailyDreaming almost 8 years ago

toil - Release 3.14.0

Added support for the HTCondor Batch System
Updated documentation.
Added the protobuf dependency for mesos.
Updates for CWL (scheduling, update CWL ref runner, update conformance tests, expose cwltool's tempdir-prefix, expose user-space-docker-cmd, and always import links for CWL)
Added an "all" option to install all extras.
Check for venv in python 2 and 3.

Bug Fixes * Added buffering for local file transfers to fix MemoryError bugs * Fix race conditions for SLURM, GridEngine, Mesos, and Torque batch systems. * Updated file size determination on Apple products. * Fixed a bug when making the docs through Sphinx. * Amended memory limit and reservation in LSF. * Removed cwl-runner command, as it was interfering with the native CWL command.

Thank you to our external contributors: @mr-c (big thanks for all of the CWL support!), @chapmanb, @ryanking, @jmchilton, @jorvis, @evan-wehi, @brainstorm, @jsmedmar

- Python
Published by DailyDreaming about 8 years ago

toil - Release 3.13.0

WDL Alpha support
Job store symlinking
Updated cwltool support (version; fixes; testing)
Better logging
Option to get stderr from subprocessDockerCall
Integration testing with ENCODE
Cactus with autoscaling

Bug Fixes * Fixed static provisioning * Fixed launch-cluster bug when terminal size was too small * Removed AWS specific code from the abstract provisioner

Thank you to our external contributors: @mr-c, @chapmanb, @ryanking, @jmchilton, @jorvis, @evan-wehi, @brainstorm, @jsmedmar

- Python
Published by DailyDreaming about 8 years ago

toil - Release 3.12.0

Forkless mode for debugging
Enabled Travis builds with a subset of tests (less than 15 minutes).
DockerCall function changed to use docker API.
Updated to Azure version 2.0.
Added tempDir convenience function to job and made job.log an alias for fileStore.LogToMaster.
Enhanced toil-cwl-runner to support SoftwareRequirements & BioContainers.
Increased default retry count to 1.
Deprecated cgcloud.
Updated EC2 instance types.
Upgraded pip to 9.0.1 and setup tools to 36.5.0 in appliance build.

Bug fixes * Parasol parses jobStore path correctly. * Parasol doesn't attempt to remove jobs that it didn't launch. * Fixed problem with dynamic batch systems not always being created. * Fix workflow effort being wasted if a single export fails. * Added a default value for maxNodes. * Fixed py.test collection errors when running locally.

Thank you to our external contributors: @jmchilton, @jorvis, @evan-wehi, @mr-c, @brainstorm

- Python
Published by ejacox over 8 years ago

toil - Release 3.11.0

Allow batch creation of jobs.
CWL: run internal jobs locally, not on cluster.
TOILTESTQUICK environment variable for running a subset of tests.
Allow clusters with multiple node types.
Put jobName into the toilworker arguments.
GridEngine: cache instead of block for statePollingWait

Bug Fixes * CWL: avoid early removal of the temporary outdir * Fixed AWS readFileStream deadlock on partial reads. * Use a socket to get the public-facing IP

Thank you to our external contributors: @chapmanb, @ryanking, @mr-c

- Python
Published by ejacox over 8 years ago

toil - Release 3.10.1

Adds missed commits that should have been in 3.10.0.

- Python
Published by ejacox over 8 years ago

toil - Release 3.10.0

Full CWL 1.0.1 support.
Improved documentation with better examples.
Added option to link files when importing into fileJobStore.
Allowed some interactive support
Improved local testing targets.
Sleep support for all state-querying functions in batch.
Opened up UDP traffic within AWS clusters.
Set mesosMasterAddress default to private IP.
CWL: symlink jobStore files instead of copying.

Bug Fixes * Fixed getEmptyFileStoreID. * Automatically use the correct LSF units. * Fixed failure in resolveindirectinner in cwltoil.py. * Fixed docker tests failing on Mac. * Stopped log level option defaults from clobbering each other. * Various other minor bug fixes.

Thank you to our external contributors: @mr-c, @tetron, @liuxf09, @ryanking, @OliverEvans96, @alaindomissy, @brainstrom, @chapmanb, @nazamy, @jmchilton, @stefanoberri

- Python
Published by ejacox over 8 years ago

toil - releases/3.9.0

Added a plugin batch systems that abstracts option settings for batch systems.
Fixed mysterious fail with rsync --protect-args on OS-X.
Added environment variable TOILAWSKEYNAME for setting AWS credentials during tests.
Tests now timeout with pytest rather than Jenkins.
Added support for PBSPro (Torque's proprietary version).
Fixed resolution of symlinks for hard links bug.
Added an option to set root volume on leader and worker nodes.
Fixed autoscaler crash.
Added environment variable TOILTORQUEARGS for setting Torque arguments.
Fixed hot-deployment with python-3-supporting libraries.
Fixed race in ClusterScaler test.
Several logging improvements and fixes.
Pip show now displays license info.

- Python
Published by ejacox over 8 years ago

toil - Release 3.8.0

Don't error if job hasn't been submitted yet (resolves #1666)
Match doc to reality for --preserve-environment
CWL: symlink jobStore files instead of copying
When killing tasks, make sure to reap child processes in the process groups
Better error checking in piped docker commands
Fix case where a job has been put in currentjobs, but not yet dispatc…
Use hard links instead of copied files on import to file job store
CWL: Add support for InitalWorkDirRequirement
Set proper value for service job preemptable requirement
Use chunked transfer in AWS job store importing
Lower timeout for domain binding in Toil Clean
Add import os.

- Python
Published by ejacox over 8 years ago

toil - 3.7.0

Updated documentation at http://toil.readthedocs.io/en/latest/
Improved Azure support
Remove upper limit on size of jobs in AWS job store
CWL workflows can now take advantage of Toil Autoscaling
CWLToil bug fixes and enhancements
Toil Appliance updated to Mesos version 1.0.1
Improve scalability of Mesos batch system
Better error message for unfulfilled promises
Fix handling of checkpointed jobs
Fix service job deadlock detection
Allow disabling of hot deployment mechanism

- Python
Published by cket almost 9 years ago

toil - 3.5.1

Hotfix release for 3.5.x - Fixes import error raised when 'AWS' extra not installed

- Python
Published by cket about 9 years ago

toil - 3.5.2

Hotfix release for 3.5.x - Fixes job state discovery race - Fixes race between job store file exporting and caching - Fixes bugs in Slurm batch system

- Python
Published by cket about 9 years ago

toil - 3.6.0

New Toil landing page at http://bd2kgenomics.github.io/toil/
Tags all Toil-provisioned instances with an 'Owner' tag to improve cost tracking
Allows users to arbitrarily tag their Toil-provisioned instances
Adds a new Torque batch system class
Improved documentation
Support for piping commands within DockerCall
Improved cluster usage statistics
Various bug fixes improving support for large file exporting, the Slurm batch system, and general Toil robustness

- Python
Published by cket about 9 years ago

toil - 3.5.0

Toil now includes a native EC2 provisioner for launching and controlling Toil clusters.
Toil now has the ability to autoscale - if enabled, Toil will increase or decrease the number of worker nodes in its cluster depending on the current demands of the workflow
Log output now refers to each Job by name.

- Python
Published by cket about 9 years ago

toil - 3.3.4

https://pypi.python.org/pypi?:action=display&name=toil&version=3.3.4 - Fix: Toil fails completely when not installed with [aws] extra (#1115) - Prevent virtualenv from silently upgrading pip and setuptools, potentially breaking build (#1270)

Core production is advised to skip this release.