Recent Releases of caper
caper - v2.3.1
- Use file metadata DB by default (
--db file). If you see any DB connection timeout then switch back toin-memoryDB with--db in-memory, then you will pass the error but call-caching will be disabled. Changed default file path for such file DB (
--file-db):${local_out_dir}/caper-db_[WDL_BASENAME]_[INPUT_JSON_BASENAME].local_out_dircan be defined either in command line or conf file (~/.caper/default.conf).Can use a relative path in input JSON. If it still doesn't work.
pip install autouri --upgradeChanged local file hashing strategy from
path+modtimetoxxh64to prevent call-caching's cache miss due to the same text file with differentmodtime(e.g. modified bytouch).Fixed
filelockissue. Runpip install autouri --upgradeto make sure to haveautouri>=0.4.4.
- Python
Published by leepc12 about 3 years ago
caper - v2.2.0
Deprecated scg and sherlock backends.
Use slurm backend instead and follow instruction on Caper's conf ~/.caper/default.conf after caper init slurm.
New HPC feature
- Unified command caper hpc for all HPC backend types (slurm, sge, pbs and lsf).
- caper hpc submit WDL -i INPUT.json --leader-job-name GOOD_NAME_FOR_LEADER [--conda | --singularity]
- caper hpc list: shows all Caper leader jobs
- caper hpc abort JOB_ID: terminate a Caper leader job including child jobs
Default walltime for a leader job is 48 hours by default. Check Caper's conf ~/.caper/default.conf to change it.
- Python
Published by leepc12 about 4 years ago
caper - v2.1.0
Fixed all Singularity/Conda issuess on HPC backends (slurm, sge, pbs and lsf)
- Fixes https://github.com/ENCODE-DCC/caper/issues/149
- docker attribute in WDL task's runtime is now safely ignored for HPC backends
- Singularity directory binding issue (sharing soft-linked input/output files between tasks)
Makes a new Cromwell STDOUT file if it exists
- For example, if cromwell.out exists on CWD then cromwell.out.1 is created.
- Python
Published by leepc12 over 4 years ago
caper - v2.0.0
New Caper 2.0.0. is out.
There is no significant update for cloud-based backends (gcp and aws).
Most updates are for HPC USERS to fix various annoying Singularity/Conda and cluster (SLURM, ...) issues.
For HPC users
HPC users must initialize Caper's configuration file. This will add new important parameters and description/instruction for them. Please make a copy of your original configuration file (~/.caper/default.conf) and then initialize it with caper init.
bash
$ cp ~/.caper/default.conf ~/.caper/default.conf.bak
$ caper init [YOUR_BACKEND] # local, slurm, sge, pbs, lsf. See README.md for details
Follow instruction comments in the configuration file. Most parameters will be the same as those in the original configuration file.
For Conda/Singularity users on HPC
YOU DO NOT NEED TO ACTIVATE CONDA ENVIRONMENT BEFORE RUNNING PIPELINES. Just make sure that pipeline's Conda environment is correctly installed and add --conda to the command line caper run ... --conda. New Caper 2.0.0 runs each WDL task inside a Conda environment.
We strongly recommend to use Singularity for new ENCODE ATAC-seq and ChIP-seq pipelines (both >=v2.0.0). caper run ... --singularity if your cluster support Singularity.
Added resource parameters for HPC
Added a resource parameter for each HPC backend so that users can customize resource parameters for the job submission command line (squeue, qsub, ...) according to their cluster configuration. It will be appended to the job submission command line. WDL syntax is allowed in ${} notation. Please find details in the configuration file after initialization. See README for details.
slurmbackend:slurm-resource-paramsgebackend:sge-resource-parampbsbackend:pbs-resource-paramlsfbackend:lsf-resource-param
- Python
Published by leepc12 over 4 years ago
caper - v1.6.3
No code changes to v1.6.2 except for documentation.
However, AWS users should reinstall AWS Batch with a new CloudFormation template built for Caper. There were several issues with the old installation method (EBS autoscaling and bucket permission issues).
Please read this instruction carefully.
- Python
Published by leepc12 about 5 years ago
caper - v1.6.0
AWS backend is updated and fully tested.
- Call-caching (re-using previous outputs) works fine but Cromwell always makes cache copies on new workflow's output directory even with
--aws-call-caching-dup-strat reference. See this issue for details.
- Call-caching (re-using previous outputs) works fine but Cromwell always makes cache copies on new workflow's output directory even with
Users can use HOCON's
includestatements in custom backend configuration file (specified with--backend-conf).
- Python
Published by leepc12 about 5 years ago
caper - v1.5.0
If you upgrade from previous version then please upgrade autouri too.
bash
$ pip install autouri --upgrade
$ pip install caper --upgrade
Upgraded Cromwell from 52 to 59. Metadata DB is compatible between two versions.
Please check your Caper configuration file at ~/.caper/default.conf to remove cromwell and womtool from the file. Then Caper will automatically download Cromwell/Womtool 59 and use them.
```ini
remove these two lines
cromwell=~/.caper/cromwelljar/cromwell-52.jar womtool=~/.caper/womtooljar/womtool-52.jar ```
Fixed a bug that takes too long to recursively localize big files from S3 to GCS
- md5 checksum skipping flag was not propagated for recursive localization (in JSON, CSV, TSV) in autouri
- Fixed in autouri
If you get the following error, then check your service account's settings this document.
400 Bad Request
POST https://lifesciences.googleapis.com/v2beta/projects/XXX/locations/us-central1/operations/XXX:cancel
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Precondition check failed.",
"reason" : "failedPrecondition"
} ],
"message" : "Precondition check failed.",
"status" : "FAILED_PRECONDITION"
}
- Python
Published by leepc12 over 5 years ago
caper - v1.4.2
Parameter name changes
- auto-update-metadata -> auto-write-metadata: in both CLI and python functions (_ instead of -).
Generation of metadata.json file on workflow's root output directory
- It was written for any change in task/subworkflow/workflows status change, which possibly stressed Cromwell server and resulted in its crash. It's written only when a main workflow is done whether successful or not.
Bug fixes - Fix wrong graceful shutdown message even when Cromwell is done successfully.
- Python
Published by leepc12 over 5 years ago
caper - v1.4.0
Change of CLI parameters:
- --hide-subworkflow -> --show-subworkflow. Caper defaults to hide subworkflows.
Added features
- --disable-auto-update-metadata: Disables auto write/update of metadata.json file on workflow's output root directory.
- NBSubprocThread
- Separated streaming (with two deamonized threads instead of one) of STDOUT and STDERR of a subprocess.
- Better error handling. Correctly sends signal (signal.SIGTERM by default) to a subprocess thread when interrupted by stop().
- CromwellRestAPI
- find() defaults to exclude_subworkflow.
- Does not retrieve all workflows unless wildcard (* or ?) is in a query as workflow_id or label (as value in a key/value pair).
Bug fixes
- --java-heap-server not working
- NBSubprocThread
- Not correctly capturing Cromwell's STDOUT/returncode while gracefully dying.
- Recursively sending too many requests to retrieve information of all workflows.
- This recursively happened every update of workflow/subworkflow/task's status change, so lead to crash of Cromwell server running WDLs (e.g. ENCODE dnase-seq-pipeline) with nested subworkflows.
- Python
Published by leepc12 almost 6 years ago
caper - v1.3.3
If pip-upgrading from a previous version, users should upgrade Python requests >= 2.20.
bash
$ pip install requests --upgrade
Bug fixes
- Job submission parameters for PBS backend.
- Python requests bug (bool type in REST API).
Use Cromwell's native subworkflow recursion for retrieving metadata. - This will fix most subworkflow-related issues.
- Python
Published by leepc12 almost 6 years ago
caper - v1.3.0
gcp-memory-retry currently doesn't work. It should be fixed on Cromwell's side. See this for details.
If upgrading from old Caper, then manually upgrade autouri to >=0.2.1 to fix the public bucket access issue.
bash
$ pip install autouri --upgrade
$ autouri -v # check if version >= 0.2.1
Linear resource analysis for workflows run on Google Cloud.
- Can run this with any workflows run with Caper>=1.2.
- Usage: caper gcp_res_analsysis WORKFLOW_ID_OR_METADATA_JSON ... MORE_WORKFLOW_ID --plot-pdf result.pdf
- For each task, collect resource data from multiple workflows and do linear regression on
- x: Input file sizes.
- To simplify the problem to a single linear regression, all input file sizes are summed up by default. But users can use multiple linear regression without summation (scatter plot won't be available).
- 2D scatter plot with a fitting line.
- y: Resource taken
- Max memory
- Max disk usage
- Examples (ENCODE ATAC-Seq pipeline)
- Plot: https://storage.googleapis.com/caper-data/gcpresourceanalysis/exampleplot/atac.pdf
- Input file variable definition JSON
- Can be defined with --in-file-var-def-json.
- If not defined, then analyze all tasks with all input variables.
- https://storage.googleapis.com/caper-data/gcpresourceanalysis/infilevarsjson/atac.json.
Change of default parameters for Python API.
- local_loc_dir: defaults to CWD/.caper_tmp.
- local_out_dir: defaults to CWD.
Bug fixes
- Cannot access to Google public buckets.
- Updated localization module autouri>=0.2.1 to use Google's anonymous client for public buckets.
- caper init fails due to ~/ not expanded correctly.
- Server instance creation shell script fails when parsing startup script.
- scripts/gcp_caper_server/create_instance.sh
- Misleading error messages
- work_dir -> loc_dir.
- Python
Published by leepc12 almost 6 years ago
caper - v1.2.0
Added caper cleanup to clean up outputs of a single workflow.
- Usage:
- caper cleanup [METADATA_JSON_FILE]: Define a metadata.json file for the first positional argument.
- caper cleanup [KEYWORD_TO_FIND_WORKFLOW]: Define a keyword to find a workflow from a running caper server.
- This DELETEs ALL FILES on a workflow root directory (under key workflowRoot in metadata JSON) written in metadata. Deletion is done in a rm -rf fashion.
- This action is NOT REVERSIBLE. Use this at your own risk.
- Add --delete to actually delete all outputs. Otherwise, caper cleanup will just dry-run.
Added caper gcp_monitor to monitor resources of workflows run on Google Clouds.
- Usage:
- caper gcp_monitor [METADATA_JSON_FILE]: Define a metadata.json file for the first positional argument.
- caper gcp_monitor [KEYWORD_TO_FIND_WORKFLOW]: Define a keyword to find a workflow from a running caper server.
- Output will be either in a TSV format or in a JSON format.
- TSV: Flatten version (with dot notation) of JSON one.
- JSON: More detailed.
- This feature is available for a workflow run with Caper>=1.2.0.
- Outputs:
- Task name / shard index / status / ...
- Instance resource info: cpu, memory, Cromwell-mounted disk.
- Statistics of resources: mean, std, min, max of each resource (memory, Cromwell-mounted disk's usage, cpu_pct).
- Python
Published by leepc12 almost 6 years ago
caper - v1.1.0
Users need to change default Cromwell from 51 to 52.
- Edit ~/.caper/default.conf to replace 51 with 52 for keys cromwell and womtool.
- Or Make a backup of the conf file and start over with caper init YOUR_BACKEND.
- See README for details about caper init.
Upgraded Cromwell from 51 to 52 - Due to change of Google API, Cromwell-51 will not work after 8/20/2020. - Recently-created service accounts will not work even before 8/20/2020.
Added a shell script to automate Caper server instance on Google Cloud Platform
- Added scripts/gcp_caper_server/create_instance.sh
Lowered logging level for some annoying messages - Exporting Google Cloud environment variables.
Bug fixes
- Double slashed directory // for metadata.json on gs://.
- CLI can catch SIGTERM for graceful shutdown of Cromwell Java thread.
- Can detect mutually exclusive parameters. e.g. --singularity and --docker.
- Python
Published by leepc12 almost 6 years ago
caper - v1.0.0
Most parts of the code is rewritten for Caper 1.0.0.
Upgraded Cromwell from 47 to 51. - Metadata DB generated with Caper<1.0 will not work with Caper>=1.0. - See this note to find DB migration instruction.
Changed hashing strategy for all local backends (local, slurm, sge, pbs).
- Default hashing strategy: file (based on md5sum, which is expensive) to path+modtime.
- Changing hashing strategy and using the same metadata DB will result in cache-miss.
Changed duplication strategy for all local backends (local, slurm, sge, pbs).
- Default file duplication strategy: hard-link to soft-link.
- For filesystems (e.g. beeGFS) that do not allow hard-linking.
- Caper<1.0 hard-linked input files even with `--soft-glob-output`.
- For Caper>=1.0, you still need to use --soft-glob-output for such filesystems.
Google Cloud Platform backend (gcp):
- Cau use a service account instead of an application default (end user's auth.).
- Added --gcp-service-account-key-json.
- Make sure that such service account has enough permission (roles) to resources on Google Cloud Platform project (--gcp-prj). See details.
- Can use Google Cloud Life Sciences API (v2beta) instead of deprecating Google Cloud Genomics API (v2alpha1).
- Added --use-google-cloud-life-sciences.
- For caper server/run, you need to specify a region --gcp-region to use Life Sciences API. Check supported regions. --gcp-zones will be ignored.
- Make sure to enable Google Cloud Life Sciences API on Google Cloud Platform console (APIs & Services -> + button on top).
- Also if you use a service account then add a role Life Sciences Admin to your service account.
- We will deprecate old Genomics API support. Life Sciences API will become a new default after next 2-3 releases.
- Added memory-retry to Caper. This is for gcp backend only.
- Retries (controlled by --max-retries) on an instance with increased memory if workflow fails due to OOM (out-of-memory) error.
- Comma-separated keys to catch OOM: --gcp-prj-memory-retry-error-keys.
- Multiplier for every retrial due to OOM: --gcp-prj-memory-retry-multiplier.
Improved Python interface.
- Old Caper<1.0 was originally designed for CLI.
- New Caper>=1.0 is designed for Python interface first and then CLI is based on such Python interface.
- Can retrieve metadata.json embedded with subworkflows' metadata JSON.
Better logging and troubleshooting.
- Defaults to write Cromwell STDOUT/STDERR to cromwell.out (controlled by --cromwell-stdout).
Notes for devs
Server/run example: ```python c = CaperRunner( locallocdir='/scratch/me/tmplocdir', localoutdir='/scratch/me/out', default_backend='Local')
get server thread
th = c.server(port=8000)
do something
while th.returncode is None: break
stop the server
th.stop()
wait
th.join()
run example
metadatadict = c.run('my.wdl', inputs='myinput.json', ...) ```
Client example ```python cs = CaperClientSubmit(hostname='localhost', port=8000)
r = c.submit('my.wdl', inputs='myinputs.json', imports='myimports.zip', ...) workflow_id = r['id']
for m in c.metadata([workflowid], embedsubworkflow=True): # m = metadata dict embedded with subworkflows' metadata JSON print(m['status']) ```
How to read from conffile. ```python from caper.caperargs import getparserand_defaults
get both argparse.ArgumentParser (defaults updated with contents in conf_file)
and confdict including key/value pairs in conffile.
such value is converted to a correct type (guessed from ArgumentParser's defaults).
parser, confdict = getparseranddefaults(conf_file='~/.caper/default.conf')
serverportfromconf = confdict['port'] ```
- Python
Published by leepc12 about 6 years ago
caper - v0.8.2.1
Bug fix for importing autouri.
Traceback (most recent call last):
File "/opt/circleci/.pyenv/versions/3.7.0/bin/caper", line 11, in <module>
from caper.caper import main
File "/opt/circleci/.pyenv/versions/3.7.0/lib/python3.7/site-packages/caper/caper.py", line 32, in <module>
from autouri import logger as autouri_logger
ImportError: cannot import name 'logger' from 'autouri'
- Python
Published by leepc12 over 6 years ago
caper - v0.8.2
IMPORTANT: Users need to re-install Caper then it will automatically install latest bug-fixed
pyhocon. Or manually upgradepyhoconto >= 0.3.53.bash $ pip install pyhocon==0.3.54If you want to keep using old Caper versions (<0.8.2), then downgradepyhoconto < 0.3.53.
Bug fixes
- For Singularity users: Can follow symlinks recursively in input JSON to get SINGULARITY_BINDPATH.
- For cluster users (SLURM, SGE, PBS, ...): Upgraded pyhocon >= 0.3.53 to fix a bug for parsing escaped characters (e.g. \\ and \").
- Python
Published by leepc12 over 6 years ago
caper - v0.8.0
Parameters
Deprecated parameters:
- --use-netrc: Autouri defaults to use ~/.netrc.
- --http-user and --http-password: Use ~/.netrc to access private URLs
Change of parameters:
- --use-gsutil-over-aws-s3 -> --use-gsutil-for-s3: Autouri uses gsutil CLI only for direct transfer between S3 and GCS buckets. Otherwise, it always use Python libraries like google-cloud-storage and boto3.
Added parameters:
- --debug and --verbose: For better logging.
New features
Localization and preventing repetitive file transfer
- When a new localization module makes a copy of source on destination cache directory, it compares md5 hash of source and destination if a file already exists on destination. All bucket URIs (s3://, gs://) and most of URLs provide md5 hash information in their headers. If md5 hash of those match, Caper skips unnecessary file transfer. For local paths, Caper calculate md5 hash of them and store md5 hash string in .md5 file since md5 hash calculation is expensive. This happens only when Caper writes on a local storage (i.e. when localizing files on local cache). .md5 file is not valid if its modification time (mtime) is older than the file itself.
- If md5sum comparison fails, Caper compares file sizes and mtimes instead. If file sizes match and mtime is newer for destination then Caper skips file transfer.
File locking - Caper uses a stable file locking tested up to multiple threads (50 for local, 10 for cloud URIs) competing to write on the same file.
Automatic subworkflow zipping
- Fixed bugs in old auto-zipping module.
- Caper can automatically zip subworkflow WDLs imported in the main WDL. It can also be manullay defined by users in command line arguments --imports. Caper will skip auto-zipping if --imports is defined.
- Enabled for caper submit only. i.e. caper run does not use any automatic subworkflow zipping since it is assumed that all sub-WDLs are already localized for caper run.
Womtool validation
- If --imports is defined or there is an auto-zipped subworkflow WDLs, then Caper creates a temporary directory and put the main WDL and unpack the zip file there. And then Caper runs Womtool to validate those WDLs.
- You can still skip Womtool validation with --ignore-womtool.
- Python
Published by leepc12 over 6 years ago
caper - v0.7.0
New features
caper initdownloads and Cromwell/Womtool JARs and adds them to Caper's default conf file~/.caper/default.conf(or whatever defined withcaper -c) so that Caper can work completely offline once those JARs are installed.- Caper made a copy of outputs on every re-ran workflows (tasks) on GCP. Added
--gcp-call-caching-dup-stratto control this behavior. It defaults back toreferenceinstead ofcopy. Define--gcp-call-caching-dup-strat copyto keep making copies on re-ran (call-cached) tasks. - Caper can soft-link globbed outputs instead of hard-linking them. This is useful on file systems where hard-linking is not allowed (e.g. beeGFS). Added a flag
--soft-glob-outputfor local backends (local,slurm,sgeandpbs). This flag cannot work with docker (with--docker) or docker-based backends (gcpandaws).
Documentation
- Heartbeat file and how to run multiple
caper serveron a single machine. - How to configure Caper for a custom backend.
- Important notes for storage choices on Sherlock cluster.
Bug fixes
metadata.jsonin output directory/bucket is updated correctly while running and after being done.caper listsent too many requests to get label of all workflows. Now it sends a single query to retrieve all information of workflows.
- Python
Published by leepc12 over 6 years ago
caper - v0.6.4
Improved job submission for SLURM backend (Sherlock, SCG, ...)
- Fix for the following submission error when server is busy. Caper can try sbatching up to 3 times.
sbatch: error: Batch job submission failed: Socket timed out on send/recv operation
Added warning for Stanford Sherlock platform (SLURM backend)
- Do not install Caper, Conda and any executable on $OAK or $SCRATCH. Install them on $HOME or $PI_HOME.
Bug fixes
- Fix for w['submission'] error.
- Python
Published by leepc12 over 6 years ago
caper - v0.6.0
IMPORATNT: Caper defaults back to NOT use a file-based metadata DB, which means no call-caching (re-using outputs from previous workflows) by default.
IMPORATNT: Even if you still want to use a file-based DB (--db file and --file-db [DB_PATH]), metadata DB generated from Caper<0.6 (with Cromwell-42) is not compatible with metadata DB generated from Caper>=0.6 (with Cromwell-47). Refer to this doc for such migration.
See this for details about metadata DB. Define a DB type with db= in your conf ~/.caper/default.conf to use a metadata DB.
Engine update - Upgraded default crowmell JAR version: 42 -> 47 - Some feature of Caper will only work with 47 (e.g. PostgreSQL support, some bug fixes).
Then how to choose a DB?
- You can choose a DB type with --db (or db= in conf file ~/.caper/default.conf). Then define chosen DB's required parameters (nothing required for in-memory DB).
- Choices: file (unstable), mysql (recommended), postgresql and in-memory (new default but no call-caching).
- mysql is recommended. We provide shell scripts (run_mysql_server_docker.sh and run_mysql_server_singularity.sh) to run a MySQL server with docker/singularity (without root).
- See details at "Metadata database" section on README.
New features - Support for PostgreSQL DB for call-caching (Cromwell >= 43)
Change of parameters
- New:
- --db: in-memory (default), file (unstable), mysql (recommended) or postgresql (experimental).
- --mysql-db-name: (optional) cromwell by default
- --postgresql-db-ip: localhost by default
- --postgresql-db-port: 5432 by default
- --postgresql-db-user: (optional) cromwell by default
- --postgresql-db-password: (optional) cromwell by default
- --postgresql-db-name: (optional) cromwell by default
- Deprecated:
- --no-file-db: File DB is disabled by default. Many users reported that a file DB is unstable.
Bug fixes
- PAPI v10 error (preemption error) on Google Cloud.
- Caper didn't run WDL without --docker on cloud backends (aws and gcp).
- Some WDL has docker image definition in each task (runtime { docker : }). Users had to specify a dummy docker image --docker ubuntu:latest to bypass this error.
- Python
Published by leepc12 over 6 years ago
caper - v0.5.4
Validation for WDL/input JSON
- added womtool validation for WDL and input JSON
- useful to find missing/wrong parameters in input JSON
New parameters
- --womtool: womtool JAR location (URL or path). 42 by default.
- --ignore-womtool: Flag. Ignore womtool validation.
Added dict_tool.py (for new tool qc2tsv)
- useful dict functions
- merge_dict(a, b): Merge dict b into dict a.
- split_dict(): Split dict into multiple dicts according to given "splitrule" (REGEX)
- `flattendict()/unflatten_dict()`: Flatten dict with a 1-level tuple key. This tuple key keeps hierachy of original dict object in it.
- Python
Published by leepc12 over 6 years ago
caper - v0.5.3
Bug fixes
- issue #27: PBS command qstat -j -> qstat.
- Deepcopied temporary TSV/CSV/JSON files with a suffix were stored on the original storage (directory), which resulted in an error if users don't have write permission on it. Deepcopied TSV/CSV/JSON files are now stored locally and then transferred to target storage's tmp directory.
- Python
Published by leepc12 over 6 years ago
caper - v0.5.0 (not working)
Can convert gs://, s3:// URIs to presigned URLs.
- Also can convert local path into a URL.
- This is an update for Croo to be able to generate presigned URLs for UCSC genome browser tracks
- Python
Published by leepc12 over 6 years ago
caper - v0.4.0
1) Deprecated old flags --use-docker and --use-singularity.
--docker and --singularity can work as flags too.
bash
$ caper run atac.wdl --docker # this will use docker image defined in WDL
$ caper run atac.wdl --docker docker_image_location
2) caper init
Choose a platform and caper will auto-generate default configuration file (~/.caper/default.conf) with only required parameters for a chosen platform.
bash
$ caper init sherlock
3) Others - A file DB is now compressed. - Better error checking and troubleshooting.
- Python
Published by leepc12 almost 7 years ago
caper - v0.3.17
IMPORTANT:
--deepcopyhas been deprecated. Deepcopy is enabled by default and can be disabled with--no-deepcopy.--use-singularityand--use-dockerare deprecated. Instead,--singularityand--dockercan work as flags.
Deprecated parameters
--deepcopy: Deepcopy is enable by default.--no-deepcopyis added to disable it.--use-docker:--dockercan work as a flag--use-singularity:--singularitycan work as a flag
New parameters
--no-deepcopy: to disable deepcopy--dry-run: Caper creates all temporary files but does not take any action.
New subcommand
debug: an alias oftroubleshoot, which is too long to type.
Bug fixes
- Fixed all deepcopy-related bugs
- Python
Published by leepc12 almost 7 years ago
caper - v0.3.14
- added
--db-timeoutto give longer time for Cromwell to connect to DB.- to prevent
db - Connection is not availableerror even when no Caper/Cromwell process is attached to the DB file - this error can happen when DB file is large
- default:
10000(ms)
- to prevent
- Python
Published by leepc12 about 7 years ago
caper - v0.3.10
- bug fix for
caper list(non-hashablelisterror) - update doc for ENCODE workshop 2019 at Seattle
caper metadataprints out Python dict object{ }instead of list of dict[ { } ]objects when there is only one workflow matched by search condition. Otherwise It keeps printing out lists of objects.
- Python
Published by leepc12 about 7 years ago
caper - v0.3.8
- added short parameters
-dfor--file-dband-nfor--no-file-db - fixed bug for
--use-netrcnot working withsubmitcommand - added
Minimum required parameterssection to default conf file so that users only need to edit small number of parameters in that section. for TL;DR users. cromwell.jarwill be downloaded on~/.caper/cromwell_jar/instead of a directory specified with--tmp-dir
- Python
Published by leepc12 about 7 years ago
caper - v0.3.6
Added parameters
--java-heap-server: Java heap sizejava -Xmxforcaper serversubcommand.--java-heap-run: Java heap sizejava -Xmxforcaper runsubcommand.
Bug fixes
--dockernot correctly caught when it's already defined in WDL- added warning when
--dockeris not defined for cloud backends
- Python
Published by leepc12 about 7 years ago
caper - v0.3.4
- final testing on AWS (since isolation issue for aliased tasks is fixed in cromwell-42)
- chip-seq-pipeline worked fine
- documentation for setting up cromwell server on AWS
- fixed bug in
file_exists()for AWSaws s3 lsdoes not show exactly matched files. it also shows partially matched files.
- Python
Published by leepc12 about 7 years ago
caper - v0.3.2
- added
usertocaper listformat - users can filter workflows by submission datetime
--hide-result-before( orhide-result-beforein your conf at~/.caper/default.conf)- you can use the same (or shorter in terms of string length) datatime format shown in
caper list. - this is just about simple datetime string sorting
- you can use the same (or shorter in terms of string length) datatime format shown in
For example,
(base) leepc12@kadru:/users/leepc12/code/test_wdl$ caper list
id status name str_label user submission
648d61d1-a6df-4d1d-8050-2b7fab0b1d7c On Hold None T5 leepc12 2019-06-13T11:12:25.661Z
afd8b507-928b-4ae9-9504-a584ce84442b On Hold None T4 leepc12 2019-06-13T11:12:21.917Z
05fd8f2b-da72-46c8-8e88-848bc4f0f5ea On Hold None T3 leepc12 2019-06-13T11:12:16.349Z
aa08ca19-0ac8-4ec6-995d-53bcfaaa1038 On Hold None T2 leepc12 2019-06-13T11:12:10.611Z
a6573ee9-5e08-432e-bf7f-5733069b7510 On Hold None T1 leepc12 2019-06-13T11:11:51.057Z
(base) leepc12@kadru:/users/leepc12/code/test_wdl$ caper list --hide-result-before 2019-06-13T11:12:16
id status name str_label user submission
648d61d1-a6df-4d1d-8050-2b7fab0b1d7c On Hold None T5 leepc12 2019-06-13T11:12:25.661Z
afd8b507-928b-4ae9-9504-a584ce84442b On Hold None T4 leepc12 2019-06-13T11:12:21.917Z
05fd8f2b-da72-46c8-8e88-848bc4f0f5ea On Hold None T3 leepc12 2019-06-13T11:12:16.349Z
(base) leepc12@kadru:/users/leepc12/code/test_wdl$ caper list --hide-result-before 2019-06-13
id status name str_label user submission
648d61d1-a6df-4d1d-8050-2b7fab0b1d7c On Hold None T5 leepc12 2019-06-13T11:12:25.661Z
afd8b507-928b-4ae9-9504-a584ce84442b On Hold None T4 leepc12 2019-06-13T11:12:21.917Z
05fd8f2b-da72-46c8-8e88-848bc4f0f5ea On Hold None T3 leepc12 2019-06-13T11:12:16.349Z
aa08ca19-0ac8-4ec6-995d-53bcfaaa1038 On Hold None T2 leepc12 2019-06-13T11:12:10.611Z
a6573ee9-5e08-432e-bf7f-5733069b7510 On Hold None T1 leepc12 2019-06-13T11:11:51.057Z
(base) leepc12@kadru:/users/leepc12/code/test_wdl$ caper list --hide-result-before 2019-06-14
id status name str_label user submission
- Python
Published by leepc12 about 7 years ago
caper - v0.3.1
- Fix for SLURM Stanford Sherlock job failing issue even though Job is actually done successfully.
squeuedoesn't seem to work correctly.- Job check-alive polling (
squeue -j {job_id}) somtimes fail when server is busy - Try
squeuefor every 20 second up to 3 times. - increase interval for polling 180 sec -> 360 sec
- Job check-alive polling (
- Python
Published by leepc12 about 7 years ago
caper - v0.3.0
- thread safe inter-storage file transfer
- added a simple spin lock (with
.lockfile) to obtain thread safety for file transfer - multiple users can share the same
tmp-dir,tmp-s3-bucketandtmp-gs-bucket. - limitation
- possible race condition
- skip copying only if file size of source and target files are same (does not check md5sum)
- source files are not locked to prevent deadlock
- make sure that source files are almost like read-only
- added a simple spin lock (with
- Python
Published by leepc12 about 7 years ago
caper - v0.2.8
- Added
--max-retries(default:1) to retry failing tasks - Replaced deprecated
actor factoryfor google backend - Cromwell server submits jobs slowly (1 per 2 second) due to possible socket error for SLURM/other cluster engines.
- Bug fix for
awsbackend (typo in code: outawsbucket->outs3bucket) - Upgrade cromwell 40 to 42 (to fix aliased task isolation issue on aws)
- Python
Published by leepc12 about 7 years ago
caper - v0.2.1.1
Bug fixes
--out-dirdidn't work with slurm, sge, pbs backends--singularity-cachedirdidn't work correctly ifSINGULARITY_CACHEDIRenv var is already defined.
added
--no-build-singularityto prevent building singularity image before running a pipeline.- but this can lead to conflicts between tasks trying to build their own local singularity image on the same target location.
- Python
Published by leepc12 about 7 years ago
caper - v0.2.0
bug fix for singularity
- added
--singularity-cachedirto prevent repeatedly building image for every WDL task. - automatically calculate
SINGULARITY_BINDPATHbased on files in input JSON file- by finding common roots of those files
- added
CaperURI
- more robust method to catch HTTPERRORCODE from cURL.
- Python
Published by leepc12 about 7 years ago
caper - v0.1.3
- bug fix
- caper_uri: typo in URL -> S3 transfer command
- URL to cloud transfer is two-step
- URL to local and then local to cloud (gcs, s3)
- use call-caching by default
- can use both _ and - in a configuration file
- added
-mto specify metadata JSON file destination - fix Java heap memory for
run(1G) andserver(2G)
- Python
Published by leepc12 about 7 years ago
caper - v0.1.2.3
Caper can be PIP-installed.
$ pip install caper
Three executables will be installed. caper, run_mysql_server_singularity.sh and run_mysql_server_docker.sh.
``` $ caper /home/ec2-user/miniconda3/lib/python3.7/site-packages/caper/caper.py usage: caper [-h] [-c FILE] {run,server,submit,abort,unhold,list,metadata} ...
positional arguments: {run,server,submit,abort,unhold,list,metadata} run Run a single workflow without server server Run a Cromwell server submit Submit a workflow to a Cromwell server abort Abort running/pending workflows on a Cromwell server unhold Release hold of workflows on a Cromwell server list List running/pending workflows on a Cromwell server metadata Retrieve metadata JSON for workflows from a Cromwell server
optional arguments: -h, --help show this help message and exit -c FILE, --conf FILE Specify config file (base) [ec2-user@ip-172-30-0-159 ~]$ runmysqlserver_ runmysqlserverdocker.sh runmysqlserversingularity.sh ```
``` $ runmysqlserverdocker.sh Usage: ./runmysqlserverdocker.sh [DB_DIR] [PORT] [MYSQL_USER] [MYSQLPASSWORD] [CONTAINERNAME]
Example: runmysqlserverdocker.sh ~/cromwelldata_dir 3307
[DBDIR]: This directory will be mapped to '/var/lib/mysql' inside a container PORT: MySQL database port for docker host (default: 3306) MYSQL_USER: MySQL username (default: cromwell) MYSQL_PASSWORD: MySQL password (default: cromwell) CONTAINER_NAME: MySQL container name (default: mysqlcromwell) ```
``` $ runmysqlserversingularity.sh Usage: ./runmysqlserversingularity.sh [DB_DIR] [PORT] [MYSQL_USER] [MYSQLPASSWORD] [CONTAINERNAME]
Example: runmysqlserversingularity.sh ~/cromwelldata_dir 3307
[DBDIR]: This directory will be mapped to '/var/lib/mysql' inside a container PORT: MySQL database port for singularity host (default: 3306) MYSQL_USER: MySQL username (default: cromwell) MYSQL_PASSWORD: MySQL password (default: cromwell) CONTAINER_NAME: MySQL container name (default: mysqlcromwell) ```
- Python
Published by leepc12 about 7 years ago
caper - v0.1.1
- added shell scripts to
/mysql/to start a MySQL server with Docker or Singularity. - supports soft linking for local-to-local file transfer.
- explicitly show
symlinkinginstead ofcopyingif it makes a symlink. - for
runmode, build singularity image before running a pipeline (local backend only).- fix for multiple tasks trying to build on the same image when there is no pre-built image.
- Python
Published by leepc12 about 7 years ago