Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of signac-flow

signac-flow - v0.29.0

[0.29.0] -- 2024-03-26

Added

  • Added memory jinja2 block around memory options to override in custom SLURM scripts (#831).

Fixed

  • Always write the nodes parameter on Expanse SBATCH scripts (#827).
  • Ensure that a bundle exists in a project before yielding scheduler jobs from it (#758, #832).

- Python
Published by cbkerr almost 2 years ago

signac-flow - v0.28.0

Version 0.28

Added

  • Support the gpu_mig40 partition on UMich Great Lakes (#811).

Changed

  • Move "Submit command" comment to end of pretend output (#805).
  • Improve aggregate registration performance (#818).
  • Hide empty "Labels" section in status output when there are no labels (#818).

Fixed

  • Generate correct SLURM scripts on SDSC Expanse (#810).
  • Suppress warning for regex operation names that match groups (#804).

Removed

  • Remove Zenodo (#816).
  • Remove stampede2 (#786).

- Python
Published by cbkerr about 2 years ago

signac-flow - Version 0.27.0

Version 0.27.0

Added

  • Official Python 3.12 support (#788).
  • Purdue Anvil environment and template (#775).

Changed

  • The Stampede2 environment is deprecated (#798).
  • Deprecate the current keys for submission/execution directives (#792).
  • Show the "Using environment configuration:" message only in verbose output (#776).

Fixed

  • srun invocation on Great-Lakes (#779).
  • Restored static output of non-singleton groups (#773, #774).
  • Remove warnings by using new jsonschema API (#787).

- Python
Published by b-butler about 2 years ago

signac-flow - Version 0.26.1

Version 0.26.1

2023-09-19

Fixed

  • Packaging of files in flow.scripts used in CLI (#765).
  • Schedule 56 cores per node on OLCF Frontier (#764).

- Python
Published by b-butler over 2 years ago

signac-flow - Version 0.26.0

Version 0.26.0

2023-09-06

Added

  • Added feature to hide the status of the progress bar (#685).
  • test-workflow CLI option for testing template environments/submission scripts (#747).
  • Frontier environment and template (#743).
  • Added -o / --operation flag to report project status information for specific operations (#725).

Changed

  • Removed -o flag as an alias for --output in status CLI subcommand (#725).

- Python
Published by b-butler over 2 years ago

signac-flow - Version 0.25.1

Version 0.25.1

2023-04-21

Fixed

  • Delta hostname detection (#740).
  • Fix filter argument (#737, #738).

Removed

  • --doc-filter argument (#738).

- Python
Published by b-butler almost 3 years ago

signac-flow - v0.25.0

Version 0.25

2023-03-30

Changed

  • Requires signac 2.0.0 (#734).

- Python
Published by b-butler almost 3 years ago

signac-flow - v0.24.0

Version 0.24.0

2023-02-22

Added

  • Added the OCLF Crusher environment (#708).

Changed

  • Switched to a pyproject.toml based build (#704).
  • Placing @FlowProject.pre and @FlowProject.post before the FlowProject.operation decorator raises an error (#700).
  • Updated CI to use GitHub Actions (#698).

Fixed

  • Bumped isort to 5.12.0 in the pre-commit configuration file (#715).
  • Fixed bundle status information not being correctly read for subclasses of FlowProject (#718, #720).

Removed

  • @flow.aggregates.aggregator() use FlowProject.operation(aggregator=...) instead (#700).
  • alias CLI argument to flow init (#700).
  • FlowGroupEntry.with_directives use directives keyword argument in FlowGroupEntry()(#700).

- Python
Published by b-butler about 3 years ago

signac-flow - v0.23.0

Version 0.23.0

2022-12-09

Added

  • Official Python 3.11 support (#697).
  • The flow.FlowProject.operation decorator now has an aggregator keyword argument: @FlowProject.operation(aggregator=aggregator.groupsof(2)) (#681).
  • The FlowGroupEntry class can be called with a directives keyword argument: FlowGroupEntry(directives={...}) (#696).

Changed

  • Deprecated using flow.aggregate.aggregator as a decorator.
  • Deprecated placing @FlowProject.pre and @FlowProject.post before the FlowProject.operation decorator (#690).
  • Require signac version 1.8.0 (#693).
  • Deprecated alias CLI argument to flow init (#693).
  • Algorithm for computing cluster job ids (#695).
  • Deprecated FlowGroupEntry.with_directives in favor of a directives keyword argument in FlowGroupEntry()(#696).

Fixed

  • Detecting correct environment on Delta GPU nodes (#682).
  • Identical aggregates are used only once in submission and running (#694, #688).

Removed

  • show_traceback from CLI and config (#690).
  • Formatting the output of a FlowCmdOperation (#686).
  • @flow.cms and flow.with_job (#686, #669).
  • @FlowProject.operation.with_directives (#686).
  • The flow.testing module (#691, #692).

- Python
Published by b-butler about 3 years ago

signac-flow - v0.22.0

Version 0.22

[0.22.0] -- 2022-10-14

Added

  • Support for formatting with operation function arguments for FlowCmdOperation (#666, #630).
  • The CLI status command can show document parameters by using flag -p doc.PARAM (#667).
  • FlowProject.operation now has cmd, with_job, and directives keyword only arguments (#679, #655, #669).

Changed

  • Deprecated formatting the output of a FlowCmdOperation (#666, #630).
  • @flow.cmd and flow.with_job are deprecated (#679, #669, #665).
  • @FlowProject.operation.with_directives is deprecated (#679, #665).
  • Deprecated the --show-traceback option for flow's CLI run and submitcommands (#674, #169).
  • flow CLI run and submit show tracebacks by default (#674, #169).
  • Broke TestBidict and TestTemplateFilters into smaller and simpler functions (#626).

- Python
Published by b-butler over 3 years ago

signac-flow - v0.21.0

Version 0.21

[0.21.0] -- 2022-08-18

Added

  • XSEDE Delta environment and template (#658).

Changed

  • Changed get_config_value template filter to error without default on missing key (#649).
  • Changed get_config_value template filter now takes a FlowProject as its first argument (#649).

Removed

  • Removed require_config_value template filter (#649).
  • Removed configuration key 'flow.importpackagedenvironments' (#653).
  • Removed configuaration key 'flow.environment_modules' (#651).

- Python
Published by b-butler over 3 years ago

signac-flow - v0.20.0

Version 0.20

[0.20.0] -- 2022-06-23

Added

  • Added support to run aggregate operations in parallel (#642, #644).
  • Added an argument, run_options, to FlowProject.make_group which allows passing options to exec for operations running in a different process (#650).

Changed

  • Deprecated configuaration key 'flow.environment_modules' (#651).
  • Deprecated configuration key 'flow.importpackagedenvironments' (#651).
  • Changed argument options to submit_options in FlowProject.make_group (#650).

Removed

  • Dropped support for cloudpickle versions below 1.6.0 (#644).
  • Removed upper bound on python_requires (#654).

- Python
Published by b-butler over 3 years ago

signac-flow - v0.19.0

Version 0.19

This release changes the names of hook triggers and improves the behavior of progress bars in Jupyter notebooks. The minimum supported Python version is now 3.8. We also welcome the first released contributions from @rohanbabbar04!

[0.19.0] -- 2022-04-07

Changed

  • Dropped support for tqdm versions older than 4.60.0 (#614).
  • Renamed hook triggers on_finish to on_exit and on_fail to on_exception (#612, #627).

Fixed

  • Progress bars shown in notebooks fall back to text-based output if ipywidgets is not available (#602, #614).

Removed

  • Internal utility functions have been removed from the public API (#615).
  • Dropped support for Python 3.6 and Python 3.7 following the recommended support schedules of NEP 29 (#628).
  • Dropped support for Jinja2 versions below 3.0.0 (#628).

- Python
Published by cbkerr almost 4 years ago

signac-flow - v0.18.1

Version 0.18.1

Hey signac users, we fixed some bugs to make signac-flow work better for you. Happy Valentine's Day from the signac team! 🌹

[0.18.1] -- 2022-02-14

Fixed

  • Fixed bug in project status output when no operations are eligible (#607, #609).
  • Improved traceback handling for errors in signac-flow (#608).

- Python
Published by kidrahahjo about 4 years ago

signac-flow - v0.18.0

Version 0.18

[0.18.0] -- 2022-02-03

Added

  • Feature to install execution hooks for the automated execution of functions with operations (#28, #189, #508).

Changed

  • Add user defined FlowGroups to status output (#547, #593).
  • Raise UserOperationError for failed execution of FlowProject operations (#571, #601).

Fixed

  • Fix issue with GPU submission on Bridges-2 (#588, #589).

- Python
Published by cbkerr about 4 years ago

signac-flow - v0.17.0

This release adds Python 3.10 support and addresses a couple bugs. Thanks to all the contributors in this release! :art:

Version 0.17

[0.17.0] -- 2021-11-15

Added

  • Add official support for Python version 3.10 (#578).

Fixed

  • Scripts are now generated correctly when project path contains spaces and special characters (#572).
  • XSEDE Expanse template has been fixed to remove leading spaces (#575, #576).

Changed

  • FlowProject configuration is now validated independently of signac Project configuration (#573).
  • jsonschema is now a dependency (#573).

- Python
Published by b-butler over 4 years ago

signac-flow - v0.16.0

This release adds support for the XSEDE Expanse Cluster, simplifies user customization of templates, improves documentation, and fixes minor bugs. Thanks to everyone who contributed :art:

Added

  • The --job-output command line flag for submission can be set for SLURM, PBS, and LSF schedulers (#550).
  • Added a custom_content block to templates for user customization (#553, #520).
  • Added official support for XSEDE Expanse cluster (#537).
  • Added FlowProjectDefinitionError for exceptions involving FlowProject definitions (#549).

Changed

  • Improved documentation of directives (#554).
  • Raise FlowProjectDefinitionError error for inaccurate FlowProject definitions (#549).

Removed

  • Removed deprecated environment classes (#556).
  • Removed support for signac < 1.3.0 (#558).
  • Removed support for decommissioned XSEDE Comet cluster (#537).
  • Removed --memory option from University of Michigan Great Lakes cluster submission. Use directives instead (#563).

- Python
Published by kidrahahjo over 4 years ago

signac-flow - v0.15.0

This release adds the ability for operations to operate on subset of jobs known as aggregates and improves submissions to HPC clusters. Thanks to @klywang as a first-time contributor for this release. :art:

Added

  • Add support for aggregation (operations acting on multiple jobs) via flow.aggregator (#464, #516, #542).
  • Add official support for Andes cluster (#500).
  • Decorator for setting directives while registering operation function FlowProject.operation.with_directives (#309, #502).
  • Add new flow command flow template create for automatic creation of custom templates (#520, #534).

Changed

  • Jinja templates are indented for easier reading (#461, #495).
  • flow.directives is deprecated in favor of flow.FlowProject.operation.with_directives (#309, #502).
  • All environments require a scheduler in order to submit, even in pretend mode (#533).
  • Submitting in pretend mode will show additional scheduler command information (#533).
  • Support fractional timeout value in Python and command line interface (#541).

Fixed

  • Errors raised during submission were not being shown to users (#517, #518).
  • Fixed dependency flag for SLURM submissions (#530, #531).

- Python
Published by kidrahahjo over 4 years ago

signac-flow - 0.14.0

This release improvements to status output, documentation on directives, and fixed regressions in version 0.13. Thanks to @Charlottez112 as a first time contributor and the other 4 people who contributed code for this release. :art:

Added

  • Documentation for all directives (#480).
  • Defined validators for the fork directive (#480).
  • Submission summary now appears in FlowProject status output, showing the number of queued, running, unknown statuses. (#472, #488).
  • Status overview now shows the number of jobs with incomplete operations and totals for the label overviews (#481, #501).

Changed

  • Renamed TorqueEnvironment and related classes to PBSEnvironment (#388, #491).
  • LSF and SLURM schedulers will appear to be present if the respective commands bjobs -V or sbatch --version exit with a non-zero error code (#498).
  • Only known JobStatus values will be written to the project document, to save space and writing time (#503).

Fixed

  • Strictly enforce that operation functions cannot be used as condition functions (and vice-versa) and prevent the registration of two operations with the same name (#496).
  • Changed default value of status_parallelization to none, to avoid bugs in user code caused by thread parallelism and overhead in process parallelism (#486).
  • Memory directives are converted to an integer number of gigabytes or megabytes in submission scripts (#482, #484).
  • Fixed behavior of --only-incomplete-operations (#481, #501).

Removed

  • Removed FlowProject.add_operation (#479, #487).
  • Removed deprecated --walltime argument (#478).
  • Removed deprecated flow.run interface (#478).
  • Removed deprecated FlowProject.export_job_statuses (#478).
  • Removed deprecated script feature, use submit --pretend instead (#478).
  • Removed deprecated CPUEnvironment, GPUEnvironment classes (#478).

- Python
Published by b-butler almost 5 years ago

signac-flow - 0.13.0

This release adds support for the new Bridges-2 cluster, expands the use of directives to include memory and walltime requests, removes deprecated features, and fixes bugs. Thanks to the 7 people who contributed code to this release, including first-time contributors @berceanu and @adgnabro!

Added

  • Add official support for Bridges-2 cluster (#441).
  • Add support for memory requests via directives (#258, #466).
  • Add support for walltime requests via directives, deprecated --walltime argument to submit (#240, #476).

Fixed

  • Support for multi-line @flow.cmd operations (#451, #453).
  • FlowProject status shows labels and correct number of jobs for projects with zero operations (#454, #460).

Removed

  • Removed public API of deprecated class JobOperation (#445).
  • Removed public API of deprecated methods eligible and complete of BaseFlowOperation and FlowGroup (#445).
  • Removed configuration option use_buffered_mode (#445).
  • Removed public API of script, next_operations and submit_operations of FlowProject (#445).
  • Removed support for decommissioned Bridges cluster (#441).
  • Removed support for memory command line argument in submit (#466).

- Python
Published by bdice almost 5 years ago

signac-flow - 0.12.0

This release includes a wide range of performance improvements and internal refactoring that will enable the addition of an "aggregation" feature in subsequent releases (not yet available). Performance of a sample workflow that checks status, runs, and submits a FlowProject with 1000 jobs, 3 operations, and 2 label functions has improved roughly 4x compared to the 0.11.0 release.

Added

  • Code is formatted with black and isort pre-commit hooks (#365).
  • Add official support for Python version 3.9 (#365).
  • Documentation has been added for all public classes and methods (#387, #389).
  • Added internal support for aggregates of jobs (#334, #348, #351, #364, #383, #390, #415, #422, #430).
  • Added code coverage to continuous integration (#405).

Changed

  • Command line interface always uses --job-id instead of --jobid (#363, #386).
  • CPUEnvironment and GPUEnvironment classes are deprecated (#381).
  • Docstrings are now written in numpydoc style (#392).
  • Default environment for the University of Minnesota Mangi cluster changed from SLURM to Torque (#393).
  • Run commands are evaluted lazily (#70, #396).
  • Deprecated method export_job_statuses (#402).
  • Improved internal caching of scheduler status (#410).
  • Refactored status fetching code (#368, #417).
  • Optimization: Directives are no longer deep-copied (#420, #421).
  • The use_buffered_mode config option is deprecated. Buffering is always internally enabled (#425).
  • Evaluate directives when called instead of when defined (#398, #402).
  • Various internal refactorings and optimizations (#371, #373, #374, #375, #376, #377, #378, #379, #380, #400, #410, #416, #423, #426).
  • Scheduler is now an abstract base class (#426).
  • flow.scheduling.fakescheduler has been renamed to flow.scheduling.fake_scheduler (#426).
  • Arguments to submit have been changed for all scheduler classes (#426).
  • Python 3.6 is only tested with oldest dependencies (#436).
  • Drop support for tqdm versions older than 4.48.1 (#436, #440).
  • Drop support for Jinja2 versions older than 2.10.0 (#436).

Fixed

  • Ensure that directives are always evaluated before running or submitting (#408, #409).
  • Cache the fully qualified domain name during environment detection to fix a performance issue on macOS (#339, #394).
  • Ensure that next CLI command displays eligible jobs for the exact operation name provided (#443).

Removed

  • Removed the deprecated method flow.util.misc.write_human_readable_statepoints (#397).
  • Removed the deprecated argument --no-parallelize (#424).
  • Removed the deprecated env argument from submission methods (#424).
  • flow.render_status.Renderer class has been removed. FlowProject.print_status no longer returns the renderer (#426).
  • Removed deprecated status.py module (#426).
  • Removed the --test argument from FlowProject.submit (#439).

- Python
Published by bdice about 5 years ago

signac-flow - 0.11.0

Added

  • Added classes _Directives and _Directive that serve as a smart mapping for directives specified by the environment or user (#265, #283).
  • Added support for pre-commit hooks (#333).
  • Add environment profile for University of Minnesota, Minnesota Supercomputing Institute, Mangi supercomputer (#353).

Changed

  • Make FlowCondition class private (#307, #315).
  • Deprecate JobOperation class, make SubmissionJobOperation a private class and deprecate the following methods of FlowProject: script, run_operations, submit_operations, next_operations. (#313)
  • Deprecate the following methods: FlowGroup.eligible, FlowGroup.complete, BaseFlowOperation.eligible, BaseFlowOperation.complete (#337).

Fixed

  • Serial execution on Summit correctly counts total node requirements (#342).
  • Fixed performance regression in job submission in large workspaces (#354).

Removed

  • Drop support for Python 3.5 (#305). The signac project will follow the NEP 29 deprecation policy going forward.
  • Remove the deprecated methods always, make_bundles, and JobOperation.get_id (#312).

- Python
Published by bdice over 5 years ago

signac-flow - 0.10.1

Fixed

  • Fix issue with the submission of bundled operations on cluster environments that do not allow slashes ('/') in cluster scheduler job names (#343).

- Python
Published by csadorf over 5 years ago

signac-flow - 0.10.0

Added

  • Add FlowGroup (one or more operations can be grouped within an execution environment) (#114).
  • Add official support for University of Michigan Great Lakes cluster (#185).
  • Add official support for Bridges AI cluster (#222).
  • Add IgnoreConditions option for submit(), run() and script() (#38).
  • Add pytest support for Testing Framework (#227, #232).
  • Add markdown and html format support for print_status() (#113, #163).
  • Add memory flag option for default Slurm scheduler (#256).
  • Add optional environment variable to specify submission script separator (#262).
  • Add status_parallelization configuration to specify the parallelization used for fetching status (#264, #271).

Changed

  • Raises ValueError when an operation function is passed to FlowProject.pre() and FlowProject.post(), or a non-operation function passed to FlowProject.pre.after() (#248, #249).
  • The option to provide the env argument to submit and submit_operations has been deprecated (#245).
  • The command line option --cmd for script has been deprecated and will trigger a DeprecationWarning upon use until removed (#243, #218).
  • Raises ValueError when --job-name is passed by the user because that interferes with status checking (#164, #241).
  • Submitting with --memory no longer assumes a unit of gigabytes on Bridges and Comet clusters (#257).
  • Buffering is enabled by default, improving the performance of status checks (#273).
  • Deprecate the use of no_parallelize argument while printing status (#264, #271).
  • Submission via the command-line interface now calls the FlowProject.submit function instead of bypassing it for FlowProject.submit_operations (#238, #286).
  • Updated Great Lakes GPU request syntax (#299).

Fixed

  • Ensure that label names are used when displaying status (#263).
  • Fix node counting for large resource sets on Summit (#294).

Removed

  • Removed ENVIRONMENT global variable in the flow.environment module (#245).
  • Removed vendored tqdm module and replaced it with a requirement (#247).

- Python
Published by bdice over 5 years ago

signac-flow - 0.9.0

Added

  • Add official support for Python version 3.8 (#190, #210).
  • Add descriptive error message when tag is not set and cannot be autogenerated for conditions (#195).
  • Add "fork" directive to enforce the execution of operations within a subprocess (#159).
  • Operation graph detection based on function comparison (#178).
  • Exceptions raised during operations always show tracebacks of user code (#169, #171).

Changed

  • Raise a warning when a condition's tag is not set and raise an error if this occurs during graph detection (#195).
  • Raise errors if a forked process or @cmd operation returns a non-zero exit code. (#170, #172).

Removed

  • Drop support for Python version 2.7 (#157, #158, #201).
  • The "always" condition has been deprecated and will trigger a DeprecationWarning upon use until removed (#179).
  • Removed deprecated UnknownEnvironment in favor of StandardEnvironment (#204).
  • Removed support for decommissioned INCITE Titan and Eos computers (#204).
  • Removed support for the legacy Python-based submission script generation (#200).
  • Removed legacy compatibility layers for Python 2, signac < 1.0, and soft dependencies (#205).
  • Removed deprecated support for implied operation names with the run command (#205).

- Python
Published by bdice about 6 years ago

signac-flow - 0.8.0

Added

  • Add feature for integrated profiling of status updates (status --profile) to aid with the optimization of a FlowProject implementation (#107, #110).
  • The status view is generated with Jinja2 templates and thus more easily customizable (#67, #111).
  • Automatically show an overview of the number of eligible jobs for each operation in status view (#134).
  • Allow the provision of multiple operation-functions to the pre.after and *.copy_from conditions (#120).
  • Add option to specify the operation execution order (#121).
  • Add a testing module to easily initialize a test project (#130).
  • Enable option to always show the full traceback with show_traceback = on within the [flow] section of the signac configuration (#61, #144).
  • Add full launcher support for job submission on XSEDE Stampede2 for large parallel single processor jobs (#85, #91).

Fixes

  • Both the nranks and omp_num_threads directives properly support callables (#118).
  • Show submission error messages in combination with a TORQUE scheduler (#103, #104).
  • Fix issue that caused the "Fetching operation status" progress bar to be inaccurate (#108).
  • Fix erroneous line in the torque submission template (#126).
  • Ensure default parameter range detection in status printing succeeds for nested state points (#154).
  • Fix issue with the resource set calculation on INCITE Summit (#101).

Changed

  • Packaged environments are now available by default. Set import_packaged_environments = off within the [flow] section of the signac configuration to revert to previous behavior.

  • The following methods of the FlowProject class have been deprecated and will trigger a DeprecationWarning upon use until their removal:

    • classify (use labels() instead)
    • next_operation (use next_operations() instead)
    • export_job_stati (replaced by export_job_statuses)
    • eligible_for_submission (removed without replacement)
    • update_aliases (removed without replacement)

  • The support for Python version 2.7 is deprecated.

Removed

  • The support for Python version 3.4 has been dropped.
  • Support for signac version 0.9 has been dropped.

- Python
Published by csadorf over 6 years ago

signac-flow - 0.7.1

Added

  • Add function to automatically print all varying state point parameters in the detailed status view triggered by providing option -p/--parameters without arguments (#19, #87).
  • Add clear environment notification when submitting job scripts (#43, #88).

Fixes

  • Fix issue where the scheduler status of job-operations would not be properly updated for ineligible operations (#96).

Fixes (compute environments)

  • Fix issue with the TORQUE scheduler that occured when there was no job scheduled at all on the system (for any user) (#92, #93).

Changed

  • The performance of status updates has been significantly improved (up to a factor of 1,000 for large data spaces) by applying a more efficient caching strategy (#94).

- Python
Published by csadorf almost 7 years ago

signac-flow - 0.7.0

Added

  • Add legend explaining the scheduler-related symbols to the detailed status view (#68).
  • Allow the specification of the number of tasks per resource set and additional jsrun arguments for Summit scripts.

Fixes (general)

  • Fixes issue where callable cmd-directives were not evaluated (#47).
  • Fixes issue where the source file of wrapped functions was not determined correctly (#55).
  • Fix a Python 2.7 incompatibility and another unrelated issue with the TORQUE scheduler driver (#54, #81).
  • Fixes issue where providing the wrong argument type to Project.submit() would go undetected and lead to unexpected behavior (#58).
  • Fixes issue where using the buffered mode would lead to confusing error messages when condition-functions would raise an AttributeError exception.
  • Fixes issue with erroneous unused-directive-keys-warning.

Fixes (compute environments)

  • Fixes issues with the Summit environment resource set calculation for parallel operations under specific conditions (#63).
  • Fix the node size specified in the template for the ORNL Eos system (#77).
  • Fixes issue with a missing --gres directive when using the GPU-shared partition on the XSEDE Bridges system (#59).
  • Fixed University of Michigan Flux hostname pattern to ignore the Flux Hadoop cluster (#82).
  • Remove the Ascent environment (host decommissioned).

Note: The official support for Python 3.4 will be dropped beginning with version 0.8.0.

- Python
Published by csadorf almost 7 years ago