Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of https://github.com/fgcz/bfabricpy

https://github.com/fgcz/bfabricpy -

[0.4.0] - 2025-09-26

Added

  • New Input Type: BfabricResourceDatasetSpec
    • Specifies a dataset which links Resource entities with a particular column
    • Copies all files into a directory (the spec's filename field)
    • Writes the dataset file into the same directory, adding a File column (configurable) with the filenames

Changed

  • Automatic Workflow Step Creation moved from dispatch action to stage action
    • Prevents creating workflowsteps for failed workunits
    • Note: This decision may be revisited in the future, as there could be benefits to creating the workflowstep early on
  • Improved error messaging when running "make stage" in read-only mode
    • Now displays clear warning messages explaining that staging is skipped
    • Provides guidance on how to remove the --read-only flag

- Python
Published by leoschwarz 5 months ago

https://github.com/fgcz/bfabricpy -

[1.13.35] - 2025-09-25

Changed

  • bfabric_list_not_existing_storage_directories.py is made more robust. Instead of the file based cache, it will check all containers modified within a sliding time window (default 14 days).

- Python
Published by github-actions[bot] 5 months ago

https://github.com/fgcz/bfabricpy -

[1.13.34] - 2025-09-22

Fixed

  • bfabric_save_importresource_sample.py now properly serializes ResultContainer objects for JSON output.

Changed

  • Update bfabric to include case-insensitive dataset column type detection support.

- Python
Published by github-actions[bot] 5 months ago

https://github.com/fgcz/bfabricpy -

[1.13.35] - 2025-09-22

Changed

  • Dataset column type detection now supports case-insensitive entity matching (e.g., "resource" is detected as "Resource" type automatically).

- Python
Published by github-actions[bot] 5 months ago

https://github.com/fgcz/bfabricpy -

[1.13.34] - 2025-09-19

Added

  • bfabric.experimental.cache which implements re-entrant lookup caching for entities (when retrieved by ID).
  • Fields filename, storage_relative_path, storage_absolute_path have been added to Resource. This should be used in the future, to ensure path handling is performed consistently

Removed

  • bfabric.experimental.entity_lookup_cache has been removed in favor of the new (experimental) API.

- Python
Published by github-actions[bot] 5 months ago

https://github.com/fgcz/bfabricpy -

[0.3.1] - 2025-09-01

Added

  • CommandPythonEnv can also execute any tools available in the Python environment (in .venv/bin), not just modules.
  • -m bfabric_app_runner.commands.command_python_env provides an experimental CLI to run arbitrary commands analogously to the app runner.
  • Automatically save links to workunits without specifying extra IDs.

Changed

  • Dispatch action detects, when the script modifies workunit_definition.yml which should not happen anymore, and notifies the user restoring the original file. The app continues without an error.

- Python
Published by github-actions[bot] 6 months ago

https://github.com/fgcz/bfabricpy -

[0.3.0] - 2025-08-26

Added

  • Add workflow_template_step_id field to BfabricAppSpec to specify template step to register workunits under automatically.
  • Dispatch action checks if a workflow template step is specified and registers the workunit under it automatically.

Changed

  • Generic dispatch functionality will not override existing workunit_definition.yml files anymore.
  • Update bfabric dependency to 1.13.33.

- Python
Published by github-actions[bot] 6 months ago

https://github.com/fgcz/bfabricpy -

[1.13.33] - 2025-08-26

Added

  • Add new field WorkunitDefinition.registration.user_id.
  • Add created/modified fields to Workunit entity.

Fixed

  • Legacy wrapper creator also writes the inputdataset name.

- Python
Published by github-actions[bot] 6 months ago

https://github.com/fgcz/bfabricpy -

[1.13.33] - 2025-08-26

Changed

  • Update bfabric to 1.13.32.

- Python
Published by github-actions[bot] 6 months ago

https://github.com/fgcz/bfabricpy -

[1.13.32] - 2025-08-26

Added

  • New entities: Workflow, WorkflowStep, WorkflowTemplate, WorkflowTemplateStep
  • New relationship: HasUser to resolve user references (since these are modeled differently from other relationships).

Fixed

  • There was an issue in import_entity.py which made loading entities with additional uppercase characters in the type fail.
  • Fix compatibility in legacy wrapper creator for dataset-flow workunits.

- Python
Published by github-actions[bot] 6 months ago

https://github.com/fgcz/bfabricpy -

[1.13.32] - 2025-08-20

Fixed

  • Legacy bfabric_save_workflowstep.py is compatible with current bfabricPy version again.

Changed

  • Update bfabric to 1.13.31.

- Python
Published by github-actions[bot] 6 months ago

https://github.com/fgcz/bfabricpy -

[1.13.31] - 2025-09-20

Changed

  • The Bfabric.connect_webapp will use the caller field from the token data to set the base_url of the client.
    • Existing code should not break, but will emit a deprecation warning as we plan to remove the old parameters.
    • Theoretically, it changes the semantics, but in practice it should yield the same result for all known use cases.

Added

  • TokenData will include the caller url now.
  • Entity User has been added.

Fixed

  • Legacy wrapper creator creates "" rather than None for empty parameter values.

- Python
Published by github-actions[bot] 6 months ago

https://github.com/fgcz/bfabricpy -

[1.13.31] - 2025-08-19

Fixed

  • bfabric-cli api read handles empty results gracefully for all output formats.

Changed

  • Update bfabric to 1.13.30. This includes a fix for legacy wrapper creator.

- Python
Published by github-actions[bot] 6 months ago

https://github.com/fgcz/bfabricpy -

[1.13.30] - 2025-08-19

Fixed

  • Fix legacy wrapper creator for orders without projects.

- Python
Published by github-actions[bot] 6 months ago

https://github.com/fgcz/bfabricpy -

[0.2.1] - 2025-07-22

Fixed

  • Makefile avoids redundant dispatch on every target.

- Python
Published by github-actions[bot] 7 months ago

https://github.com/fgcz/bfabricpy -

[0.2.0] - 2025-07-15

This release consolidates various commands in bfabric-app-runner streamlining the user experience. It also brings an improved Makefile which should obsolete the manual installation of bfabric-app-runner providing the user with the configured version by default.

Removed

  • Removed experimental bfabric-app-runner deploy build-app-zip command and associated app.zip functionality. This functionality has been superseded by the more robust CommandPythonEnv approach for Python environment management.
  • Removed bfabric-app-runner app command (covered by bfabric-app-runner run and bfabric-app-runner prepare now).
  • Removed bfabric-app-runner chunk command (covered by bfabric-app-runner action now).
  • Removed module path support for app specification. CommandPythonEnv is cleaner than the workflow this tried to enable.
  • The app related commands do not support AppVersion inputs anymore. Instead, you should always specify a AppSpec file.

Changed

  • bfabric-app-runner prepare workunit does not accept module refs anymore and will resolve the app spec path.
  • bfabric-app-runner prepare workunit --force-app-version to force a specific app version for the workunit.
  • Old bfabric-app-runner app run and bfabric-app-runner app dispatch use the new makefile function from cmd_prepare.

Added

  • copier based template/demo application for development and end-to-end testing of bfabric-app-runner.
  • Added ResolvedDirectory type to represent directories resolved from resource archives.
  • Added BfabricResourceArchiveSpec to specify input archives which should be extracted (and select which files are needed).
  • Validation logic has been added for ResolvedDirectory and BfabricResourceArchiveSpec. In particular a ResolvedDirectory may never overlap with a ResolvedFile or ResolvedStaticFile path.
  • Using uv tool the Makefile will provide the correct version of the app runner when called. To opt-out of this behavior, one can set USE_EXTERNAL_RUNNER=true for the makefile.

- Python
Published by github-actions[bot] 8 months ago

https://github.com/fgcz/bfabricpy -

[0.1.2] - 2025-07-08

  • CommandPythonEnv with refresh=True now will create a separate environment to avoid breaking apps which are already using a particular Python environment without locking it.

- Python
Published by github-actions[bot] 8 months ago

https://github.com/fgcz/bfabricpy -

[0.1.1] - 2025-07-07

Changed

  • CommandPythonEnv computes the hash more carefully.

- Python
Published by github-actions[bot] 8 months ago

https://github.com/fgcz/bfabricpy -

[1.13.30] - 2025-07-04

Added

  • bfabric_flask.py provides support for token-based authentication for B-Fabric.

Removed

  • bfabric_flask.py is not exported as a script anymore, because misuse can lead to security issues and should be deployed properly.

- Python
Published by github-actions[bot] 8 months ago

https://github.com/fgcz/bfabricpy -

[1.13.29] - 2025-07-04

Changed

  • bfabric.rest.token_data.get_token_data now only requires the base_url instead of the whole config.

- Python
Published by github-actions[bot] 8 months ago

https://github.com/fgcz/bfabricpy -

[1.13.29] - 2025-06-27

Removed

  • Deprecated bfabric-cli api log subcommand was removed.

Added

  • bfabric-cli executable dump command to export YAML and XML of executables
  • bfabric-cli feeder create-importresource command to register importresource objects in B-Fabric
  • bfabric-cli executable upload supports XML

Fixed

  • bfabric-cli workunit not-available shows nodelist for workunits using new submitter.

Changed

  • Columns of tables named after B-Fabric entities, containing only integers, will be set as the specified type when saving to B-Fabric.
  • Update bfabric to 1.13.28.
  • Update legacy bfabric_logthis.py, the workunit target logic has been removed (unused).

- Python
Published by github-actions[bot] 8 months ago

https://github.com/fgcz/bfabricpy -

[0.1.0] - 2025-06-27

Added

  • Command type python_env for commands which require a provisioned Python environment.

Changed

  • Command implementation is separated from command definition for cleaner implementation.
  • New submitter requires BFABRICPY_CONFIG_ENV and XDG_CACHE_HOME to be set.
  • Update bfabric dependency to 1.13.28.

- Python
Published by github-actions[bot] 8 months ago

https://github.com/fgcz/bfabricpy -

[1.13.28] - 2025-06-27

Breaking

  • base_url is now a mandatory parameter in the configuration.

Added

  • It is now possible to configure ~/.bfabricpy.yml without a default environment. In that case it will always be necessary to specify the requested config environment to be used.
  • bfabric.entities.Executable has parameters relationship now

Removed

  • Some old submitter related functionality is deleted.

Changed

  • Columns of tables named after B-Fabric entities, containing only integers, will be set as the specified type when saving to B-Fabric (in experimental.upload_dataset).

- Python
Published by github-actions[bot] 8 months ago

https://github.com/fgcz/bfabricpy -

[0.0.23] - 2025-06-02

Added

  • bfabric_app_runner.bfabric_integration.slurm and associated packages.
  • Command bfabric-app-runner run workunit to run the whole app end-to-end for a workunit.

Removed

  • AppVersion does not have a submitter field anymore.
  • Some old submitter related functionality is deleted.

- Python
Published by github-actions[bot] 9 months ago

https://github.com/fgcz/bfabricpy -

[1.13.28] - 2025-05-21

Removed

  • Removed bfabric_delete.py. Use bfabric-cli api delete instead.
  • Removed bfabric_list_not_available_proteomics_workunits.py. Use bfabric-cli workunit not-available instead.

Changed

  • Update bfabric to 1.13.27.
  • bfabric-cli api delete will use the type of the entity in CLI messages.

- Python
Published by github-actions[bot] 9 months ago

https://github.com/fgcz/bfabricpy -

[0.0.22] - 2025-05-21

Added

  • Apps can now be referred to by module path rather than just file paths. This is going to be a primary building block to very simple package-based deployment of apps.
  • The static_file input spec type has been integrated properly.
  • Missing integration for file input spec type has been added.
  • The workunit makefile now directly shows how to use the GitHub app runner version instead, which is sometimes required while debugging.
  • CommandExec allows prepending paths to PATH and setting environment variables and is less ambiguous than shell.
  • bfabric-app-runner action interface which standardizes the various actions of running app steps.
  • bfabric-app-runner prepare workunit to prepare a workunit execution and sets up a app_env.yml and Makefile.
  • bfabric-app-runner deploy build-app-zip experimental command to build an app zip file which can be deployed, for a particular Python application.

Changed

  • Silently interpolateconfigstrings log messages.
  • Update bfabric dependency to 1.13.27.
  • App versions do not always require a version key as it will default to "latest", but only one version can have a particular version key per app definition.

Fixed

  • Use most recent cyclopts version again, i.e. issue 168 is fixed.
  • Compatibility with pandera 0.24.0 was restored.

- Python
Published by github-actions[bot] 9 months ago

https://github.com/fgcz/bfabricpy -

[1.13.27] - 2025-05-21

Added

  • Attribute Bfabric.config_data to obtain a ConfigData object directly.
  • TokenData.load_entity convenience method to load an entity from the token data.
  • Entities Instrument, Plate, Run were added (but with no extra functionality).
  • Workunit.{application_parameters, submitter_parameters} to access parameter values.

Changed

  • Submitter parameters will not be written any longer to WorkunitExecution parameters.

Fixed

  • Compatibility with upcoming change that Application can have multiple technology values.

Deprecated

  • Workunit.parameter_values will be removed in favor of Workunit.application_parameters and Workunit.submitter_parameters in a future version.

- Python
Published by github-actions[bot] 9 months ago

https://github.com/fgcz/bfabricpy -

[1.13.26] - 2025-04-26

This release introduces an environment variable BFABRICPY_CONFIG_OVERRIDE to configure the Bfabric client completely, along with a new method for creating an instance of the Bfabric client, Bfabric.connect(). This will allow us to propagate any configuration to subprocesses reliably. BFABRICPY_CONFIG_ENV remains available with the same semantics, but lower priority than BFABRICPY_CONFIG_OVERRIDE.

This also simplifies the logic that was present in Bfabric.from_config which is why this introduced with a new API to prevent configuration mix-ups.

Added

  • New environment variable BFABRICPY_CONFIG_OVERRIDE to configure the Bfabric client completely
  • New method Bfabric.connect() for creating an instance of the Bfabric client

Changed

  • Renamed Bfabric.from_token to Bfabric.connect_webapp() (along with some changes, no known users of this API yet)
  • Disallowed default as an environment config name
  • bfabric.cli_integration.utils.use_client uses Bfabric.connect() instead of Bfabric.from_config()

Deprecated

  • Bfabric.from_config is now deprecated in favor of Bfabric.connect()

- Python
Published by github-actions[bot] 10 months ago

https://github.com/fgcz/bfabricpy -

[1.13.27] - 2025-04-22

Changed

  • bfabric-cli dataset upload will print warnings when trailing whitespace is detected and not print the whole response anymore, but rather the important information only.

Added

  • Optional support for uploading xlsx (currently behind excel optional feature).

- Python
Published by github-actions[bot] 10 months ago

https://github.com/fgcz/bfabricpy -

[1.13.25] - 2025-04-22

Breaking

  • Bfabric.from_token returns the TokenData in addition to the client instance. While this is breaking, I'm not aware of any existing users of this API and I noticed that this information is going to be needed in this context often.

Added

  • bfabric.experimental.upload_dataset.warn_on_trailing_spaces function used by bfabric-scripts to validate

- Python
Published by github-actions[bot] 10 months ago

https://github.com/fgcz/bfabricpy -

Changed

  • Update bfabric to 1.13.24.

Removed

  • Remove bfabric-cli api save -> use bfabric-cli api create and bfabric-cli api update instead.

- Python
Published by github-actions[bot] 11 months ago

https://github.com/fgcz/bfabricpy -

Removed

  • cyclopts is not a dependency of bfabric anymore, but rather of bfabric-scripts and bfabric-app-runner.

Changed

  • Bfabric client now does not take engine argument anymore, but rather this information comes from ClientConfig. For compatibility reasons, the engine argument is still accepted in Bfabric.from_config and Bfabric.from_token.

- Python
Published by github-actions[bot] 11 months ago

https://github.com/fgcz/bfabricpy -

Added

  • workunit can now be interpolated in config files.
  • SubmittersSpec to define a slurm submitter.

Changed

  • Submitter params in app definition, use key params rather than config to be more explicit.

Fixed

  • Temporary workaround for https://github.com/fgcz/bfabricPy/issues/168.

- Python
Published by github-actions[bot] 11 months ago

https://github.com/fgcz/bfabricpy - bfabric_scripts/1.13.25

Fixed

  • Temporary workaround for https://github.com/fgcz/bfabricPy/issues/168.

- Python
Published by leoschwarz 11 months ago

https://github.com/fgcz/bfabricpy -

Changed

  • Input staging is now more efficient for large numbers of similar input types, by batching the transformation into resolved operations.

Fixed

  • workunit.mk correctly chooses between app_version.yml and app_definition.yml depending on availability
  • Multiple inputs with the same output filename, will yield an error now.
  • File timestamps will not be modified anymore, if the file is already up-to-date.

Added

  • SaveLinkSpec to save a B-Fabric link e.g. to a workunit.

Removed

  • file_scp spec has been removed, one should use file instead (FileSpec)
  • A lot of the old input handling code has been removed, it should not cause any problems, but mentioning this in case it shows up after the release.

- Python
Published by github-actions[bot] 11 months ago

https://github.com/fgcz/bfabricpy -

Fixed

  • Unsuccessful deletions are detected by checking the B-Fabric response.
  • Handle problematic characters in Workunit.store_output_folder.
  • BfabricRequestError did not properly subclass RuntimeError.

Changed

  • WorkunitDefinition uses PathSafeStr to normalize app and workunit names.
  • Internal: ResultContainer has no optional constructor arguments anymore to avoid confusion.

Added

  • Generic functionality in bfabric.utils.path_safe_name to validate names for use in paths.
  • bfabric.entities.Dataset.{write_parquet, get_parquet} methods for writing parquet

- Python
Published by github-actions[bot] 11 months ago

https://github.com/fgcz/bfabricpy -

Added

  • dispatch_resource_flow output table allows null filename
  • --force-storage is available in more commands now

Changed

  • The workunit.mk Makefile now specifies Python 3.13 in the uv venv, so it is more reliable.

- Python
Published by github-actions[bot] about 1 year ago

https://github.com/fgcz/bfabricpy -

Added

  • static_yaml input type to write parameters etc.
  • dispatch.dispatch_resource_flow implements a generic solution for dispatching resource flow workunits without having to perform entity look up in many cases yourself.
  • App Definition now supports omitting the collect step.

Changed

  • BfabricResourceSpec defaults to file basename vs resource name, if no name is specified.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Fixed

  • Update bfabric to 1.13.22 for dataset fix.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Fixed

  • Update bfabric to 1.13.22 for dataset fix.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Fixed

  • Correctly read datasets, if columns were swapped in B-Fabric.

Changed

  • Dataset.types was renamed to Dataset.column_types, since there is only 1 usage and it is recent, this is not considered breaking.

- Python
Published by github-actions[bot] about 1 year ago

https://github.com/fgcz/bfabricpy -

Added

  • Implement --force-storage to pass a yaml to a forced storage instead of the real one.
  • A Makefile will be created in the app folder for easier interaction with the app-runner (it uses uv and PyPI).

Changed

  • CopyResourceSpec.updateexisting now defaults to `ifexists`.
  • Resolve workunit_ref to absolute path if it is a Path instance for CLI.

- Python
Published by github-actions[bot] about 1 year ago

https://github.com/fgcz/bfabricpy -

Added

  • bfabric-cli dataset {upload, download, show} to replace the old dataset-related scripts.
  • bfabric-cli api update command to update an existing entity
  • bfabric-cli api create command to create a new entity
  • bfabric-cli api delete command to delete an existing entity

- Python
Published by github-actions[bot] about 1 year ago

https://github.com/fgcz/bfabricpy -

Added

  • Entity.load_yaml and Entity.dump_yaml
  • Bfabric.from_token to create a Bfabric instance from a token
  • bfabric.rest.token_data to get token data from the REST API, low-level functionality

Changed

  • Internally, the user password is now in a pydantic.SecretStr until we construct the API call. This should prevent some logging related accidents.

- Python
Published by github-actions[bot] about 1 year ago

https://github.com/fgcz/bfabricpy -

Fixed

  • Fix printing of YAML for parsing with shyaml, previously line breaks could have been introduced.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Added

  • bfabric-cli workunit not-available:
    • allows sorting by arbitrary fields, e.g. application id
    • allows filtering inclusive or exclusive by user

Changed

  • Pin bfabricPy version to avoid future headaches.
  • bfabric-cli api read
    • Removes the automatic output type logic
    • Multiple values can be submitted for the same key (just specify it multiple times)
    • The actual query will be printed as a line of bfabricPy code
    • --file parameter to write the output to a specific file
    • Argument parsing is handled with pydantic now
    • Added tsv support

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Breaking

  • the old bfabric.cli_formatting and bfabric.bfabric2 modules have been deleted

Changed

  • move use_client to bfabric.utils.cli_integration: this will allow reuse between bfabric-scripts and bfabric-app-runner
  • move functionality from bfabric.cli_formatting to bfabric.utils.cli_integration

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Added

  • New input type file which replaces file_scp and preserves timestamps whenever possible and allows to create symlinks instead of copying the file, as needed.
  • BfabricOrderFastaSpec.required which allows specifying whether the order fasta is required or not

Changed

  • Better error when app version is not found.

Fixed

  • Config: Log messages are shown by default again.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Fixed

  • Config: Log messages of app runner are shown by default again.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - app_runner/0.0.14

Fixed

  • Correctly consume bfabricPy from PyPI.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - app_runner/0.0.13

Added

  • WorkunitDefinition.registration.workunit_name field.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - 1.13.18

Changed

  • bfabric_upload_resource.py does not print a list anymore, but rather only the dict of the uploaded resource.

Fixed

  • Some scripts were not Python 3.9 compatible, which is restored now. However, we can consider increasing the Python requirement soon.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - 1.13.17

Added

  • bfabric-cli workunit export-definition to export workunit_definition.yml files
  • Executable.storage relationship

Fixed

  • Workunit.parameters and Workunit.resources are optional
  • bfabric-cli had unmet dependencies, that were not caught by the tests either

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - app_runner/0.0.12

Added

  • New input type bfabric_order_fasta which will place an order fasta file to the specified path, or create an empty file if there is no order fasta available.
  • --filter flag has been added to inputs prepare and inputs clean commands.
  • The app-runner app commands now support passing a AppVersion yaml file instead of just a AppSpec yaml file.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Added

  • Add missing Entity.__contains__ implementation to check if a key is present in an entity.
  • polars_utils.py which contains functionality to normalize relational fields in tables
  • Add bfabric-cli executable inspect command to inspect executables registered in B-Fabric.
  • Add bfabric-cli executable upload command to upload executables to B-Fabric.

Fixed

  • Order.project is optional

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - app_runner/0.0.11

Added

  • New input type BfabricAnnotationSpec.
    • For now, it only supports one type of annotation, which is "resource_sample".

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - app_runner/0.0.10

Added

  • FileScpSpec to copy a file from a remote server to the local filesystem, without using B-Fabric information.
  • CommandDocker.hostname so it won't have to be passed by custom_args in the future.
  • DispatchSingleDatasetFlow dispatch a workunit in dataset-flow which consists of only one execution unit.
  • DispatchSingleResourceFlow dispatch a workunit in resource-flow which consists of only one execution unit.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Added

  • Entity Sample
  • Relationship Resource.sample.
  • Extract logic for container resolution into HasContainerMixin which is right now shared between Sample and Workunit.

Fixed

  • Fix bug in bfabric_save_fasta.py.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - app_runner/0.0.9

Added

  • App specs can now define multiple versions in one file. (AppSpec = Collection of app versions and other information.)
    • To avoid boilerplate, mako templates can be used inside of strings.
    • Apps will resolve the version to use based on the application_version field.
    • Validation functionality for the new app specification has been added.
  • App versions can define a submitter, however this information is not yet used.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - app_runner/0.0.8

Added

  • Register single file command: bfabric-app-runner outputs register-single-file
  • Implement copy resource UpdateExisting.IF_EXISTS and UpdateExisting.REQUIRED support.
  • The following fields have been added to WorkunitRegistrationDefinition:
    • storage_id
    • storage_output_folder
    • application_id
    • application_name

Changed

  • App-runner code related to output staging accepts workunit-definition file like the other steps.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Changed

  • Slurm submitter prints more diagnostics about PATH variable etc.

Fixed

  • Some commands in bfabric-cli are broken because __future__.annotations is imported and this breaks cyclopts.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - 1.13.13

Changed

  • Move bfabric-cli read to bfabric-cli api read.
  • (internal) use_client decorator is introduced to simplify and standardize the client usage in the new CLI code.

Added

  • Functionality to log by specifying the workunit instead of external job, which is used in some legacy scripts.
  • Add logging commands to bfabric-cli api log.
  • Add bfabric-cli api save.
  • Submitter script prints the id and hostname of apps, which in practice can be very useful.

Fixed

  • A bug in bfabricsaveworkflowstep.py that crashed the script.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Changed

  • The submitter ensures that workunits always get set to processing.

Fixed

  • The script bfabric_setResourceStatus_available.py and other uses of report_resource, correctly search files which have a relative path starting with / as in the case of the legacy wrapper creator.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy -

Fixed

  • The script bfabric_setWorkunitStatus.py did always set the status to available, instead of the specified status.

Changed

  • bfabric_setResourceStatus_available.py calls the report_resource function, in general this functionality has been refactored.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - 1.13.10

Fixed

  • If bfabricpy.yml contains a root-level key which is not a dictionary, a correct error message is shown instead of raising an exception.
  • A bug introduced while refactoring in slurm.py which passed Path objects to subprocess instead of strings.
  • Submitter
    • does not use unset JOB_ID environment variable anymore.
    • does not set unused STAMP environment variable anymore.
  • Fix bfabric_save_workflowstep.py bugs from refactoring.

Added

  • Experimental bfabric-cli interface. Please do not use it in production yet as it will need a lot of refinement.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - 1.13.9

From this release onwards, the experimental app runner is not part of the main bfabric package and instead a separate Python package with its individual changelog.

Added

  • Relationship: ExternalJob.executable
  • (experimental) EntityLookupCache that allows to cache entity lookups in a script to avoid redundant requests.
  • Specific use case script: bfabricsaveresource_description.py (the functionality will be available in a future CLI).

Fixed

  • Entity.find_all returns no values when an empty list is passed as an argument.

Changed

  • Except for macOS x86_64 (which we assume is Rosetta emulation nowadays), we use the faster polars instead of polars-lts-cpu.
  • BfabricRequestError is now a RuntimeError subclass.
  • Add py.typed marker.

Removed

  • bfabric_legacy.py has been removed.
  • math_helper.py has been removed.

- Python
Published by leoschwarz about 1 year ago

https://github.com/fgcz/bfabricpy - app_runner/0.0.7

Fixed

  • When executing app run the experimental entity cache created incorrect behavior. The caching is temporarily disabled, until the issue is resolved.

- Python
Published by leoschwarz over 1 year ago

https://github.com/fgcz/bfabricpy - app_runner/0.0.6

- Python
Published by leoschwarz over 1 year ago

https://github.com/fgcz/bfabricpy -

This release contains mainly internal changes and ongoing development on the experimental app interface functionality.

Added

  • Entities can be compared and sorted by ID now.
  • Show Python version in version info string.
  • Caching for bfabriclistnotexistingstorage_directories.py.
  • (experimental) add initial code for a resource based application dispatch
  • (experimental) new app_runner cli that integrates all commands into a single interface

Fixed

  • bfabric_read.py is a bit more robust if "name" misses and tabular output is requested.

Changed

  • bfabric.scripts has been moved into a namespace package bfabric_scripts so we can later split it off.
  • (internal) migrate to src layout
  • (experimental) the former process step of the app runner has been split into a process and collect step where, the collect step is responsible for generating the output.yml file that will then be used to register the results.
  • (experimental) app runner apps by default reuse the default resource

- Python
Published by leoschwarz over 1 year ago

https://github.com/fgcz/bfabricpy - 1.13.7

Fixed

  • bfabric_save_csv2dataset.py considers all rows for schema inference, only considering the first 100 rows did cause problems with some files previously.

Added

  • Script logging configuration
    • If BFABRICPY_DEBUG environment variable is set, log messages in scripts will be set to debug mode.
    • Log messages from __main__ will also be shown by default.
  • bfabric.entities.MultiplexKit to extract multiplex kit information.
  • bfabric.entities.Workunit.store_output_folder implements the old rule, but more deterministically and reusable.
  • (Experimental) bfabric.experimental.app_interface core functionality is implemented
    • This can be used as a building block to standardize the input preparation for applications.

Changed

  • Correctly support optional workunit parameters.

Removed

  • bfabric.entities.Resource association application has been removed as it does not exist

- Python
Published by leoschwarz over 1 year ago

https://github.com/fgcz/bfabricpy -

Added

  • Entity.find_by has new parameter max_results.

Changed

  • bfabric_read.py prints non-output information exclusively through logger and does not restrict entity types anymore.
  • bfabric_delete.py accepts multiple ids at once and does not restrict entity types anymore.
  • Both engines raise a BfabricRequestError if the endpoint was not found.

Removed

  • bfabric.endpoints list is deleted, since it was out of date and generally not so useful.

- Python
Published by leoschwarz over 1 year ago

https://github.com/fgcz/bfabricpy - 1.13.5

Added

  • The Bfabric instance is now pickleable.
  • Entities mapping:
    • Add Entity.id and Entity.web_url properties.
    • Add Entity.__getitem__ and Entity.get to access fields from the data dictionary directly.
    • Add Entity.find_by to find entities by a query.
    • More types and relationships
    • Relationships defer imports to descriptor call, i.e. circular relationships are possible now.
    • HasOne and HasMany allow defining optional=True to indicate fields which can be missing under some circumstances.
  • Add nodelist column and application name to bfabric_list_not_available_proteomics_workunits.py output.

Changed

  • Entity.find_all supports more than 100 IDs now by using the experimental MultiQuery API.

- Python
Published by leoschwarz over 1 year ago

https://github.com/fgcz/bfabricpy - 1.13.4

Added

  • Add Workunit, Parameter, and Resource entities.
  • Add concept of hasmany and hasone relationships to entities.
  • bfabric_slurm_queue_status.py to quickly check slurm queue status.
  • Bfabric.save provides method which can be set to checkandinsert for specific use cases.

Changed

  • Most messages are now logged to debug level.
  • The old verbose version information is now always logged, to INFO level, since it could entail useful information for error reporting.

- Python
Published by leoschwarz over 1 year ago

https://github.com/fgcz/bfabricpy -

Added

  • Flask
    • New endpoint GET /config/remote_base_url for testing

Changed

  • Flask
    • Simplify logging by using loguru only.
    • Simplified setup logic since the production use case should use a WSGI server.

Fixed

  • bfabric_save_csv2dataset.py had an undeclared dependency on numpy and a few bugs which was improved.

- Python
Published by leoschwarz over 1 year ago

https://github.com/fgcz/bfabricpy -

Added

  • Add bfabric.entities.Dataset to easily read datasets.
  • Pydantic-based configuration parsing
    • The config format did not change.
    • The code is easier to maintain now.
    • Additionally, there is a lot more validation of the configuration file now, that should catch errors early.
  • Make host and port configurable in bfabric_flask.py (currently only dev mode).

- Python
Published by leoschwarz over 1 year ago

https://github.com/fgcz/bfabricpy -

Changed

  • bfabricsavecsv2dataset will raise an error if problematic characters are found in any of the cells
  • Correctly define bfabric_setWorkunitStatus_available.py, and processing and failed variants.

Added

  • Add loguru for future logging refactoring.
  • Easily runnable tests with nox and standardized formatting using pre-commit.

Removed

  • Pandas is no longer a dependency, and has been replaced by polars.

- Python
Published by leoschwarz over 1 year ago

https://github.com/fgcz/bfabricpy - 1.13.0

This is a major release refactoring bfabricPy's API.

Changed

  • The Bfabric class operations now return ResultContainer objects.
    • These provide a list-like interface to access individual items or iterate over them.
    • Individual items are a dictionary, potentially nested, and not specific to suds/zeep anymore.
    • Convenience conversions, e.g. to a polars DataFrame, can be provided there.
  • Configuration is now defined in ~/.bfabricpy.yml and supports multiple configurations, which can be selected by the BFABRICPY_CONFIG_ENV environment variable. Please consult the README for an example configuration.
  • Use pyproject.toml for package configuration.
  • Scripts have been refactored on a case-by-case basis.

Added

  • Zeep can be used instead of suds for SOAP communication.
  • Bfabric can be instantiated without authentication, that can be provided later. This is useful in a server setup.
  • Pagination support in Bfabric, specify the number of max_results and a potential offset. Pages handling is abstracted away.
  • Detect errors in responses, e.g. invalid login.

Removed

  • Several old scripts have been moved into a deprecated_scripts folder.
  • Wrapper creator related code is currently not updated but has been extracted into a dedicated folder wrapper_creator as well.

- Python
Published by leoschwarz almost 2 years ago

https://github.com/fgcz/bfabricpy - bfabricPy 0.13.8

- Python
Published by leoschwarz almost 2 years ago

https://github.com/fgcz/bfabricpy - bfabricPy 0.13.3

bfabric-0.13.3.tar.gz

- Python
Published by cpanse over 2 years ago

https://github.com/fgcz/bfabricpy - dataset

bfabric-0.11.7.tar.gz

enables fetching and saving datasets to and from csv files using the following command line scripts:

  • bfabric_save_csv2dataset.py
  • bfabric_save_dataset2csv.py

- Python
Published by cpanse about 4 years ago

https://github.com/fgcz/bfabricpy - bfabric-9

Dear B-Fabric User,

we are happy to announce the new release of B-Fabric which comes with a number of significant changes and extensions (see http://fgcz-bfabric.uzh.ch/wiki/tiki-index.php?page=Release+Notes). First of all, please note that we have integrated the concepts of samples and extracts. The new system knows only the notion of a sample. As before in the case of extracts, samples can also be derived from other samples, thus creating sample hierarchies. All your existing extracts were transformed and integrated into corresponding samples. Also note that the supported sample types were revised and the batch editing of samples improved. In particular, it is now possible to batch edit custom attributes of samples.

Please clear your browser's cache and enjoy. As always, your feedback is welcome.

Many thanks in advance and best regards,

The FGCZ B-Fabric Team

- Python
Published by cpanse over 7 years ago

https://github.com/fgcz/bfabricpy - bfabric-9.6

contains the MaxQuant linux wrapper and SGE script to perform MaxQuant version 1.6.2.3

- Python
Published by cpanse over 7 years ago

https://github.com/fgcz/bfabricpy - bfabric8

this release was used until 2017-10.

- Python
Published by cpanse over 8 years ago