https://github.com/zenml-io/zenml - 0.84.3

The 0.84.3 release introduces ZenML Pro service account authentication support and includes important Kubernetes integration fixes. This release enhances authentication flexibility for automated workflows and improves the reliability of Kubernetes-based orchestration.

🚀 New Features

ZenML Pro Service Account Authentication

• CLI Login Support: Added ability to authenticate with ZenML Pro using service account API keys via zenml login --api-key
• Programmatic Access: Service account API keys can now be used for programmatic access to ZenML Pro workspaces
• Organization-Level Access: Service accounts provide access to all workspaces within an organization for automated workflows

🛠️ Fixes

Kubernetes Integration

• Name Sanitization: Fixed Kubernetes resource name sanitization to properly handle special characters and ensure compliance with Kubernetes naming requirements

Dependencies

• Click Version: Relaxed Click dependency version constraints to improve compatibility with other packages

What's Changed

• Add version 0.84.1 to legacy docs by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/3885
• Fix PyPI download stats badge by @strickvl in https://github.com/zenml-io/zenml/pull/3882
• Don't include decorator code in step cache key computation by @schustmi in https://github.com/zenml-io/zenml/pull/3880
• Add an agent in production example by @htahir1 in https://github.com/zenml-io/zenml/pull/3890
• Fix kubernetes name sanitization by @schustmi in https://github.com/zenml-io/zenml/pull/3887
• Fix SyntaxWarning in exception_utils regex pattern by @strickvl in https://github.com/zenml-io/zenml/pull/3892
• Add agent framework integration examples by @htahir1 in https://github.com/zenml-io/zenml/pull/3898
• Add weekly agent examples test workflow by @strickvl in https://github.com/zenml-io/zenml/pull/3902
• Add ability to use a ZenML Pro API key with zenml login by @stefannica in https://github.com/zenml-io/zenml/pull/3895
• Add docs for workload manager env vars by @schustmi in https://github.com/zenml-io/zenml/pull/3907
• Bump allowed versions of click by @strickvl in https://github.com/zenml-io/zenml/pull/3905
• Allow workspace service accounts and API keys to be created by @stefannica in https://github.com/zenml-io/zenml/pull/3908

Full Changelog: https://github.com/zenml-io/zenml/compare/0.84.2...0.84.3

- Python
Published by schustmi 6 months ago

https://github.com/zenml-io/zenml - 0.84.2

The 0.84.2 release brings significant performance improvements and reliability enhancements that strengthen ZenML's orchestration capabilities. This release features a major refactor of the Kubernetes orchestrator with enhanced restart capabilities, faster pipeline compilation for large pipelines, critical security fixes, and improved logging performance.

Kubernetes Orchestrator

• Enhanced Robustness: Complete rework to use Jobs instead of raw pods for orchestrator execution with state reconstruction support, enabling automatic restarts for a much more robust orchestration experience
• Deprecated Settings: Several settings attributes are now deprecated due to the shift from pods to Jobs, streamlining the configuration interface
• Updated Logging Behavior: The orchestrator pod no longer streams logs from step pods directly, improving performance and resource usage

🚀 Improvements

• Faster Pipeline Compilation: Significantly improved pipeline compilation performance for large pipelines

🛠️ Fixes

• Run Creation Deadlock: Fixed deadlock that could happen if two steps try to create a run at the same time
• Security Enhancement: Enhanced path materializer to properly validate symlinks and hardlinks to prevent path traversal attacks
• Logging Performance: Improved logging thread performance by avoiding unnecessary sleep when shutdown is requested
• WandB Integration: Fixed WandB experiment tracker flavor initialization when the integration is not installed
• Type Annotations: Fixed type annotations for experiment_tracker and step_operator parameters to accept both string and boolean values

What's Changed

• Add version 0.84.0 to legacy docs by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/3863
• Add 0.84.1 to the migration tests by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/3865
• Fix wandb flavor needing integration to be installed by @stefannica in https://github.com/zenml-io/zenml/pull/3871
• Verify symlinks and hardlinks in the path materializer by @stefannica in https://github.com/zenml-io/zenml/pull/3870
• Don't sleep in logging thread if shutdown was requested by @schustmi in https://github.com/zenml-io/zenml/pull/3872
• Fix run creation deadlock by @schustmi in https://github.com/zenml-io/zenml/pull/3876
• Fix nonlinear scaling of pipeline compilation time by @schustmi in https://github.com/zenml-io/zenml/pull/3873
• Fix type annotations by @schustmi in https://github.com/zenml-io/zenml/pull/3878
• Orchestrator pod restarts by @schustmi in https://github.com/zenml-io/zenml/pull/3869

Full Changelog: https://github.com/zenml-io/zenml/compare/0.84.1...0.84.2

- Python
Published by schustmi 7 months ago

https://github.com/zenml-io/zenml - 0.84.1

The 0.84.1 release delivers important stability improvements and feature enhancements that strengthen ZenML's orchestration capabilities and developer experience. This release focuses on enhanced Kubernetes orchestrator management with schedule support and better error handling, improved step exception collection for debugging, external service account support, dynamic fan-out/fan-in patterns for run templates, and critical fixes for Vertex step operators and logging reliability.

🚀 Improvements

• Step Exception Handling: Improved collection of step run exception information for better debugging
• External Service Accounts: Added support for external service accounts for improved flexibility

Kubernetes Orchestrator

• Enhanced Schedule Management: Added schedule management capabilities (updating and deleting) for the Kubernetes orchestrator
• Better Error Handling: Added explicit Kubernetes failure reasons to logs and improved pod monitoring
• Secret Cleanup: Fixed cleanup of Kubernetes secrets when orchestrator pods fail to start

🛠️ Fixes

• Vertex step operator credential refresh: Fixed retry logic and credential refresh for the Vertex step operator
• Logging Fixes: Resolved race conditions in logging for better reliability

🛠️ Documentation & Examples

• Agent Examples: Updated README and added comprehensive agent examples
• Template Updates: Bumped LLM template version for latest improvements
• Deprecation Cleanup: Removed deprecation warnings in quickstart for cleaner user experience
• Dynamic Fan-out/Fan-in: Added support for dynamic fan-out/fan-in patterns with run templates for more flexible pipeline architectures

What's Changed

• Add version 0.83.1 to legacy docs by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/3828
• Add dynamic fan-out/fan-in with run templates by @htahir1 in https://github.com/zenml-io/zenml/pull/3826
• Add version 0.84.0 to DB migration script by @schustmi in https://github.com/zenml-io/zenml/pull/3833
• Bump LLM template by @schustmi in https://github.com/zenml-io/zenml/pull/3837
• Update README and add agent example by @htahir1 in https://github.com/zenml-io/zenml/pull/3815
• Remove deprecation warning in quickstart by @schustmi in https://github.com/zenml-io/zenml/pull/3834
• Fix appending to DB migrations in release action by @schustmi in https://github.com/zenml-io/zenml/pull/3836
• Add intermediate cleanup in CLI profiling CI by @schustmi in https://github.com/zenml-io/zenml/pull/3835
• Purge unused/inactive local docker services by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3768
• Fix the retry logic and credential refresh for the Vertex step operator by @stefannica in https://github.com/zenml-io/zenml/pull/3853
• Schedule management for the Kubernetes orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/3847
• Cleanup Kubernetes secret if orchestrator pod fails to start by @schustmi in https://github.com/zenml-io/zenml/pull/3846
• Enable schedule RBAC by @schustmi in https://github.com/zenml-io/zenml/pull/3848
• Add explicit k8s failure reasons to logs by @avishniakov in https://github.com/zenml-io/zenml/pull/3854
• Collect step run exception info by @schustmi in https://github.com/zenml-io/zenml/pull/3838
• Block bot traffic from Segment analytics to prevent MTU quota exhaustion by @strickvl in https://github.com/zenml-io/zenml/pull/3858
• Fail on certain container waiting reasons during job monitoring by @schustmi in https://github.com/zenml-io/zenml/pull/3851
• Add support for external service accounts by @stefannica in https://github.com/zenml-io/zenml/pull/3793
• Solving the race conditions for logging by @bcdurak in https://github.com/zenml-io/zenml/pull/3855

Full Changelog: https://github.com/zenml-io/zenml/compare/0.84.0...0.84.1

- Python
Published by schustmi 7 months ago

https://github.com/zenml-io/zenml - 0.84.0

The 0.84.0 release delivers significant architectural improvements and orchestration enhancements that boost ZenML's reliability, performance, and developer experience. This release focuses on pipeline execution resilience with step retries and early stopping capabilities, enhanced Kubernetes orchestrator features, and improved server-side processing for better scalability.

🚨 Breaking Changes

Kubernetes Orchestrator Compatibility

• Client-Orchestrator Version Compatibility: This release introduces changes to the Kubernetes orchestrator that make the new client version incompatible with previous versions. When using the Kubernetes orchestrator, the ZenML version in both the client and orchestrator pod must match exactly. This means that if you upgrade your ZenML client to 0.84.0, you must also ensure that the orchestrator pod images are also using ZenML 0.84.0 to avoid compatibility issues.

🚀 Orchestration Enhancements

Pipeline Control & Reliability

• Early Pipeline Stopping: Added ability to stop pipelines early with Kubernetes orchestrator implementation for better resource management
• Step Retries: Implemented configurable step retry mechanisms for improved pipeline resilience
• Step Status Refresh: Enhanced step status monitoring with real-time refresh capabilities

Kubernetes Orchestrator Improvements

• Run Steps using Jobs: Leverage Kubernetes jobs for better retry behavior when running steps
• Enhanced Pod Management: Always build pipeline images for Kubernetes orchestrator and improved pod caching
• Pod Logs Access: Added orchestrator pod logs functionality for better debugging and monitoring
• Kubernetes Model Validation: Added warnings for invalid Kubernetes model data

⚡ Performance Improvements

• Threadsafe RestZenStore: Implemented thread-safe operations for improved concurrent access
• Server-side Processing: Moved parent step computation and cascading tags processing to server-side for better performance
• Improved Fetching: Enhanced pipeline and step run fetching mechanisms for faster response times

🔧 Developer Experience

CLI & Login Improvements

• New ZenML Login: Redesigned login experience with improved CLI styling and user interface

Configuration & Compatibility

• Easier Configuration: Simplified step operator and experiment tracker configuration for steps
• Unified Imports: Standardized config imports to match documentation examples

🔒 Infrastructure & Security

• Helm Chart Improvements: Added PVC creation option for local configuration path
• Service Connector Security: Hidden service connector secrets as internal implementation details

🛠️ Fixes

• Model Version Fetching: Fixed fetching model versions by UUID passed as string
• Visualization Handling: Ensured visualizations are committed before potential transaction rollbacks
• SageMaker Configuration: Made SagemakerOrchestratorSettings.processor_tags optional
• Data Artifact Access: Fixed data artifact fetching issues
• Pydantic Validation: Added support for step/parameter names with leading underscores in run templates
• Path Sanitization: Added missing remote path and Docker tag sanitizations

📚 Documentation

• Best Practices: Updated documentation with minimum permissions for cloud stack components
• Concurrent Execution: Added guidance for handling concurrent pipeline execution in separate containers
• User Guides: Enhanced user guides with llm-complete-guide project and improved examples
• ZenML Pro: Updated URLs in documentation to use zenml.io/pro
• VSCode Integration: Added VSCode tutorial pipeline testing to CI

What's Changed

• Update issue template configurations and URLs by @htahir1 in https://github.com/zenml-io/zenml/pull/3756
• Add new release to migration tests by @schustmi in https://github.com/zenml-io/zenml/pull/3764
• Split run submission and monitoring by @schustmi in https://github.com/zenml-io/zenml/pull/3723
• Made --no-verify-ssl work for pro tenants as well by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3767
• Threadsafe RestZenStore by @schustmi in https://github.com/zenml-io/zenml/pull/3758
• Server-side parent step computation by @schustmi in https://github.com/zenml-io/zenml/pull/3762
• Unified config imports to match docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3743
• Fixed misleading error message by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3759
• Adjusted settings to include submodules by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3757
• Switch from pkg_resources to importlib by @schustmi in https://github.com/zenml-io/zenml/pull/3722
• Helm chart: Add option to create PVC for local configuration path by @jsuchome in https://github.com/zenml-io/zenml/pull/3761
• Hide service connector secrets as internal implementation details by @stefannica in https://github.com/zenml-io/zenml/pull/3770
• Fix fetching model version by UUID passed as string by @schustmi in https://github.com/zenml-io/zenml/pull/3777
• Ensure visualizations are committed before potential transaction rollback by @schustmi in https://github.com/zenml-io/zenml/pull/3769
• Make SagemakerOrchestratorSettings.processor_tags optional by @schustmi in https://github.com/zenml-io/zenml/pull/3773
• Improved pipeline/step run fetching by @schustmi in https://github.com/zenml-io/zenml/pull/3776
• Misc CI fixes by @schustmi in https://github.com/zenml-io/zenml/pull/3780
• Improved Kubernetes orchestrator pod caching by @schustmi in https://github.com/zenml-io/zenml/pull/3719
• Compute cascading tags server-side by @schustmi in https://github.com/zenml-io/zenml/pull/3781
• Add llm-complete-guide project to user guides by @strickvl in https://github.com/zenml-io/zenml/pull/3788
• Docs: Handle concurrent pipeline execution in separate containers by @htahir1 in https://github.com/zenml-io/zenml/pull/3799
• Ability to stop pipelines early + Kubernetes Orchestrator implementation by @bcdurak in https://github.com/zenml-io/zenml/pull/3716
• Updated docs with minimum permissions sets for cloud stack components by @htahir1 in https://github.com/zenml-io/zenml/pull/3796
• Easier option to enable a step operator/experiment tracker for a step by @schustmi in https://github.com/zenml-io/zenml/pull/3774
• Always build pipeline image for Kubernetes orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/3797
• Add VSCode tutorial pipeline testing to CI by @strickvl in https://github.com/zenml-io/zenml/pull/3737
• Update best practices for upgrading ZenML servers by @htahir1 in https://github.com/zenml-io/zenml/pull/3795
• Orchestrator pod logs for Kubernetes by @bcdurak in https://github.com/zenml-io/zenml/pull/3778
• Warn if passing invalid kubernetes model data by @schustmi in https://github.com/zenml-io/zenml/pull/3805
• Update ZenML Pro URLs in documentation to use zenml.io/pro by @htahir1 in https://github.com/zenml-io/zenml/pull/3813
• Update README.md by @strickvl in https://github.com/zenml-io/zenml/pull/3783
• Fix data artifact fetching by @schustmi in https://github.com/zenml-io/zenml/pull/3811
• Step retries by @schustmi in https://github.com/zenml-io/zenml/pull/3789
• Use pydantic validation aliases to support step/parameter names with leading underscores by @schustmi in https://github.com/zenml-io/zenml/pull/3814
• New zenml login and CLI style improvements by @znegrin in https://github.com/zenml-io/zenml/pull/3790
• Onboarding changes by @schustmi in https://github.com/zenml-io/zenml/pull/3817
• Add ignore list for tutorial pipeline testing script by @strickvl in https://github.com/zenml-io/zenml/pull/3808
• Add missing remote path sanitizations by @schustmi in https://github.com/zenml-io/zenml/pull/3816
• Tiny README fixes by @strickvl in https://github.com/zenml-io/zenml/pull/3818
• Add docker tag sanitization by @schustmi in https://github.com/zenml-io/zenml/pull/3820
• Add field descriptions to config for forms by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3779
• Step Status Refresh Functionality + Kubernetes Orchestrator Implementation by @bcdurak in https://github.com/zenml-io/zenml/pull/3735
• Upper limit datasets version by @schustmi in https://github.com/zenml-io/zenml/pull/3824

Full Changelog: https://github.com/zenml-io/zenml/compare/0.83.1...0.84.0

- Python
Published by schustmi 8 months ago

https://github.com/zenml-io/zenml - 0.83.1

The 0.83.1 release delivers significant performance improvements and optimizations that enhance ZenML's efficiency and responsiveness. This release focuses on architectural improvements including step configuration separation from deployment, idempotent POST requests with request caching, and enhanced artifact loading performance, alongside important bug fixes and documentation updates.

⚡ Performance Improvements

• Architectural Enhancement: Separated step configurations from deployment for improved modularity and reduced overhead
• Request Optimization: Implemented idempotent POST requests with intelligent request caching for faster API responses
• Artifact Loading: Enhanced artifact loading performance by using the active artifact store as cache

Fixes

• Fixed potential race condition during pipeline run status updates
• Fixed missing artifact nodes in DAG visualization
• Fixed run template fetching issues that could cause errors during pipeline execution
• Fixed missing pipeline substitutions for legacy steps
• Reset active project after deleting local database

Documentation

• Updated Kubernetes configuration documentation with clearer guidance
• Enhanced run template documentation with better examples and explanations
• Added shared CSS guidance for HTML visualizations
• Updated kaniko project status documentation
• Fixed authentication confusion in ZenML Pro API documentation

What's Changed

• More explicit handling of carriage return logs in steps by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3714
• Add 0.82.1 to the legacy docs by @bcdurak in https://github.com/zenml-io/zenml/pull/3715
• Reset active project after deleting local database by @schustmi in https://github.com/zenml-io/zenml/pull/3689
• Keep visualizations when using existing tag by @schustmi in https://github.com/zenml-io/zenml/pull/3710
• Use the active artifact store as cache when loading artifacts by @stefannica in https://github.com/zenml-io/zenml/pull/3721
• Fix potential race condition during run status update by @schustmi in https://github.com/zenml-io/zenml/pull/3720
• Random cleanup and bugfixes by @schustmi in https://github.com/zenml-io/zenml/pull/3709
• Added documentation for a PowerShell quirk when referencing secrets in experiment tracker by @hala201 in https://github.com/zenml-io/zenml/pull/3717
• Fix some links by @htahir1 in https://github.com/zenml-io/zenml/pull/3729
• Fix run template fetching by @schustmi in https://github.com/zenml-io/zenml/pull/3726
• Fix missing artifact nodes in DAG by @schustmi in https://github.com/zenml-io/zenml/pull/3727
• Revert "Keep visualizations when using existing tag" by @stefannica in https://github.com/zenml-io/zenml/pull/3732
• Tag improvements by @bcdurak in https://github.com/zenml-io/zenml/pull/3724
• Improve docs imports by @strickvl in https://github.com/zenml-io/zenml/pull/3731
• Kubernetes configuration documentation update by @bcdurak in https://github.com/zenml-io/zenml/pull/3736
• Fix authentication confusion in ZenML Pro API documentation by @strickvl in https://github.com/zenml-io/zenml/pull/3734
• Include missing pipeline substitutions for legacy steps by @schustmi in https://github.com/zenml-io/zenml/pull/3740
• Better run template docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3730
• Add shared CSS guidance for HTML visualizations by @strickvl in https://github.com/zenml-io/zenml/pull/3742
• Update kaniko project status in docs by @strickvl in https://github.com/zenml-io/zenml/pull/3744
• Fix line breaks in llms.txt docs by @strickvl in https://github.com/zenml-io/zenml/pull/3746
• Separate step configurations from deployment by @schustmi in https://github.com/zenml-io/zenml/pull/3739
• Dynamically import entrypoint in root init by @schustmi in https://github.com/zenml-io/zenml/pull/3728
• Idempotent POST requests, request caching and other performance improvements by @stefannica in https://github.com/zenml-io/zenml/pull/3738

New Contributors

• @hala201 made their first contribution in https://github.com/zenml-io/zenml/pull/3717

Full Changelog: https://github.com/zenml-io/zenml/compare/0.83.0...0.83.1

- Python
Published by schustmi 8 months ago

https://github.com/zenml-io/zenml - 0.83.0

🚀 Major Performance Release - ZenML 0.83.0 introduces significant performance improvements and response optimizations that dramatically reduce API response times and database query overhead. This release includes breaking changes and is not compatible with earlier client/server versions.

⚡ Performance Improvements

This release delivers substantial performance enhancements across the ZenML server:

• Optimized API Responses: Pipeline run responses no longer include unpaginated step lists, dramatically reducing response sizes for large pipelines
• Database Query Optimization: Improved query patterns with strategic joined loads to minimize database roundtrips
• Reduced Response Payloads: Many attributes moved from body to resources to avoid unnecessary data transfer when objects are embedded in other responses

🔄 Breaking Changes

⚠️ Client/Server Compatibility: This version is not compatible with earlier ZenML client/server versions. Please ensure both client and server are upgraded to 0.83.0.

API Response Changes

• Pipeline run responses no longer include metadata.steps or metadata.step_substitutions
• Many model attributes moved from body to resources for performance
• Project metadata structure simplified across all responses
• Model version responses no longer include comprehensive artifact and run ID lists

Method Deprecations

• Model.get_pipeline_run(...) and ModelVersionResponse.get_pipeline_run(...) have been removed

For a comprehensive list of all response changes and migration details, see PR #3675.

🚀 Orchestrator Enhancements

Kubernetes Orchestrator

• Enhanced Caching: Added caching capabilities in Kubernetes orchestrator entrypoint to improve performance and reduce unnecessary pod creations

Skypilot Orchestrator

• Updated Version and Settings: Updated to new Skypilot version and added new setting options

What's Changed

• Adding 0.82.0 to the legacy docs by @bcdurak in https://github.com/zenml-io/zenml/pull/3671
• New deployment scenarios by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3666
• Add instruction to WANDB_DISABLED to True for the quickstart by @bcdurak in https://github.com/zenml-io/zenml/pull/3673
• Fix service connector docs example by @strickvl in https://github.com/zenml-io/zenml/pull/3679
• Fix YAML extension check by @strickvl in https://github.com/zenml-io/zenml/pull/3677
• Fix typos, bugs, and improve test precision by @strickvl in https://github.com/zenml-io/zenml/pull/3678
• Performance boost fix: don't fetch entire pipeline run to verify pipeline API token validity by @stefannica in https://github.com/zenml-io/zenml/pull/3684
• Async wrapper for FastAPI endpoints to run serialization in event loop by @schustmi in https://github.com/zenml-io/zenml/pull/3685
• Don't log parent image digest warning if build is skipped by @schustmi in https://github.com/zenml-io/zenml/pull/3676
• Convert string to raw to avoid warnings in python 3.12+ by @jlopezpena in https://github.com/zenml-io/zenml/pull/3687
• Use correct artifact store for nested materializers by @schustmi in https://github.com/zenml-io/zenml/pull/3670
• Tiny Discord docs fix by @strickvl in https://github.com/zenml-io/zenml/pull/3691
• Store validated config with converted types in DB by @schustmi in https://github.com/zenml-io/zenml/pull/3668
• Added zenml codespace env detection by @htahir1 in https://github.com/zenml-io/zenml/pull/3686
• Fix setuptools vulnerabilities and deprecate pip as DockerSettings default by @stefannica in https://github.com/zenml-io/zenml/pull/3694
• Improve alerter documentation with comprehensive ask step coverage by @strickvl in https://github.com/zenml-io/zenml/pull/3693
• Add comprehensive agent guidelines with AGENTS.md and CLAUDE.md by @strickvl in https://github.com/zenml-io/zenml/pull/3680
• Performance test utilities, stats and stress test pipeline updates by @stefannica in https://github.com/zenml-io/zenml/pull/3690
• Fix GCP service connector expiry by @stefannica in https://github.com/zenml-io/zenml/pull/3697
• Fix trivy image scanning GitHub actions by @stefannica in https://github.com/zenml-io/zenml/pull/3699
• Add ability to strip the timestamps from the logs, on request by @avishniakov in https://github.com/zenml-io/zenml/pull/3683
• Send POST request for RBAC permission checks to avoid URL length limits by @schustmi in https://github.com/zenml-io/zenml/pull/3702
• Response and database improvements by @schustmi in https://github.com/zenml-io/zenml/pull/3675
• Add Caching in Kubernetes orchestrator entrypoint by @schustmi in https://github.com/zenml-io/zenml/pull/3703
• New Pro onboarding by @schustmi in https://github.com/zenml-io/zenml/pull/3704
• Suppress repeated DockerSettings warnings by @stefannica in https://github.com/zenml-io/zenml/pull/3705
• New OSS dashboard fixes by @schustmi in https://github.com/zenml-io/zenml/pull/3706
• Remove unused import from metadata docs by @strickvl in https://github.com/zenml-io/zenml/pull/3707
• Project consistency fixes for Model and ModelVersionResponse by @schustmi in https://github.com/zenml-io/zenml/pull/3708
• Update Skypilot orchestrator settings and features by @htahir1 in https://github.com/zenml-io/zenml/pull/3612

Full Changelog: https://github.com/zenml-io/zenml/compare/0.82.1...0.83.0

- Python
Published by schustmi 9 months ago

https://github.com/zenml-io/zenml - 0.82.1

The 0.82.1 release focuses on incremental improvements to run template management, Kubernetes orchestration, Docker build performance, and overall robustness, while shipping a wide range of documentation updates and quality-of-life enhancements. Key highlights include configurable Kubernetes job clean-up behavior, cascading tags for cached step runs, pyproject.toml support for DockerSettings, improved login stability, and faster Docker build caching through parent image digests.

Features

• Added pyproject.toml support for configuring DockerSettings, making container builds easier to manage.
• Added a unique instance label to the Helm chart to simplify the operation of multiple ZenML deployments in the same cluster.
• Introduced a new stress-test example that showcases ZenML scalability and can be used to benchmark installations.

Improvements

• Added cascading tags for cached step runs to improve cache reuse and pipeline run performance.
• Added configurable Kubernetes job clean-up options for the Kubernetes orchestrator.
• Added a limit to the maximum number of concurrent template runs and improved the overall run template UX.
• Prevented unnecessary hydration in project-scoped API responses, reducing payload size and improving performance.
• Optimized Docker build caching by using parent image digests and extending development Dockerfiles.
• Pinned setuptools to a stable version and removed redundant script dependencies to avoid build failures.

Fixes

• Fixed DockerHub repository digest detection when building images.
• Fixed miscellaneous login issues and introduced an API login lock for added robustness.
• Fixed dashboard bolt icon rendering.
• Updated Alembic to address compatibility issues.

Documentation

• Added a "5-minute quick wins" guide and a new dedicated docs section regarding orchestrator selection.
• Added documentation for dashboard features and an accurate list of workload manager options.
• Added 0.81.0 to legacy docs and fixed artifact visualization guidance.
• Numerous minor documentation fixes and cleanup.

What's Changed

• Adding 0.81.0 to the legacy docs by @bcdurak in https://github.com/zenml-io/zenml/pull/3630
• Extending the Dockerfiles by @bcdurak in https://github.com/zenml-io/zenml/pull/3632
• Use parent image digest for cache invalidation by @schustmi in https://github.com/zenml-io/zenml/pull/3617
• Pro API login lock and other robustness improvements by @stefannica in https://github.com/zenml-io/zenml/pull/3625
• Fixing images for the Hello World and various other fixes by @bcdurak in https://github.com/zenml-io/zenml/pull/3637
• Pin setuptools and remove it from scripts by @stefannica in https://github.com/zenml-io/zenml/pull/3636
• Update link validation to skip GitHub links and improve progress tracking by @htahir1 in https://github.com/zenml-io/zenml/pull/3641
• Added docs section by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3640
• Various improvements to the release flow by @bcdurak in https://github.com/zenml-io/zenml/pull/3638
• Fix Dockerhub repo digest detection by @schustmi in https://github.com/zenml-io/zenml/pull/3621
• Add unique instance label to helm chart by @stefannica in https://github.com/zenml-io/zenml/pull/3639
• Slight doc fix. Fixes #3645 by @htahir1 in https://github.com/zenml-io/zenml/pull/3646
• Add 5-min quick wins page to docs by @strickvl in https://github.com/zenml-io/zenml/pull/3633
• Limit max concurrent template runs by @schustmi in https://github.com/zenml-io/zenml/pull/3627
• Fix bolt icon by @strickvl in https://github.com/zenml-io/zenml/pull/3648
• Update docs with accurate list of workload manager options by @stefannica in https://github.com/zenml-io/zenml/pull/3643
• Workflow to deploy workspaces for PRs by @bcdurak in https://github.com/zenml-io/zenml/pull/3618
• Format link checker by @schustmi in https://github.com/zenml-io/zenml/pull/3656
• Improve run template UX by @schustmi in https://github.com/zenml-io/zenml/pull/3602
• Prevent unnecessary hydration in project-scoped responses by @schustmi in https://github.com/zenml-io/zenml/pull/3657
• Update alembic version to "^1.8.1" in pyproject.toml by @htahir1 in https://github.com/zenml-io/zenml/pull/3529
• Fixing artifact visualization docs by @bcdurak in https://github.com/zenml-io/zenml/pull/3661
• Add stress test example by @stefannica in https://github.com/zenml-io/zenml/pull/3663
• Misc login fixes by @schustmi in https://github.com/zenml-io/zenml/pull/3654
• Report run template trigger usage by @schustmi in https://github.com/zenml-io/zenml/pull/3659
• Specify end date for template runs that failed during spinup by @schustmi in https://github.com/zenml-io/zenml/pull/3664
• Make Kubernetes job cleanup options configurable by @schustmi in https://github.com/zenml-io/zenml/pull/3644
• Dashboard features documentation by @strickvl in https://github.com/zenml-io/zenml/pull/3662
• Pyproject.toml support for DockerSettings by @schustmi in https://github.com/zenml-io/zenml/pull/3292
• Utilizing cascading tags for cached step runs by @bcdurak in https://github.com/zenml-io/zenml/pull/3655

Full Changelog: https://github.com/zenml-io/zenml/compare/0.82.0...0.82.1

- Python
Published by bcdurak 9 months ago

https://github.com/zenml-io/zenml - 0.82.0

The 0.82.0 release delivers significant improvements to Kubernetes orchestrator, enhanced documentation, and numerous fixes to improve overall stability and performance. Key highlights include configurable max parallelism for Kubernetes orchestrator, customizable pod name prefixes and scheduler options, improved runner timeouts, and support for private service connections in Vertex AI. This release also includes comprehensive documentation updates, and library compatibility improvements for NumPy and Pandas.

Features

• Added max parallelism option for Kubernetes orchestrator
• Added support for pod name prefixes and scheduler configuration
• Added private service connect option for Vertex AI orchestrator
• Made runner timeout easily configurable
• Added storage for list of Python packages
• Added an ability to deep refresh the status of your run from the dashboard

Improvements

• Adjusted GitHub code repo regex pattern for better compatibility
• Improved build invalidation when parent Dockerfile changes
• Enhanced directory handling during code download
• Added ability to list model versions without models
• Added support for extra attributes in all ZenML models
• Disabled default project behavior for pro workspaces

Fixes

• Fixed run templates listing
• Eliminated premature active project warning logs
• Updated scikit-learn requirement in SklearnIntegration
• Updated NumPy integration to work with both 1.x and 2.x library versions
• Added Pandas custom data type error handling and logging
• Removed unnecessary and invalid settings
• Various frontend bug fixes

Documentation

• Completed comprehensive documentation revamp
• Added documentation for self-hosted run templates
• Removed outdated redirects and sections
• Fixed broken absolute links and restored missing sections
• Added documentation for run template TTL environment variable
• Updated image paths for ZenML pipeline screenshots
• Migrated starter guide to unified log_metadata method

Breaking Changes

• Removed scikit-image as a requirement of the sklearn integration

What's Changed

• Adding 0.80.2 to legacy docs by @bcdurak in https://github.com/zenml-io/zenml/pull/3547
• Add 0.81.0 to migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3546
• Document self-hosted run templates by @stefannica in https://github.com/zenml-io/zenml/pull/3552
• Adjusted github code repo regex pattern matching to work in more cases by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3550
• Documentation revamp by @bcdurak in https://github.com/zenml-io/zenml/pull/3524
• Remove outdated redirects and sections by @htahir1 in https://github.com/zenml-io/zenml/pull/3564
• Removed orphan assets by @htahir1 in https://github.com/zenml-io/zenml/pull/3567
• Fixed broken absolute links and brought back some sections by @htahir1 in https://github.com/zenml-io/zenml/pull/3585
• Fix listing run templates by @schustmi in https://github.com/zenml-io/zenml/pull/3587
• Don't log missing active project warnings preemtively by @schustmi in https://github.com/zenml-io/zenml/pull/3586
• Docs fixes by @strickvl in https://github.com/zenml-io/zenml/pull/3565
• More docs changes by @htahir1 in https://github.com/zenml-io/zenml/pull/3592
• Update scikit-learn requirement in SklearnIntegration by @htahir1 in https://github.com/zenml-io/zenml/pull/3551
• Small docs fixes by @strickvl in https://github.com/zenml-io/zenml/pull/3584
• More small docs fixes by @strickvl in https://github.com/zenml-io/zenml/pull/3601
• Docs: Migrate to unified log_metadata method in starter guide by @marwan37 in https://github.com/zenml-io/zenml/pull/3603
• Add docs for run template TTL env var by @schustmi in https://github.com/zenml-io/zenml/pull/3604
• Make runner timeout easily configurable by @schustmi in https://github.com/zenml-io/zenml/pull/3589
• Disable default project behavior for pro workspaces by @schustmi in https://github.com/zenml-io/zenml/pull/3605
• Remove unnecessary and invalid settings by @schustmi in https://github.com/zenml-io/zenml/pull/3548
• Update image paths for ZenML pipeline screenshots by @htahir1 in https://github.com/zenml-io/zenml/pull/3611
• Update NumPy integration to work with 1.x and 2.x versions of the library by @htahir1 in https://github.com/zenml-io/zenml/pull/3532
• Invalidate the build when parent dockerfile changes by @schustmi in https://github.com/zenml-io/zenml/pull/3607
• Ignore existing directories during code download by @schustmi in https://github.com/zenml-io/zenml/pull/3609
• Max parallelism option for kubernetes orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/3606
• Add option to specify pod name prefix and add scheduler by @schustmi in https://github.com/zenml-io/zenml/pull/3591
• Fix CVE-2024-6345 CVE-2024-6345 CVE-2022-40897 and CVE-2025-0508 by @stefannica in https://github.com/zenml-io/zenml/pull/3610
• Docs: Starter guide fixes by @marwan37 in https://github.com/zenml-io/zenml/pull/3616
• Remove some unused code by @schustmi in https://github.com/zenml-io/zenml/pull/3614
• Add option to set a private service connect to the Vertex AI orchestrator by @stefannica in https://github.com/zenml-io/zenml/pull/3613
• Add Pandas custom data type error handling and logging by @htahir1 in https://github.com/zenml-io/zenml/pull/3539
• Allowing listing model versions without models by @bcdurak in https://github.com/zenml-io/zenml/pull/3593
• Store list of python packages by @schustmi in https://github.com/zenml-io/zenml/pull/3608
• Allow (and ignore) extra attributes in all ZenML models by @stefannica in https://github.com/zenml-io/zenml/pull/3620
• Utilizing uv in the quickstart dockerfiles by @bcdurak in https://github.com/zenml-io/zenml/pull/3623
• Fixing the azure integration and marshmallow by @bcdurak in https://github.com/zenml-io/zenml/pull/3626

Full Changelog: https://github.com/zenml-io/zenml/compare/0.81.0...0.82.0

- Python
Published by bcdurak 10 months ago

https://github.com/zenml-io/zenml - 0.81.0

The 0.81.0 release focuses on significant improvements to artifact management, enhanced security features, and advanced resource sharing capabilities. Key highlights include a new Path materializer for directory and file uploads, artifact visualization enhancements, client-side pipeline run logs storage, and support for resource sharing between teams and external users. This release also improves logging, adds security protections, and enhances orchestration capabilities across multiple cloud platforms.

## Features

• Added a Path materializer to upload directories and files
• Added save_visualizations method to materializers
• Implemented ZenML artifact store provisioning scripts for AWS/GCP/Azure
• Added endpoint to download artifact data
• Enabled resource sharing for teams and external users
• Added support for custom job parameters in Vertex orchestrator
• Added client-side pipeline run logs storage in the artifact store

## Improvements

• Improved logging with step ID instead of name and avoided extra newlines
• Added tracking for stack creation and pipeline runs with remote artifact store
• Implemented failure retries and pending timeouts for the k8s orchestrator pod
• Added configurable workload token expiration leeway
• Migrated onboarding state for better user experience

## Fixes

• Fixed artifact control plane vs model control plane GIF links in docs
• Used step invocation ID in run template error message
• Prevented path traversal attacks during file extraction
• Fixed missing vertex orchestrator dependencies in server image
• Resolved pydantic 2.11 deprecation warnings
• Removed duplicate step name from logs

## Documentation

• Added triggering pipelines mini-tutorial
• Improved SDK Client documentation with thematic grouping
• Added Kubernetes schedule documentation and fixed heading hierarchy

## What's Changed * Added a Path materializer to upload directories and files by @htahir1 in https://github.com/zenml-io/zenml/pull/3496 * Avoid extra newlines in logs by @avishniakov in https://github.com/zenml-io/zenml/pull/3518 * Use step ID instead of name in step logs by @schustmi in https://github.com/zenml-io/zenml/pull/3519 * Fix artifact control plane vs model control plane GIF links in docs by @strickvl in https://github.com/zenml-io/zenml/pull/3508 * Adding 0.80.1 to the legacy docs by @bcdurak in https://github.com/zenml-io/zenml/pull/3521 * Adding 0.80.2 to the migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3517 * Enabling the greatexpectations test again by @bcdurak in https://github.com/zenml-io/zenml/pull/3516 * Add save_visualizations method to materializers by @htahir1 in https://github.com/zenml-io/zenml/pull/3520 * Add triggering pipelines mini-tutorial by @strickvl in https://github.com/zenml-io/zenml/pull/3495 * ZenML artifact store provisioning scripts for AWS/GCP/Azure by @stefannica in https://github.com/zenml-io/zenml/pull/3514 * Improve SDK Client documentation with thematic grouping by @strickvl in https://github.com/zenml-io/zenml/pull/3528 * Track stack creation and pipeline run with remote artifact store for onboarding state by @schustmi in https://github.com/zenml-io/zenml/pull/3525 * Add Kubernetes schedule documentation and fix heading hierarchy by @strickvl in https://github.com/zenml-io/zenml/pull/3530 * Use step invocation ID in run template error message by @schustmi in https://github.com/zenml-io/zenml/pull/3534 * Migrate onboarding state by @schustmi in https://github.com/zenml-io/zenml/pull/3533 * Prevent path traversal attacks during file extraction by @htahir1 in https://github.com/zenml-io/zenml/pull/3535 * Implement failure retries and pending timeouts for the k8s orchestrator pod by @stefannica in https://github.com/zenml-io/zenml/pull/3523 * Add endpoint to download artifact data by @schustmi in https://github.com/zenml-io/zenml/pull/3507 * Enable resource sharing for teams and external users by @schustmi in https://github.com/zenml-io/zenml/pull/3461 * Add support for custom job parameters in Vertex orchestrator by @htahir1 in https://github.com/zenml-io/zenml/pull/3536 * Add configurable workload token expiration leeway by @schustmi in https://github.com/zenml-io/zenml/pull/3526 * Fix llms.txt docs by @strickvl in https://github.com/zenml-io/zenml/pull/3540 * Store client-side pipeline run logs in the artifact store by @stefannica in https://github.com/zenml-io/zenml/pull/3498 * Add missing vertex orchestrator dependencies to server image by @schustmi in https://github.com/zenml-io/zenml/pull/3541 * Resolve pydantic 2.11 deprecation warnings by @schustmi in https://github.com/zenml-io/zenml/pull/3542 * Remove duplicate step name from logs by @schustmi in https://github.com/zenml-io/zenml/pull/3543

Full Changelog: https://github.com/zenml-io/zenml/compare/0.80.2...0.81.0

- Python
Published by bcdurak 10 months ago

https://github.com/zenml-io/zenml - 0.80.2

The 0.80.2 release focuses on several bug fixes, usability enhancements, and documentation improvements. Key highlights include fixing the yanked fsspec pip resolution issue, adding sample visualization for Pandas materializer, and upgrading pydantic to 2.11 for improved performance. This release also adds new tutorials and guides, fixes several critical issues, and introduces new functionality for pipeline development.

## Features

• Added poetry add support for CLI
• Added retry option on step.with_options
• Added service annotations to helm chart
• Added step name to console logs for better tracing

## Improvements

• Added sample visualization for Pandas materializer
• Improved run templates with small enhancements and hidden option support
• Fixed service connector list-resources endpoint and CLI command

## Fixes

• Fixed timestamps in logs
• Fixed yanked fsspec pip resolution issue
• Fixed Kubernetes Orchestrator step pod failure status updates
• Fixed CI failures and test issues
• Updated error message for setting pipeline parameters via API

## Documentation

• Added new ZenML Tutorials section to user guide
• Added mini-tutorials for pipeline scheduling and organization with tags/projects
• Updated documentation for MCP Server and VSCode Extension
• Fixed various documentation issues

## What's Changed

• Updating the missing legacy docs by @bcdurak in https://github.com/zenml-io/zenml/pull/3473
• add new version to the migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3471
• Add docs for MCP Server and VSCode Extension by @strickvl in https://github.com/zenml-io/zenml/pull/3475
• Log job completion for Vertex orchestrator by @strickvl in https://github.com/zenml-io/zenml/pull/3478
• Cleaning up the CI failures by @bcdurak in https://github.com/zenml-io/zenml/pull/3480
• Remove remaining use of tj-actions/changed-files by @stefannica in https://github.com/zenml-io/zenml/pull/3481
• Fixing the timestamps in logs by @bcdurak in https://github.com/zenml-io/zenml/pull/3479
• Fix incorrect step access method in documentation by @marwan37 in https://github.com/zenml-io/zenml/pull/3483
• Relaxing the mlflow dependency by @bcdurak in https://github.com/zenml-io/zenml/pull/3446
• Add two mini-tutorials (pipeline scheduling and organization with tags/projects) by @strickvl in https://github.com/zenml-io/zenml/pull/3485
• Add ZenML Tutorials section to user guide by @htahir1 in https://github.com/zenml-io/zenml/pull/3488
• Add service annotations to helm chart by @jsybel in https://github.com/zenml-io/zenml/pull/3482
• Adding step name to console logs by @bcdurak in https://github.com/zenml-io/zenml/pull/3486
• Fix yanked fsspec pip resolution issue by @stefannica in https://github.com/zenml-io/zenml/pull/3493
• Fixed some docs here and there by @htahir1 in https://github.com/zenml-io/zenml/pull/3484
• Update error message for setting pipeline parameters via API by @htahir1 in https://github.com/zenml-io/zenml/pull/3491
• Fix the service connector list-resources endpoint and CLI command by @stefannica in https://github.com/zenml-io/zenml/pull/3492
• Restructure organization viewer role in ZenML Pro by @htahir1 in https://github.com/zenml-io/zenml/pull/3501
• Add a sample visualization for Pandas materializer by @htahir1 in https://github.com/zenml-io/zenml/pull/3494
• Add retry option on step.with_options by @schustmi in https://github.com/zenml-io/zenml/pull/3499
• Fix tutorials + small improvements to docs by @htahir1 in https://github.com/zenml-io/zenml/pull/3489
• CI fixes with huggingface and crud tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3502
• Upgrading pydantic to 2.11 by @bcdurak in https://github.com/zenml-io/zenml/pull/3472
• Update step status if Kubernetes Orchestrator step pods fail by @stefannica in https://github.com/zenml-io/zenml/pull/3497
• Add poetry add support for CLI by @avishniakov in https://github.com/zenml-io/zenml/pull/3470
• Small run template improvements by @schustmi in https://github.com/zenml-io/zenml/pull/3503
• Move uv env var to constants by @schustmi in https://github.com/zenml-io/zenml/pull/3504
• Add hidden option for run templates by @schustmi in https://github.com/zenml-io/zenml/pull/3500
• Fix security check for poetry by @bcdurak in https://github.com/zenml-io/zenml/pull/3509
• Minor auth improvement by @schustmi in https://github.com/zenml-io/zenml/pull/3512

## New Contributors

• @marwan37 made their first contribution in https://github.com/zenml-io/zenml/pull/3483
• @jsybel made their first contribution in https://github.com/zenml-io/zenml/pull/3482

Full Changelog: https://github.com/zenml-io/zenml/compare/0.80.1...0.80.2

- Python
Published by bcdurak 11 months ago

https://github.com/zenml-io/zenml - 0.80.1

The 0.80.1 release focuses on bug fixes and performance improvements following the major 0.80.0 update. This release addresses several critical issues, particularly improving the CLI functionality when used with the REST API through a deployed ZenML instance. Additionally, this version introduces a restructured documentation architecture for improved user experience.

Improvements

• Import integrations lazily for better performance
• Added ability to store a default project for users

Fixes

• Fixed CLI combined with RestZenStore and filters with multiple entries
• Fixed stack validation for incluster Kubernetes orchestrator
• Fixed stack and component URL when connected to a cloud workspace
• Fixed code repository host fallback
• Fixed version validation
• Various other minor bugfixes

Documentation

• Restructured entire documentation for better organization
• Fixed broken links in API documentation
• Refined logging to debug level for service connectors
• Removed redundant log messages

What's Changed

• Adding 0.80.0 to the migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3442
• Adding the disabled flavor test back by @bcdurak in https://github.com/zenml-io/zenml/pull/3431
• Stop CLI profiler running so much by @strickvl in https://github.com/zenml-io/zenml/pull/3449
• Add missing fallback host for code repositories by @schustmi in https://github.com/zenml-io/zenml/pull/3434
• Fix stack and component URL when connected to a cloud workspace by @schustmi in https://github.com/zenml-io/zenml/pull/3451
• Fix stack validation for incluster Kubernetes orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/3450
• Bump click dependency by @strickvl in https://github.com/zenml-io/zenml/pull/3445
• Fix 0.80.0 database migration by @stefannica in https://github.com/zenml-io/zenml/pull/3453
• Pin the ZenML Terraform provider version by @stefannica in https://github.com/zenml-io/zenml/pull/3443
• Import integrations lazily by @stefannica in https://github.com/zenml-io/zenml/pull/3419
• Remove Segment analytics script and scarf image load. by @htahir1 in https://github.com/zenml-io/zenml/pull/3455
• Restructure entire docs by @htahir1 in https://github.com/zenml-io/zenml/pull/3447
• Refactor logging to debug level for service connectors by @htahir1 in https://github.com/zenml-io/zenml/pull/3456
• Removing redundant log messages by @bcdurak in https://github.com/zenml-io/zenml/pull/3459
• Fix broken link in API documentation table by @htahir1 in https://github.com/zenml-io/zenml/pull/3462
• Add the ability to store a default project for a user by @schustmi in https://github.com/zenml-io/zenml/pull/3457
• Fixing the CLI combined with RestZenStore and filters with multiple entries by @bcdurak in https://github.com/zenml-io/zenml/pull/3464
• Don't ask for active project when listing projects by @stefannica in https://github.com/zenml-io/zenml/pull/3466
• Use the build python version to collect stack requirements for run templates by @stefannica in https://github.com/zenml-io/zenml/pull/3465
• Fix version validation by @bcdurak in https://github.com/zenml-io/zenml/pull/3467

Full Changelog: https://github.com/zenml-io/zenml/compare/0.80.0...0.80.1

- Python
Published by bcdurak 11 months ago

https://github.com/zenml-io/zenml - 0.80.0

The 0.80.0 release is one of our biggest updates in a while! This version introduces a major refactoring of workspaces into projects, enhances tagging capabilities, and improves GitLab repository support. This release also features significant performance optimizations for Docker builds and CLI operations.

Features

• For our Pro users: Refactored workspaces into projects with improved RBAC API resource format (We will release separate docs on this soon.)
• Enhanced tagging system with resource type filtering and exclusive tag behavior
• Added persistent resource support for the Vertex orchestrator
• Store build duration information for better tracking
• Allow passing step artifacts to specify upstream steps
• Support for environment variables in KubernetesPodSettings

Improvements

• Updated devel dockerfiles to make rebuilds faster
• Improved CLI response time through optimized imports
• Allow registering public GitLab repositories without token
• Enable Weave integration in Wandb settings
• Allow the service account project ID to be overridden in the GCP service connector
• Pass API token as Kubernetes secret, allowing Kubernetess orchestrator to run workloads without exposing any sensitive API tokens in the environment

Fixes

• Fixed GitLab URL parsing and matching
• Corrected CLI command to describe flavors
• Fixed taggable filter model and filter models with multiple inputs
• Fixed project statistics endpoint and ZenML Pro project URLs
• Fixed the ACR support in the Azure service connector
• Resolved SkyPilot Orchestrator cluster name handling
• Fixed deprecation messages for GitHub code repository
• Don't retry REST API calls if runtime errors occur

Documentation

• Renamed API Docs to SDK Docs for clarity
• Fixed SDK docs rendering with proper directory structure and links
• Removed deprecated caveat from Kubernetes docs
• Various documentation fixes and clarifications

What's Changed

• Update devel dockerfiles to make rebuilds faster by @stefannica in https://github.com/zenml-io/zenml/pull/3385
• Deepchecks fix for the CI by @bcdurak in https://github.com/zenml-io/zenml/pull/3389
• Fixing the CI by @bcdurak in https://github.com/zenml-io/zenml/pull/3391
• Fixing the zenml login hint for separated names by @bcdurak in https://github.com/zenml-io/zenml/pull/3388
• bugfix: correctly parse and match Gitlab URLs by @dragosmc in https://github.com/zenml-io/zenml/pull/3392
• bugfix: pass iterator to gitlab by @dragosmc in https://github.com/zenml-io/zenml/pull/3393
• Allow registering public gitlab repositories without token by @schustmi in https://github.com/zenml-io/zenml/pull/3394
• Fix CLI command to describe flavors by @schustmi in https://github.com/zenml-io/zenml/pull/3390
• Store build duration by @schustmi in https://github.com/zenml-io/zenml/pull/3386
• Removed deprecated caveat from kubernetes docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3395
• fix doc confusion by @VicSev in https://github.com/zenml-io/zenml/pull/3397
• Improved tagging by @bcdurak in https://github.com/zenml-io/zenml/pull/3360
• Refactor workspaces into projects by @stefannica in https://github.com/zenml-io/zenml/pull/3364
• Fix taggable filter model by @schustmi in https://github.com/zenml-io/zenml/pull/3403
• Testing the CLI with the profiler by @bcdurak in https://github.com/zenml-io/zenml/pull/3400
• Don't retry REST API calls if runtime errors occur by @stefannica in https://github.com/zenml-io/zenml/pull/3408
• Allow the service account project ID to be overridden in the GCP service connector by @stefannica in https://github.com/zenml-io/zenml/pull/3398
• Allow passing step artifacts to specify upstream steps by @schustmi in https://github.com/zenml-io/zenml/pull/3401
• Add Reo Javascript snippet to main.html by @htahir1 in https://github.com/zenml-io/zenml/pull/3409
• Rename workspace to project by @stefannica in https://github.com/zenml-io/zenml/pull/3407
• Enable Weave integration in Wandb settings by @htahir1 in https://github.com/zenml-io/zenml/pull/3359
• Add persistent resource support for the vertex orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/3396
• Minor fix for the docs by @avishniakov in https://github.com/zenml-io/zenml/pull/3411
• Listing tags filtered by resource type by @bcdurak in https://github.com/zenml-io/zenml/pull/3406
• Adding removing tags with various update models by @bcdurak in https://github.com/zenml-io/zenml/pull/3404
• Exclusive tag behavior by @bcdurak in https://github.com/zenml-io/zenml/pull/3405
• Rename tenant to workspace and implement new RBAC API resource format by @stefannica in https://github.com/zenml-io/zenml/pull/3414
• API Docs -> SDK Docs by @htahir1 in https://github.com/zenml-io/zenml/pull/3415
• Removed step by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3416
• Fix unbound variable access by @schustmi in https://github.com/zenml-io/zenml/pull/3412
• Allow setting environment variables through KubernetesPodSettings by @schustmi in https://github.com/zenml-io/zenml/pull/3413
• Fix SDK docs rendering with proper directory structure and links by @strickvl in https://github.com/zenml-io/zenml/pull/3374
• Fix deprecation message for github code repository by @schustmi in https://github.com/zenml-io/zenml/pull/3418
• Fix project statistics endpoint by @schustmi in https://github.com/zenml-io/zenml/pull/3420
• Track project creation in onboarding state by @schustmi in https://github.com/zenml-io/zenml/pull/3423
• Fix the ACR support in the Azure service connector by @stefannica in https://github.com/zenml-io/zenml/pull/3424
• Limiting the mlflow dependency by @bcdurak in https://github.com/zenml-io/zenml/pull/3422
• Pass API token as kubernetes secret by @schustmi in https://github.com/zenml-io/zenml/pull/3421
• Improve the CLI response time through imports by @bcdurak in https://github.com/zenml-io/zenml/pull/3399
• Fetch model hydrated during deletion process by @schustmi in https://github.com/zenml-io/zenml/pull/3427
• Fix ZenML Pro project URLs for pipeline runs and model versions by @stefannica in https://github.com/zenml-io/zenml/pull/3426
• Add missing functions and classes to root init exports by @schustmi in https://github.com/zenml-io/zenml/pull/3428
• Fix doc links and comment test out by @htahir1 in https://github.com/zenml-io/zenml/pull/3430
• fix: SkypilotBaseOrchestrator handle given cluster_name and correct reuse by @BjoernBiltzinger in https://github.com/zenml-io/zenml/pull/3417
• Upgrading the skypilot dependency by @bcdurak in https://github.com/zenml-io/zenml/pull/3429
• Fixing filter models with multiple inputs by @bcdurak in https://github.com/zenml-io/zenml/pull/3410
• Add project usage tracking by @schustmi in https://github.com/zenml-io/zenml/pull/3435

New Contributors

• @dragosmc made their first contribution in https://github.com/zenml-io/zenml/pull/3392
• @VicSev made their first contribution in https://github.com/zenml-io/zenml/pull/3397
• @BjoernBiltzinger made their first contribution in https://github.com/zenml-io/zenml/pull/3417

Full Changelog: https://github.com/zenml-io/zenml/compare/0.75.0...0.80.0

- Python
Published by bcdurak 11 months ago

https://github.com/zenml-io/zenml - 0.75.0

The 0.75.0 release introduces dashboard enhancements for stack component management along with improvements to documentation and service connector capabilities. Users can now create and update stack components directly from the dashboard.

Features

• Create and update stack components directly from the dashboard
• Custom authentication method support during auto-configuration of service connectors
• Enhanced model artifact retrieval by creation date instead of version name
• Additional SageMaker environment settings

Improvements

• Expanded fastapi dependency range for better compatibility
• Improved pipeline source root documentation and logging
• Better sorting functionality when using custom fetching

Fixes

• Fixed registration of components with custom flavors
• Fixed sorting logic when using custom fetching criteria
• Prevented inner fsspec logs from being flushed to the artifact store

Documentation

• Added LLM messaging and video resources to documentation
• Improved formatting for model deployers documentation
• Fixed GCP service connector docs
• Added SDK documentation links
• Enhanced README with LLM messaging

What's Changed

• Adding 0.74.0 to the migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3351
• Fixing the release preparation workflow by @bcdurak in https://github.com/zenml-io/zenml/pull/3348
• Expand fastapi dependency range by @strickvl in https://github.com/zenml-io/zenml/pull/3340
• Document the programmatic API access options by @stefannica in https://github.com/zenml-io/zenml/pull/3352
• Fix some docs links by @schustmi in https://github.com/zenml-io/zenml/pull/3353
• [docs] Rename llms.txt file, add header and docs by @wjayesh in https://github.com/zenml-io/zenml/pull/3346
• Add llms.txt YouTube video to docs by @strickvl in https://github.com/zenml-io/zenml/pull/3354
• Fix model deployers docs formatting by @strickvl in https://github.com/zenml-io/zenml/pull/3356
• Get the latest artifact of a model by creation date instead of version name by @pierre-godard in https://github.com/zenml-io/zenml/pull/3343
• Improve source root docs/logs when running a pipeline by @schustmi in https://github.com/zenml-io/zenml/pull/3357
• Fix registration of components with custom flavors by @schustmi in https://github.com/zenml-io/zenml/pull/3363
• Fix GCP service connector docs by @stefannica in https://github.com/zenml-io/zenml/pull/3365
• Allow auth method to be customized during auto-configuration of service connectors by @stefannica in https://github.com/zenml-io/zenml/pull/3367
• Add some sdkdocs links by @htahir1 in https://github.com/zenml-io/zenml/pull/3358
• doc: fix link by @tanguyantoine in https://github.com/zenml-io/zenml/pull/3369
• Fix sorting when using custom fetching by @schustmi in https://github.com/zenml-io/zenml/pull/3366
• Add sagemaker env settings by @stefannica in https://github.com/zenml-io/zenml/pull/3368
• Update README with LLM messaging and llms.txt by @wjayesh in https://github.com/zenml-io/zenml/pull/3362
• CI Linting fix by @bcdurak in https://github.com/zenml-io/zenml/pull/3377
• Don't flush inner fsspec logs to the artifact store by @stefannica in https://github.com/zenml-io/zenml/pull/3373
• Bugfix for Sagemaker env variables by @bcdurak in https://github.com/zenml-io/zenml/pull/3380
• Correct isintance check for sagemaker env variables by @bcdurak in https://github.com/zenml-io/zenml/pull/3382

New Contributors

• @pierre-godard made their first contribution in https://github.com/zenml-io/zenml/pull/3343
• @tanguyantoine made their first contribution in https://github.com/zenml-io/zenml/pull/3369

Full Changelog: https://github.com/zenml-io/zenml/compare/0.74.0...0.75.0

- Python
Published by bcdurak 12 months ago

https://github.com/zenml-io/zenml - 0.74.0

The 0.74.0 release introduces several major features including SageMaker pipeline scheduling capabilities, Azure Container Registry (ACR) implicit authentication support, and Vertex AI persistent resource handling for step operators. Additionally, this release includes comprehensive improvements to timezone handling and significant enhancements to database performance.

Features

• API Tokens support in the dashboard for time-boxed API authentication
• SageMaker pipeline scheduling capabilities
• Azure Container Registry (ACR) and Storage Account implicit authentication
• Vertex AI persistent resource support for step operators
• Support for custom log formats
• Run metadata and tag indices for improved performance
• Core concepts video added to documentation

Improvements

• Comprehensive timezone consistency improvements across the platform
• Enhanced database query performance for pipelines, run templates, models, and artifacts
• Better handling of configured parameters during pipeline preparation
• Support for passing run configurations as dictionaries when triggering pipelines
• Enhanced sorting capabilities for columns with empty values in the dashboard
• Improved queries for pipelines, run templates, models, and artifacts
• Better filtering functionality for run metadata
• More efficient artifact filtering
• Various Helm chart improvements and reorganization
• Updated materializer support for newer PyTorch versions
• Improved code repository management and downloading
• Better handling of SecretStr values in store configurations

Fixes

• Kubernetes service connector issues resolved
• Fixed sorting for columns with potentially empty values
• Corrected timestamp utilization for better timezone consistency
• Resolved issues with vLLM pipeline config file usage
• Fixed code download functionality for custom flavor components
• Addressed various documentation and broken links
• Corrected MySQL database connection warnings
• Fixed issues with Vertex AI experiment tracker documentation

What's Changed

• Fix some docs by @htahir1 in https://github.com/zenml-io/zenml/pull/3302
• Replace deprecated datetime.utcnow() with datetime.now(timezone.utc) by @aiakide in https://github.com/zenml-io/zenml/pull/3265
• Adding the missing VertexAI experiment tracker docs by @bcdurak in https://github.com/zenml-io/zenml/pull/3308
• Create Sagemaker pipeline schedules if specified by @htahir1 in https://github.com/zenml-io/zenml/pull/3271
• Formatting by @schustmi in https://github.com/zenml-io/zenml/pull/3307
• Remove trailing slashes from zenml login URLs by @stefannica in https://github.com/zenml-io/zenml/pull/3312
• Fix Kubernetes service connector by @stefannica in https://github.com/zenml-io/zenml/pull/3313
• Add notes on missing features for on-prem ZenML Pro deployments by @stefannica in https://github.com/zenml-io/zenml/pull/3301
• Fix wrong warning log when directly connecting to MySQL DB by @schustmi in https://github.com/zenml-io/zenml/pull/3311
• Fix typo by @schustmi in https://github.com/zenml-io/zenml/pull/3316
• Minor fix for Sagemaker by @bcdurak in https://github.com/zenml-io/zenml/pull/3318
• Rework timestamp utilization for timezone consistency by @stefannica in https://github.com/zenml-io/zenml/pull/3314
• Add broken links checker by @htahir1 in https://github.com/zenml-io/zenml/pull/3305
• Schedule timezone fixes by @schustmi in https://github.com/zenml-io/zenml/pull/3315
• Misc code repository improvements by @schustmi in https://github.com/zenml-io/zenml/pull/3306
• Add core concepts video by @htahir1 in https://github.com/zenml-io/zenml/pull/3324
• Fix code download for custom flavor components by @schustmi in https://github.com/zenml-io/zenml/pull/3323
• Allow passing run configuration as dict when triggering pipelines by @schustmi in https://github.com/zenml-io/zenml/pull/3326
• Fix sorting by columns with potentially empty values by @schustmi in https://github.com/zenml-io/zenml/pull/3325
• Allow custom log formats by @schustmi in https://github.com/zenml-io/zenml/pull/3288
• Add vertex persistent resource to settings for step operator by @htahir1 in https://github.com/zenml-io/zenml/pull/3304
• Fix use of config file in vLLM pipelines by @wjayesh in https://github.com/zenml-io/zenml/pull/3322
• Fixing the CI with the new huggingface-hub version by @bcdurak in https://github.com/zenml-io/zenml/pull/3329
• Handling string values as SecretStrs in store configurations by @bcdurak in https://github.com/zenml-io/zenml/pull/3319
• More code repository improvements by @schustmi in https://github.com/zenml-io/zenml/pull/3327
• Fix materializer for new pytorch version by @schustmi in https://github.com/zenml-io/zenml/pull/3331
• Add some nicer docs by @htahir1 in https://github.com/zenml-io/zenml/pull/3328
• Add run metadata and tag indices by @schustmi in https://github.com/zenml-io/zenml/pull/3310
• Fix markdown link checker for external PRs by @schustmi in https://github.com/zenml-io/zenml/pull/3333
• feat: implement implicit authentication for ACR and Storage Account by @lukas-reining in https://github.com/zenml-io/zenml/pull/3274
• Add support for symlinks in GH download by @schustmi in https://github.com/zenml-io/zenml/pull/3332
• ZenML Helm chart improvements by @stefannica in https://github.com/zenml-io/zenml/pull/3320
• Move helm chart out of the source tree by @stefannica in https://github.com/zenml-io/zenml/pull/3338
• Add option to skip stack validation by @schustmi in https://github.com/zenml-io/zenml/pull/3337
• Improve queries for pipelines, run templates, models and artifacts by @schustmi in https://github.com/zenml-io/zenml/pull/3335
• Improve configured parameter detection when preparing pipeline by @schustmi in https://github.com/zenml-io/zenml/pull/3339
• Minor fix for the Artifact filter model by @bcdurak in https://github.com/zenml-io/zenml/pull/3334
• Allow (un)installing integrations with system-wide uv installations by @schustmi in https://github.com/zenml-io/zenml/pull/3342
• Fix filtering by run metadata by @schustmi in https://github.com/zenml-io/zenml/pull/3344

New Contributors

• @lukas-reining made their first contribution in https://github.com/zenml-io/zenml/pull/3274

Full Changelog: https://github.com/zenml-io/zenml/compare/0.73.0...0.74.0

- Python
Published by bcdurak about 1 year ago

https://github.com/zenml-io/zenml - 0.73.0

The 0.73.0 release contains various changes and improvements, but most importantly it introduces the support to deploy and enroll un-managed ZenML Pro tenants in the ZenML Pro control plane (Helm deployment options, secure enrollment, CSRF tokens) and other features necessary for self-hosted, multi-domain ZenML Pro installations.

Other Features

• Vertex AI experiment tracker integration
• Experiment comparison tooling
• Support for new Airflow KubernetesPodOperator import paths
• Updated Slack alerter implementation
• Independent memory resource configuration for migration pods in Helm charts

Improvements

• Added environment variable to allow non-ASCII characters in JSON dumps
• Removed gluon from MLflow log suppression list
• Enhanced resource reporting with automatic conversion
• Documentation updates for Kubeflow Pipelines and LLMs
• Various bugfixes for the ZenML dashboard

What's Changed

• On-prem Pro tenants: secure enrollment, CSRF tokens, and cross-domain authorization flow by @stefannica in https://github.com/zenml-io/zenml/pull/3264
• Fix the misc release actions by @schustmi in https://github.com/zenml-io/zenml/pull/3286
• Add 0.72.0 to the migration tests by @schustmi in https://github.com/zenml-io/zenml/pull/3285
• Fix links to Kubeflow Pipelines docs in kubeflow.md by @matemijolovic in https://github.com/zenml-io/zenml/pull/3289
• Add experiment comparison tool docs by @strickvl in https://github.com/zenml-io/zenml/pull/3287
• Fix broken links by @strickvl in https://github.com/zenml-io/zenml/pull/3291
• Add support for new Airflow KubernetesPodOperator import by @schustmi in https://github.com/zenml-io/zenml/pull/3295
• Updated Slack Alerter by @bcdurak in https://github.com/zenml-io/zenml/pull/3282
• Allow non-ASCII in JSON dump with env var by @Frank995 in https://github.com/zenml-io/zenml/pull/3257
• Remove gluon from mlflow log suppression list by @htahir1 in https://github.com/zenml-io/zenml/pull/3298
• Convert reportable resources if necessary by @schustmi in https://github.com/zenml-io/zenml/pull/3296
• Vertex AI Experiment Tracker Integration by @nkhusainov in https://github.com/zenml-io/zenml/pull/3260
• Document on-prem ZenML Pro deployments by @stefannica in https://github.com/zenml-io/zenml/pull/3294
• generate llms.txt for our docs by @wjayesh in https://github.com/zenml-io/zenml/pull/3273
• [helm] Independent setting of memory resources for migration pods by @wjayesh in https://github.com/zenml-io/zenml/pull/3281
• Prepare release 0.73.0 by @github-actions in https://github.com/zenml-io/zenml/pull/3300

New Contributors

• @matemijolovic made their first contribution in https://github.com/zenml-io/zenml/pull/3289
• @Frank995 made their first contribution in https://github.com/zenml-io/zenml/pull/3257
• @nkhusainov made their first contribution in https://github.com/zenml-io/zenml/pull/3260

Full Changelog: https://github.com/zenml-io/zenml/compare/0.72.0...0.73.0

- Python
Published by bcdurak about 1 year ago

https://github.com/zenml-io/zenml - 0.72.0

The 0.72.0 release contains various bug fixes, performance improvements and improvements to our documentation.

What's Changed

• Fix typo in readme by @schustmi in https://github.com/zenml-io/zenml/pull/3247
• adding 0.71.0 to migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3250
• Fix workload token expiration for cached steps/runs by @schustmi in https://github.com/zenml-io/zenml/pull/3243
• Implement wandb settings conversion for latest release by @schustmi in https://github.com/zenml-io/zenml/pull/3246
• Add CPU usage note to Modal docs by @strickvl in https://github.com/zenml-io/zenml/pull/3253
• Re-authenticate requests that failed authentication by @stefannica in https://github.com/zenml-io/zenml/pull/3256
• Add new toc by @htahir1 in https://github.com/zenml-io/zenml/pull/3255
• Add step run unique constraint by @schustmi in https://github.com/zenml-io/zenml/pull/3236
• Fix build reuse after stack updates by @schustmi in https://github.com/zenml-io/zenml/pull/3251
• Fix fetching run template using the client by @schustmi in https://github.com/zenml-io/zenml/pull/3258
• Improved deprecation messages for artifact configs and run metadata by @bcdurak in https://github.com/zenml-io/zenml/pull/3261
• Filtering and sorting by @bcdurak in https://github.com/zenml-io/zenml/pull/3230
• Fix hyperparam tuning docs by @stefannica in https://github.com/zenml-io/zenml/pull/3259
• Include user of latest run in pipeline response by @schustmi in https://github.com/zenml-io/zenml/pull/3262
• Create model versions server-side to avoid race conditions by @schustmi in https://github.com/zenml-io/zenml/pull/3254
• Fix request model validation by @schustmi in https://github.com/zenml-io/zenml/pull/3245
• Improve docs to encourage using secrets by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3272
• Include service connector requirements in custom flavor registration by @schustmi in https://github.com/zenml-io/zenml/pull/3267
• Fix the onboarding state to account for zenml login by @stefannica in https://github.com/zenml-io/zenml/pull/3270
• Improve the efficiency of some SQL queries by @schustmi in https://github.com/zenml-io/zenml/pull/3263
• Fix broken link by @strickvl in https://github.com/zenml-io/zenml/pull/3276
• Bump NLP template by @schustmi in https://github.com/zenml-io/zenml/pull/3275
• Fixed and improved sorting by @bcdurak in https://github.com/zenml-io/zenml/pull/3266
• Add matplotlib visualization to ZenML dashboard by @htahir1 in https://github.com/zenml-io/zenml/pull/3278
• Fix azure integration by @schustmi in https://github.com/zenml-io/zenml/pull/3279

Full Changelog: https://github.com/zenml-io/zenml/compare/0.71.0...0.72.0

- Python
Published by schustmi about 1 year ago

https://github.com/zenml-io/zenml - 0.71.0

ZenML version 0.71.0 delivers a new Modal step operator integration as its core feature, enabling efficient cloud execution for ML pipelines with granular hardware configuration options. The release strengthens enterprise capabilities through improved token management and dashboard features while expanding artifact handling with dynamic naming and enhanced visualization support. Additionally, it includes various infrastructure improvements and bug fixes that enhance the platform's stability and usability, particularly around Docker connectivity, Kubernetes management, and service connector operations.

New Feature: Modal Step Operator Integration

ZenML now integrates with Modal, bringing lightning-fast cloud execution capabilities to your ML pipelines. This new step operator allows you to execute individual pipeline steps on Modal's specialized compute instances, offering notable speed, particularly for Docker image building and hardware provisioning. With simple configuration options, you can precisely specify hardware requirements like GPU type, CPU count, and memory for each step, making it ideal for resource-intensive ML workloads.

New Feature: AWS Image Builder

Don't want to worry about Docker locally? Now build images remotely in AWS. Docs: https://docs.zenml.io/stack-components/image-builders/aws

Other Highlights

• Workload API Token Management: Refactored token management for improved security with a generic API token dispenser.
• Dashboard Enhancements:
  • Introduced service account management capabilities.
  • Added API key creation and integration features.
• Dynamic Artifact Naming: Introduced capability to dynamically name artifacts.
• Visualization Enhancements: Made dictionaries and lists visualizable, added JSON visualization type.

Additional Features and Improvements

• Improved error messages for Docker daemon connectivity
• Enhanced SageMaker URL handling
• Simplified model version artifact linkage
• Added testing for pipeline templates
• Improved Kubernetes pod and label length management
• Allowed skipping type annotations for step inputs
• Enabled using feature service instances instead of just names

Bug Fixes

• Fixed issues with getting out of an inaccessible active stack
• Fixed race conditions in the service connector type registry
• Resolved migration test complications
• Corrected documentation links
• Fixed artifact store and artifact URI handling
• Addressed various scalability and compatibility issues

Documentation Updates

• Added documentation redirects
• Updated PyTorch documentation links
• Improved service connector documentation

What's Changed

• Refactored workload API token management for better security and implemented generic API token dispenser by @stefannica in https://github.com/zenml-io/zenml/pull/3154
• Add 0.70.0 to the migration tests by @avishniakov in https://github.com/zenml-io/zenml/pull/3190
• Adjustments to the PR template by @bcdurak in https://github.com/zenml-io/zenml/pull/3194
• [docs] Fix links in the how-to section of docs by @wjayesh in https://github.com/zenml-io/zenml/pull/3196
• Fixing sagemaker urls to take the settings into consideration by @bcdurak in https://github.com/zenml-io/zenml/pull/3195
• Add cached run into testing of migrations by @avishniakov in https://github.com/zenml-io/zenml/pull/3199
• Fix service connector type registry race conditions by @stefannica in https://github.com/zenml-io/zenml/pull/3202
• Refactor container resource configuration in Vertex Orchestrator test by @avishniakov in https://github.com/zenml-io/zenml/pull/3203
• [docs] Add missing redirects by @wjayesh in https://github.com/zenml-io/zenml/pull/3200
• Add links to uv new PyTorch documentation by @strickvl in https://github.com/zenml-io/zenml/pull/3204
• Fix broken docs link by @strickvl in https://github.com/zenml-io/zenml/pull/3208
• Bugfix for getting out of an inaccessible active stack with no permissions by @bcdurak in https://github.com/zenml-io/zenml/pull/3198
• Simplify model version artifact linkage by @schustmi in https://github.com/zenml-io/zenml/pull/3175
• Reenable macos testing by @avishniakov in https://github.com/zenml-io/zenml/pull/3205
• Various fixes and improvements by @stefannica in https://github.com/zenml-io/zenml/pull/3211
• Pass config path during zenml pipeline build by @schustmi in https://github.com/zenml-io/zenml/pull/3212
• Add test for running templates by @schustmi in https://github.com/zenml-io/zenml/pull/3192
• Fix service connector docstring by @schustmi in https://github.com/zenml-io/zenml/pull/3216
• Improve error message when docker daemon is not reachable by @schustmi in https://github.com/zenml-io/zenml/pull/3214
• Don't run migration for empty updates by @schustmi in https://github.com/zenml-io/zenml/pull/3210
• Remove --check from format script and fix naming by @safoinme in https://github.com/zenml-io/zenml/pull/3218
• More scalability improvements by @schustmi in https://github.com/zenml-io/zenml/pull/3206
• Use correct keyword for artifact store open by @schustmi in https://github.com/zenml-io/zenml/pull/3220
• Fix passing of some sagemaker settings by @schustmi in https://github.com/zenml-io/zenml/pull/3221
• Add hint when trying to connect with api key by @schustmi in https://github.com/zenml-io/zenml/pull/3222
• Allow passing None values as parameter for optional complex types by @schustmi in https://github.com/zenml-io/zenml/pull/3215
• Limit kubernetes pod and label length by @schustmi in https://github.com/zenml-io/zenml/pull/3217
• Updating the quickstart example to use the new log_metadata by @bcdurak in https://github.com/zenml-io/zenml/pull/3188
• Allow skipping type annotations for step inputs by @schustmi in https://github.com/zenml-io/zenml/pull/3223
• Modal Step Operator by @strickvl in https://github.com/zenml-io/zenml/pull/2948
• Add dynamic artifacts naming, documentation and tests by @avishniakov in https://github.com/zenml-io/zenml/pull/3201
• Run template CLI command and bugfix by @schustmi in https://github.com/zenml-io/zenml/pull/3225
• Make dicts/lists visualizable and add JSON as viz type by @wjayesh in https://github.com/zenml-io/zenml/pull/2882
• Instances of the FeatureServices are now used instead of only the names of the FeatureServices. by @aiakide in https://github.com/zenml-io/zenml/pull/3209
• Quickstart fixes by @schustmi in https://github.com/zenml-io/zenml/pull/3227
• Add missing docs by @schustmi in https://github.com/zenml-io/zenml/pull/3226
• Misc cleanup by @schustmi in https://github.com/zenml-io/zenml/pull/3229
• Fix input resolution for steps with dynamic artifact names by @schustmi in https://github.com/zenml-io/zenml/pull/3228
• Follow-up on the run_metadata changes by @bcdurak in https://github.com/zenml-io/zenml/pull/3193
• Fixed broken links by @htahir1 in https://github.com/zenml-io/zenml/pull/3232
• Fixed wandb login problem in Quickstart by @htahir1 in https://github.com/zenml-io/zenml/pull/3233
• Misc bugfixes by @schustmi in https://github.com/zenml-io/zenml/pull/3234
• Add additional way to fetch docker repo digest by @schustmi in https://github.com/zenml-io/zenml/pull/3231
• AWS Image Builder implementation by @stefannica in https://github.com/zenml-io/zenml/pull/2904
• Disable client-side caching for some orchestrators by @schustmi in https://github.com/zenml-io/zenml/pull/3235
• Fix artifact uris for artifacts with name placeholders by @schustmi in https://github.com/zenml-io/zenml/pull/3237
• Materializer test fix on Windows by @bcdurak in https://github.com/zenml-io/zenml/pull/3238
• Fix GET step run endpoint to return unhydrated response if requested by @schustmi in https://github.com/zenml-io/zenml/pull/3240
• Pipeline run API token fixes and improvements by @stefannica in https://github.com/zenml-io/zenml/pull/3242

Full Changelog: https://github.com/zenml-io/zenml/compare/0.70.0...0.71.0

- Python
Published by bcdurak about 1 year ago

https://github.com/zenml-io/zenml - 0.70.0

The ZenML 0.70.0 release includes a significant number of database schema changes and migrations, which means upgrading to this version will require extra caution. As always, please make sure to make a copy of your production database before upgrading.

Key Changes

• Artifact Versioning Improvements: The handling of artifact versions has been improved, including the API improvements like the ability to batch artifact version requests to improve the execution times and more types for the step input/output artifacts, including multiple versions of the same artifact (e.g. model checkpoints), to improve the UX using ZenML UI or while working directly with the API.
• Scalability Enhancements: Various scalability improvements have been made, such as reducing unnecessary server requests and incrementing artifact versions server-side. These enhancements are expected to provide significant speed and scale improvements for ZenML users.
• Metadata management: Now, all the metadata-creating functions are gathered under one method called log_metadata. It is possible to call this method with different inputs to log run metadata for artifact versions, model versions, steps, and runs.
• The oneof filtering: This allows to filter entities using a new operator called oneof. You can use this with IDs (UUID type) or tags (or other string-typed attributes) like this PipelineRunFilter(tag='oneof:["cats", "dogs"]').
• Documentation Improvements: The ZenML documentation has been restructured and expanded, including the addition of new sections on finetuning and LLM/ML engineering resources.
• Bug Fixes: This release includes several bug fixes, including issues with in-process main module source loading, and more.

Caution: Make sure to back up your data before upgrading!

While this release brings many valuable improvements, the database schema changes and migrations pose a potential risk to users. It is strongly recommended that users:

• Test the upgrade on a non-production environment: Before upgrading a production system, test the upgrade process in a non-production environment to identify and address any issues.
• Back up your data: Ensure that you have a reliable backup of your ZenML data before attempting the upgrade.

What's Changed

• Optimizing the CI workflows by @bcdurak in https://github.com/zenml-io/zenml/pull/3145
• Adding 0.68.0 to the migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3144
• Move step durations to body by @schustmi in https://github.com/zenml-io/zenml/pull/3046
• Docs on ZenML setup by @strickvl in https://github.com/zenml-io/zenml/pull/3100
• Remove wrongly set Model.wascreatedinthisrun attribute by @schustmi in https://github.com/zenml-io/zenml/pull/3129
• Allow specifying run tags in pipeline configuration by @schustmi in https://github.com/zenml-io/zenml/pull/3130
• Fix materializer type compatibility check during loading by @schustmi in https://github.com/zenml-io/zenml/pull/3105
• [docs] Add icons to headers in docs by @wjayesh in https://github.com/zenml-io/zenml/pull/3149
• fix icons and remove redundant file by @wjayesh in https://github.com/zenml-io/zenml/pull/3150
• Merge 0.68.1 release into develop by @schustmi in https://github.com/zenml-io/zenml/pull/3153
• Allow filtering pipeline runs by stack component by @schustmi in https://github.com/zenml-io/zenml/pull/3142
• Allow artifact response as step input by @schustmi in https://github.com/zenml-io/zenml/pull/3134
• Filter component by user name by @schustmi in https://github.com/zenml-io/zenml/pull/3126
• [docs] Restructure how-to section to make it more readable by @wjayesh in https://github.com/zenml-io/zenml/pull/3147
• ZenML Pro web login implementation by @stefannica in https://github.com/zenml-io/zenml/pull/3141
• Scalability improvements: Reduce misc/hydration server requests by @schustmi in https://github.com/zenml-io/zenml/pull/3093
• Fix in-process main module source loading by @schustmi in https://github.com/zenml-io/zenml/pull/3119
• Catch assertion in GH library by @schustmi in https://github.com/zenml-io/zenml/pull/3160
• Enable cache precomputation for run templates by @schustmi in https://github.com/zenml-io/zenml/pull/3156
• Add LLM and ML engineering books to README by @htahir1 in https://github.com/zenml-io/zenml/pull/3159
• Add helper method to quickly create run template from pipeline by @schustmi in https://github.com/zenml-io/zenml/pull/3155
• Add CLI command to export stack requirements by @schustmi in https://github.com/zenml-io/zenml/pull/3158
• Scalability improvements: Increment artifact version server side by @schustmi in https://github.com/zenml-io/zenml/pull/3095
• Update OpenAI integration by @safoinme in https://github.com/zenml-io/zenml/pull/3163
• Remove deprecated torch version constraint by @safoinme in https://github.com/zenml-io/zenml/pull/3166
• vLLM model deployer by @dudeperf3ct in https://github.com/zenml-io/zenml/pull/3032
• Don't initialize client during flavor sync by @schustmi in https://github.com/zenml-io/zenml/pull/3168
• Cleanup materializer temporary directories after step execution by @schustmi in https://github.com/zenml-io/zenml/pull/3162
• Fix langchain in API docs by @avishniakov in https://github.com/zenml-io/zenml/pull/3171
• Finetuning guide by @strickvl in https://github.com/zenml-io/zenml/pull/3157
• Fix mypy issue vllm evidently by @safoinme in https://github.com/zenml-io/zenml/pull/3169
• Add artifact version batch request by @schustmi in https://github.com/zenml-io/zenml/pull/3164
• Add missing section links by @strickvl in https://github.com/zenml-io/zenml/pull/3172
• Fix uvloop mypy by @avishniakov in https://github.com/zenml-io/zenml/pull/3174
• Multiple output versions for a step outputs by @avishniakov in https://github.com/zenml-io/zenml/pull/3072
• Simplify Metadata handling by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3096
• assign value to component_name in preset stack registration by @hirekk in https://github.com/zenml-io/zenml/pull/3178
• Updating the template versions with zenml login by @bcdurak in https://github.com/zenml-io/zenml/pull/3177
• Better input artifacts typing by @avishniakov in https://github.com/zenml-io/zenml/pull/3099
• Refactor environment setup and caching by @safoinme in https://github.com/zenml-io/zenml/pull/3077
• Fix spelling errors by @safoinme in https://github.com/zenml-io/zenml/pull/3181
• Prevent some race conditions by @schustmi in https://github.com/zenml-io/zenml/pull/3167
• Update stack deployments with latest features by @stefannica in https://github.com/zenml-io/zenml/pull/3183
• Terraform best practices by @htahir1 in https://github.com/zenml-io/zenml/pull/3131
• Fix sagemaker pipeline URLs by @stefannica in https://github.com/zenml-io/zenml/pull/3176
• Fix lightning orchestrator for multi-step pipelines by @wjayesh in https://github.com/zenml-io/zenml/pull/3170
• Port bugfixes from #2497 by @avishniakov in https://github.com/zenml-io/zenml/pull/3179
• Removing the enable_cache from the config files by @bcdurak in https://github.com/zenml-io/zenml/pull/3184
• Don't pass tags to step config by @schustmi in https://github.com/zenml-io/zenml/pull/3186
• New log_metadata function, new oneof filtering, additional run_metadata filtering by @bcdurak in https://github.com/zenml-io/zenml/pull/3182

New Contributors

• @hirekk made their first contribution in https://github.com/zenml-io/zenml/pull/3178

Full Changelog: https://github.com/zenml-io/zenml/compare/0.68.1...0.70.0

- Python
Published by avishniakov over 1 year ago

https://github.com/zenml-io/zenml - 0.68.1

Bug fixes

Fixes an issue with some partially cached pipelines running on remote orchestrators.

What's Changed

• Remove unavailable upstream steps during cache precomputation by @schustmi in https://github.com/zenml-io/zenml/pull/3146

Full Changelog: https://github.com/zenml-io/zenml/compare/0.68.0...0.68.1

- Python
Published by schustmi over 1 year ago

https://github.com/zenml-io/zenml - 0.68.0

Highlights

• Stack Components on the Dashboard: We're bringing back stack components. With this release, you will get access to the list of your stack components on the ZenML dashboard. More functionality is going to follow in the next releases.
• Client-Side Caching: Implemented client-side computation for cached steps, significantly reducing time and costs associated with remote orchestrator spin-up.
• Streamlined Onboarding Process: Unified the starter and production setup into a single sequential flow, providing a more intuitive user experience.
• BentoML Integration: Updated to version 1.3.5 with enhanced containerization support.
• Artifact Management: Introduced register_artifact function enabling direct linking of existing data in the artifact store, particularly useful for tools like PyTorch-Lightning that manage their own checkpoints.
• Enhanced Error Handling: Added Error Boundary to visualization components for improved reliability and user experience.

Additional Features and Improvements

• Added multiple access points for deleting pipeline runs
• Improved pipeline detail view functionality
• Improved service account handling for Kaniko image builder

Breaking Changes and Deprecations

• Discontinued Python 3.8 support
• Removed legacy pipeline and step interface
• Removed legacy post execution workflow
• Removed legacy dashboard option
• Removed zenml stack up/down CLI commands
• Removed zenml deploy and zenml <stack-component> deploy
• Removed StepEnvironment class
• Removed the option to specify a specific model version for step output artifacts using the ArtifactConfig class
• Removed the option to use the ExternalArtifact class to load an artifact from a model version
• Removed Client.list_runs, replacing it with Client.list_pipeline_runs
• Removed ArtifactVersionResponse.read, replacing it with ArtifactVersionResponse.load

Documentation Updates

Added new guides for the following topics:

• Kubernetes per-pod configuration
• Factory generation of artifact names
• Common stacks best practices
• Azure 1-click dashboard deployment
• ZenML server upgrade best practices
• Custom Dataset classes and Materializers
• Comprehensive ZenML Pro documentation
• Image building optimization during pipeline runs
• Enhanced BentoML integration documentation

What's Changed

• Release 0.67.0 migration testing by @bcdurak in https://github.com/zenml-io/zenml/pull/3050
• Prevent too large requests by @avishniakov in https://github.com/zenml-io/zenml/pull/3048
• Fix Neptune linting after 1.12.0 release by @avishniakov in https://github.com/zenml-io/zenml/pull/3055
• Fix Lightning Orchestrator (remove -y from pip install) by @wjayesh in https://github.com/zenml-io/zenml/pull/3058
• Fix artifact pruning endpoint path by @schustmi in https://github.com/zenml-io/zenml/pull/3052
• Update python versioning in docs by @avishniakov in https://github.com/zenml-io/zenml/pull/3059
• Fix infinite loop while fetching artifact store in logs storage class by @avishniakov in https://github.com/zenml-io/zenml/pull/3061
• Make sync a setting for sagemaker/azureml orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/3062
• Remove some deprecated features by @schustmi in https://github.com/zenml-io/zenml/pull/2926
• Fix MySQL warning when filtering pipelines by latest run by @schustmi in https://github.com/zenml-io/zenml/pull/3051
• Remove more deprecated stuff by @schustmi in https://github.com/zenml-io/zenml/pull/3063
• Remove log versions from versioned buckets in S3 by @avishniakov in https://github.com/zenml-io/zenml/pull/3060
• add docs on k8s per pod settings by @wjayesh in https://github.com/zenml-io/zenml/pull/3066
• Remove Python 3.8 support by @strickvl in https://github.com/zenml-io/zenml/pull/3034
• register_artifact function by @avishniakov in https://github.com/zenml-io/zenml/pull/3053
• Fix bad link in docs by @avishniakov in https://github.com/zenml-io/zenml/pull/3069
• Fix model linkage for the lazy loading scenarios by @avishniakov in https://github.com/zenml-io/zenml/pull/3054
• Updating template versions after the Python 3.8 changes by @bcdurak in https://github.com/zenml-io/zenml/pull/3070
• Add UUID materializer by @htahir1 in https://github.com/zenml-io/zenml/pull/3073
• Fix pipeline and model URLs for ZenML Pro on-prem deployments by @stefannica in https://github.com/zenml-io/zenml/pull/3083
• Update bentoml integration to 1.3.5 and add containerization by @wjayesh in https://github.com/zenml-io/zenml/pull/3045
• Fix mlflow linting by @schustmi in https://github.com/zenml-io/zenml/pull/3085
• Add docs for factory generation of artifact names by @strickvl in https://github.com/zenml-io/zenml/pull/3084
• Remove unnecessary metadata fields in UUID materializer test by @htahir1 in https://github.com/zenml-io/zenml/pull/3088
• Client-side computation of cached steps by @schustmi in https://github.com/zenml-io/zenml/pull/3068
• Fix Kaniko image builder service account passing by @schustmi in https://github.com/zenml-io/zenml/pull/3081
• Bugfix in GitLab Code Repository integration by @4gt-104 in https://github.com/zenml-io/zenml/pull/3076
• Add docs on common stacks best practices by @strickvl in https://github.com/zenml-io/zenml/pull/3092
• [docs] Update stacks page and add azure 1-click from dashboard docs by @wjayesh in https://github.com/zenml-io/zenml/pull/3082
• Local development how-to section by @strickvl in https://github.com/zenml-io/zenml/pull/3090
• [docs] best practices for upgrading zenml server by @wjayesh in https://github.com/zenml-io/zenml/pull/3087
• Fix S3 ArtifactStore auth issue by @avishniakov in https://github.com/zenml-io/zenml/pull/3086
• Reduce migration testing runtime by @avishniakov in https://github.com/zenml-io/zenml/pull/3078
• [docs] Dedicated docs on how to skip building an image on pipeline run by @wjayesh in https://github.com/zenml-io/zenml/pull/3079
• Fix filtering by tag for pipeline runs by @schustmi in https://github.com/zenml-io/zenml/pull/3097
• Remove deprecated features: zenml deploy and zenml <stack-component> deploy by @stefannica in https://github.com/zenml-io/zenml/pull/3089
• Do not tag model via Model class on creation by @avishniakov in https://github.com/zenml-io/zenml/pull/3098
• Sagemaker add pipeline tags by @htahir1 in https://github.com/zenml-io/zenml/pull/3080
• [docs] Add custom Dataset classes and Materializers in ZenML by @htahir1 in https://github.com/zenml-io/zenml/pull/3091
• Delete Scarf related scripts and workflow files by @htahir1 in https://github.com/zenml-io/zenml/pull/3103
• Add more detailed docs for ZenML Pro by @wjayesh in https://github.com/zenml-io/zenml/pull/3065
• Add missing code hash filter in client method by @schustmi in https://github.com/zenml-io/zenml/pull/3094
• Remove lineage graph and legacy dashboard support by @schustmi in https://github.com/zenml-io/zenml/pull/3064
• Add unittest to cover gitlab CR regex. by @4gt-104 in https://github.com/zenml-io/zenml/pull/3102
• Automating the release process using Github workflows by @bcdurak in https://github.com/zenml-io/zenml/pull/3101
• Bugfix for release automation by @bcdurak in https://github.com/zenml-io/zenml/pull/3107
• Bugfix for new version in the release automation by @bcdurak in https://github.com/zenml-io/zenml/pull/3108
• using the right parent image name by @bcdurak in https://github.com/zenml-io/zenml/pull/3109
• Making the new release automation scripts executable by @bcdurak in https://github.com/zenml-io/zenml/pull/3110
• Fixing the env variables for the release automation by @bcdurak in https://github.com/zenml-io/zenml/pull/3111
• Adding the right Github configuration before using the gh CLI to fetch the version by @bcdurak in https://github.com/zenml-io/zenml/pull/3112
• Fixing the outputs of the first step in the release automation by @bcdurak in https://github.com/zenml-io/zenml/pull/3113
• Handling github auth and release notes for release automation by @bcdurak in https://github.com/zenml-io/zenml/pull/3114
• Fixing the cloudbuild call for release automation by @bcdurak in https://github.com/zenml-io/zenml/pull/3116
• Fixing the update tenant call in the script by @bcdurak in https://github.com/zenml-io/zenml/pull/3118
• Release automation with the new redeploy logic by @bcdurak in https://github.com/zenml-io/zenml/pull/3120
• Fixing the automation triggers for other branches by @bcdurak in https://github.com/zenml-io/zenml/pull/3125
• Update link for llm-complete-guide repository.- Updated link to poi… by @htahir1 in https://github.com/zenml-io/zenml/pull/3128
• Fixing the migration testing for the release branches by @bcdurak in https://github.com/zenml-io/zenml/pull/3127
• Update pipeline deletion docs by @strickvl in https://github.com/zenml-io/zenml/pull/3123
• Disabling the cache for the quickstart tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3133
• Update Argilla integration for v2.x SDK by @sdiazlor in https://github.com/zenml-io/zenml/pull/2915
• Using pip instead of gh CLI in the migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/3136
• Adapting tags to work with older versions of Sagemaker by @bcdurak in https://github.com/zenml-io/zenml/pull/3135
• Manual trigger for the release_finalize workflow by @bcdurak in https://github.com/zenml-io/zenml/pull/3137
• Fixing the prepare trigger for the release automation by @bcdurak in https://github.com/zenml-io/zenml/pull/3138

New Contributors

• @4gt-104 made their first contribution in https://github.com/zenml-io/zenml/pull/3076
• @sdiazlor made their first contribution in https://github.com/zenml-io/zenml/pull/2915

Full Changelog: https://github.com/zenml-io/zenml/compare/0.67.0...0.68.0

- Python
Published by bcdurak over 1 year ago

https://github.com/zenml-io/zenml - 0.67.0

Highlights

• Improved Sagemaker Orchestrator: Now supports warm pools for AWS Sagemaker, enhancing performance and reducing startup times for TrainingJobs.
• New DAG Visualizer: Shipped major enhancements to the DAG Visualizer for Pipeline Runs:
  • Preview of the actual DAG before pipeline completion
  • Visual adjustments for improved clarity
  • Real-time updates during pipeline execution
• Environment Variable References in Configurations: Introduced the ability to reference environment variables in both code and configuration files using the syntax ${ENVVARIABLENAME}, increasing flexibility in setups.
• Enhanced UX for Major Cloud Providers: Displaying direct pipeline/log URL when working with major cloud platforms.
• Skypilot with Kubernetes Support: Added compatibility for running Skypilot orchestrator on Kubernetes clusters.
• Updated Deepchecks Integration: The Deepchecks integration has been refreshed with the latest features and improvements.

Features and Improvements

• AWS Integration:
  • Added permissions to workflow to enable assuming AWS role.
  • Fixed expired credentials error when using the docker service connector.
• Error Handling: Improved error messages for stack components of uninstalled integrations.
• API Key Management: Added an option to write API keys to a file instead of using the CLI.

Pipeline Execution:

• Implemented fixes for executing steps as single step pipelines.
• Added filter option for templatable runs.
• Added additional filtering options for pipeline runs.
• MLflow Integration: Linked registered models in MLflow with the corresponding MLflow run.
• Analytics: Added missing analytics event to improve user insights.

Documentation Updates

• Updated documentation for various integrations including:
  • Lightning AI orchestrator
  • Kubeflow
  • Comet experiment tracker
  • Neptune
  • Hugging Face deployer
  • Weights & Biases (wandb)
• Added documentation for run templates.
• Fixed incorrect method name in Pigeon docs.
• Various small documentation fixes and improvements.

Bug Fixes

• Fixed YAML formatting issues.
• Resolved RBAC issues for subpages in response models.
• Fixed step output annotation in Discord test.
• Addressed MLFlow integration requirements duplication.
• Fixed Lightning orchestrator functionality.

What's Changed

• Error message for stack components of uninstalled integrations by @bcdurak in https://github.com/zenml-io/zenml/pull/2996
• Enable migration testing for version 0.66.0 by @schustmi in https://github.com/zenml-io/zenml/pull/2998
• Add permissions to workflow to enable assuming AWS role by @schustmi in https://github.com/zenml-io/zenml/pull/2999
• Add option to write api key to file instead of CLI by @schustmi in https://github.com/zenml-io/zenml/pull/3001
• Fix yaml formatting by @schustmi in https://github.com/zenml-io/zenml/pull/3004
• Update ZenML Pro links for consistency.- Update ZenML Pro links for c… by @htahir1 in https://github.com/zenml-io/zenml/pull/3007
• Fix incorrect method name in Pigeon docs by @strickvl in https://github.com/zenml-io/zenml/pull/3008
• Fixes for executing steps as single step pipelines by @schustmi in https://github.com/zenml-io/zenml/pull/3006
• Add filter option for templatable runs by @schustmi in https://github.com/zenml-io/zenml/pull/3000
• Add missing analytics event by @schustmi in https://github.com/zenml-io/zenml/pull/3009
• Fix expired credentials error when using the docker service connector by @schustmi in https://github.com/zenml-io/zenml/pull/3002
• Fix Lightning docs by @strickvl in https://github.com/zenml-io/zenml/pull/3013
• Remove image builder warning by @htahir1 in https://github.com/zenml-io/zenml/pull/3014
• Fixed kubeflow docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3018
• Update Comet experiment tracker docs by @htahir1 in https://github.com/zenml-io/zenml/pull/3019
• Small docs fixes by @strickvl in https://github.com/zenml-io/zenml/pull/3022
• Feature/cleanup unused file by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3023
• MLFlow integration requirements duplicate fix by @bcdurak in https://github.com/zenml-io/zenml/pull/3011
• Fix Neptune docs by @htahir1 in https://github.com/zenml-io/zenml/pull/3026
• Fix huggingface deployer docs by @htahir1 in https://github.com/zenml-io/zenml/pull/3024
• Fix step output annotation in Discord test by @wjayesh in https://github.com/zenml-io/zenml/pull/3029
• Fix RBAC for subpages in response models by @schustmi in https://github.com/zenml-io/zenml/pull/3031
• Allow env variable placeholders in configurations by @schustmi in https://github.com/zenml-io/zenml/pull/3003
• Leverage warm pools for AWS Sagemaker by @avishniakov in https://github.com/zenml-io/zenml/pull/3027
• Updated wandb docs by @htahir1 in https://github.com/zenml-io/zenml/pull/3030
• Add hyperlint by @htahir1 in https://github.com/zenml-io/zenml/pull/3035
• Bump NLP template by @avishniakov in https://github.com/zenml-io/zenml/pull/3036
• Add additional filtering options by @schustmi in https://github.com/zenml-io/zenml/pull/2951
• Bump starter template version by @schustmi in https://github.com/zenml-io/zenml/pull/3038
• Docs for run templates by @bcdurak in https://github.com/zenml-io/zenml/pull/3028
• Update Lightning AI orchestrator documentation by @strickvl in https://github.com/zenml-io/zenml/pull/3016
• Add default value for PipelineRun.is_templatable by @schustmi in https://github.com/zenml-io/zenml/pull/3040
• Use a generic OAuth2 client credentials flow to login to the Cloud API by @stefannica in https://github.com/zenml-io/zenml/pull/3041
• fix lightning orchestrator by @safoinme in https://github.com/zenml-io/zenml/pull/3010
• Linking registered models in MLflow with the corresponding MLflow run by @aiakide in https://github.com/zenml-io/zenml/pull/3020
• Bugfixing mlflow registry linting issue by @bcdurak in https://github.com/zenml-io/zenml/pull/3043
• Enhancing the orchestrator UX for major cloud providers by @bcdurak in https://github.com/zenml-io/zenml/pull/3005
• Skypilot with Kubernetes by @safoinme in https://github.com/zenml-io/zenml/pull/3033
• Update deepchecks integration by @wjayesh in https://github.com/zenml-io/zenml/pull/2987

New Contributors

• @aiakide made their first contribution in https://github.com/zenml-io/zenml/pull/3020

Full Changelog: https://github.com/zenml-io/zenml/compare/0.66.0...0.67.0

- Python
Published by bcdurak over 1 year ago

https://github.com/zenml-io/zenml - 0.66.0

New Features and Improvements

Python 3.12 support

This release adds support for Python 3.12, which means you can now develop your ZenML pipelines with the latest python features.

Easier way to specify component settings

Before this release, settings for stack components had to be specified with both the component type as well as the flavor. We simplified this and it is now possible to specify settings just using the component type: ```python

Before

@pipeline(settings={"orchestrator.sagemaker": SagemakerOrchestratorSettings(...)}) def my_pipeline(): ...

Now

@pipeline(settings={"orchestrator": SagemakerOrchestratorSettings(...)}) def my_pipeline(): ... ```

Breaking changes

• In order to slim down the ZenML library, we removed the numpy and pandas libraries as dependencies of ZenML. If your code uses these libraries, you have to make sure they're installed in your local environment as well as the Docker images that get built to run your pipelines (Use DockerSettings.requirements or DockerSettings.required_integrations).

What's Changed

• Add 0.65.0 to migration testing by @avishniakov in https://github.com/zenml-io/zenml/pull/2963
• Hotfix for release flow by @avishniakov in https://github.com/zenml-io/zenml/pull/2961
• Fix the one-click AWS and GCP stack deployments by @stefannica in https://github.com/zenml-io/zenml/pull/2964
• Fix wandb mypy error by @strickvl in https://github.com/zenml-io/zenml/pull/2967
• Fix accelerate docs for 0.65.0+ by @avishniakov in https://github.com/zenml-io/zenml/pull/2968
• Dynamic model version names docs by @avishniakov in https://github.com/zenml-io/zenml/pull/2970
• Logging nits by @avishniakov in https://github.com/zenml-io/zenml/pull/2972
• Fix excess Azure logging by @strickvl in https://github.com/zenml-io/zenml/pull/2965
• Fix typo in docs by @strickvl in https://github.com/zenml-io/zenml/pull/2976
• Pass code path to template run by @schustmi in https://github.com/zenml-io/zenml/pull/2973
• Prevent extra attributes in component configs by @schustmi in https://github.com/zenml-io/zenml/pull/2978
• Dependency cleanup and Python 3.12 support by @bcdurak in https://github.com/zenml-io/zenml/pull/2953
• Few nits in docs based on integrations review by @avishniakov in https://github.com/zenml-io/zenml/pull/2983
• Update slack alerter docs by @stefannica in https://github.com/zenml-io/zenml/pull/2981
• Update Kubeflow orchestrator docs by @stefannica in https://github.com/zenml-io/zenml/pull/2985
• Build docker images for python 3.12 by @schustmi in https://github.com/zenml-io/zenml/pull/2988
• Allow shortcut keys for component settings by @schustmi in https://github.com/zenml-io/zenml/pull/2957
• Remove references to workspaces from docs by @strickvl in https://github.com/zenml-io/zenml/pull/2991
• Added some adjustments for colab by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2966
• Reverting the installation of mlstacks after its new release by @bcdurak in https://github.com/zenml-io/zenml/pull/2980
• Small dependency and docs updates by @strickvl in https://github.com/zenml-io/zenml/pull/2982

Full Changelog: https://github.com/zenml-io/zenml/compare/0.65.0...0.66.0

- Python
Published by schustmi over 1 year ago

https://github.com/zenml-io/zenml - 0.65.0

Important note for OSS users

In the latest release, in the onboarding flow, the first pipeline has been labeled python run.py --training-pipeline but rather should be simply python run.py only. If you see an error while trying to completely the onboarding , simply use the latter command instead!

New Features and Improvements

New Quickstart Experience

This example demonstrates how ZenML streamlines the transition of machine learning workflows from local environments to cloud-scale operations.

Run Single Step as a ZenML Pipeline

If you want to run just an individual step on your stack, you can simply call the step as you would with a normal Python function. ZenML will internally create a pipeline with just your step and run it on the active stack.

Other improvements and fixes

• Updated AzureML Step Operator to work with SDKv2 and use Service Connectors
• Added timestamps to log messages
• Fixed issue with loading artifacts from the artifact store outside of the current active artifact store
• Support of templated names for Model Version ({date} and {time} are currently supported placeholders)
• run_with_accelerate step wrapper can be used as a Python Decorator on top of ZenML steps

Breaking changes

• Workspace scoped POST endpoint full-stack was removed and merged with stacks POST endpoint
• If you use 0.65.0 with any prior server version of ZenML it might lead to a situation where a Model Version is created for every step of the pipeline, while the Model class was only once configured on the pipeline level. This is considered an expected behavior and you should not use the mismatching versions of ZenML Client and Server, in general. Minimal example: ```python from zenml import step, pipeline, Model

@step def step_1()->None: print("1")

@step def step_2()->None: print("2")

@pipeline(model=Model(name="mymodel")) def mypipeline()->None: step1() step2()

if name=="main": mypipeline() `` In this case on server versions prior to0.65.0you will get 2 Model Versions ofmymodelcreated: one instep1and one instep2. Please upgrade your server to0.65.0+` and you will get only one Model Version for the same code snippet.

What's Changed

• Remove broken JIRA sync workflow by @strickvl in https://github.com/zenml-io/zenml/pull/2924
• Fix Hugging Face Spaces permissions by @strickvl in https://github.com/zenml-io/zenml/pull/2925
• Fixes for run_with_accelerate by @avishniakov in https://github.com/zenml-io/zenml/pull/2935
• Bump azure skypilot to a stable 0.6.1 by @avishniakov in https://github.com/zenml-io/zenml/pull/2933
• Add Timestamps to Logs and Update Dashboard URL Message by @htahir1 in https://github.com/zenml-io/zenml/pull/2934
• Adding 0.64.0 to migration tests by @bcdurak in https://github.com/zenml-io/zenml/pull/2923
• Removed docker build docs + fixed CLI command for zenml pipeline build list by @htahir1 in https://github.com/zenml-io/zenml/pull/2938
• Throw an error when running integration installs when uv == False but pip is not installed by @mennoliefstingh in https://github.com/zenml-io/zenml/pull/2930
• Update AzureML step operator to SDK v2 and add service connector support by @stefannica in https://github.com/zenml-io/zenml/pull/2927
• Improving the AzureML orchestrator docs by @bcdurak in https://github.com/zenml-io/zenml/pull/2940
• Update mlflow docs by @htahir1 in https://github.com/zenml-io/zenml/pull/2941
• Tell users where they can import DockerSettings from by @strickvl in https://github.com/zenml-io/zenml/pull/2947
• Fail early when specifying invalid materializers by @schustmi in https://github.com/zenml-io/zenml/pull/2950
• Add GitHub Codespaces and VS Code Remote Container support by @htahir1 in https://github.com/zenml-io/zenml/pull/2949
• Automatically detect whether code download is necessary by @schustmi in https://github.com/zenml-io/zenml/pull/2946
• Enable running a single step on the active stack by @schustmi in https://github.com/zenml-io/zenml/pull/2942
• Dynamic (templated) names for model versions by @avishniakov in https://github.com/zenml-io/zenml/pull/2909
• Adding an orchestrator URL to the AzureML orchestrator by @bcdurak in https://github.com/zenml-io/zenml/pull/2952
• Update python version of latest docker image by @schustmi in https://github.com/zenml-io/zenml/pull/2954
• Make run_with_accelerate a pythonic decorator by @avishniakov in https://github.com/zenml-io/zenml/pull/2943
• Bugfix for artifacts coming from a different artifact store by @bcdurak in https://github.com/zenml-io/zenml/pull/2928
• Stack Request cleanup and improvements by @bcdurak in https://github.com/zenml-io/zenml/pull/2906
• Silence pydantic protected namespace warnings by @schustmi in https://github.com/zenml-io/zenml/pull/2955
• Update key for finished onboarding survey by @schustmi in https://github.com/zenml-io/zenml/pull/2956
• Extend notebook source replacement code to other objects apart from ZenML steps by @schustmi in https://github.com/zenml-io/zenml/pull/2919
• Fix stack register CLI command by @schustmi in https://github.com/zenml-io/zenml/pull/2958
• Lightening studio orchestrator by @safoinme in https://github.com/zenml-io/zenml/pull/2931
• Introduce new quickstart with a focus on Stack switching by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2937
• Bugfix for the required prompts for the AzureML wizard by @bcdurak in https://github.com/zenml-io/zenml/pull/2959

New Contributors

• @mennoliefstingh made their first contribution in https://github.com/zenml-io/zenml/pull/2930

Full Changelog: https://github.com/zenml-io/zenml/compare/0.64.0...0.65.0

- Python
Published by avishniakov over 1 year ago

https://github.com/zenml-io/zenml - 0.64.0

New Features and Improvements

Notebook Integration

ZenML now supports running steps defined in notebook cells with remote orchestrators and step operators. This feature enhances the development workflow by allowing seamless transition from experimentation to production.

• Details: Running remote pipelines from notebooks

Reduced Docker Builds with Code Uploads

We've introduced an option to upload code to the artifact store, enabling Docker build reuse. This feature can significantly speed up iteration, especially when working with remote stacks.

• Default: Enabled
• Configuration: To disable, set DockerSettings.allow_download_from_artifact_store=False for steps or pipelines
• Benefits:
  • Faster development cycles
  • No need to register a code repository to reuse builds
  • Builds only occur when requirements or DockerSettings change
• Documentation: Which files are built into the image

AzureML Orchestrator Support

ZenML now supports AzureML as an orchestrator, expanding our list of supported cloud platforms.

• Full Azure Guide: Setting up an Azure stack
• Documentation: AzureML orchestrator

Terraform Modules

We've released new Terraform modules on the Hashicorp registry for provisioning complete MLOps stacks across major cloud providers.

• Features:
  • Automate infrastructure setup for ZenML stack deployment
  • Handle registration of configurations to ZenML server
• More Information: MLOps Terraform ZenML blog post

These updates aim to streamline the MLOps workflow, making it easier to develop, deploy, and manage machine learning pipelines with ZenML.

What's Changed

• Add 0.63.0 to migration testing by @bcdurak in https://github.com/zenml-io/zenml/pull/2893
• Document terraform stack deployment modules by @stefannica in https://github.com/zenml-io/zenml/pull/2898
• README update by @htahir1 in https://github.com/zenml-io/zenml/pull/2901
• Enable Databricks Unity Catalog for MLflow by @safoinme in https://github.com/zenml-io/zenml/pull/2900
• Make urls pop out from the sea of purple/cyan in the logs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2894
• Add terraform as a supported stack deployment provider by @stefannica in https://github.com/zenml-io/zenml/pull/2902
• Fix Model imports in docs by @strickvl in https://github.com/zenml-io/zenml/pull/2907
• Remove hub references by @schustmi in https://github.com/zenml-io/zenml/pull/2905
• Bump NLP template by @avishniakov in https://github.com/zenml-io/zenml/pull/2912
• Updated step operator docs by @htahir1 in https://github.com/zenml-io/zenml/pull/2908
• Added lightning studio check by @htahir1 in https://github.com/zenml-io/zenml/pull/2910
• Upload code to artifact store by @schustmi in https://github.com/zenml-io/zenml/pull/2895
• AzureML orchestrator by @bcdurak in https://github.com/zenml-io/zenml/pull/2873
• Run steps defined in notebooks with remote orchestrators by @schustmi in https://github.com/zenml-io/zenml/pull/2899
• Fix broken / unparsable md docs file by @strickvl in https://github.com/zenml-io/zenml/pull/2916
• Bump mlflow to 2.15.0 by @christianversloot in https://github.com/zenml-io/zenml/pull/2896
• Remove extra button by @schustmi in https://github.com/zenml-io/zenml/pull/2918
• Added last timestamp to zenserver by @htahir1 in https://github.com/zenml-io/zenml/pull/2913
• A pipeline can't finish successfully in this case by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2903
• Fix the GCP Workload Identity Federation support in the GCP Service Connector by @stefannica in https://github.com/zenml-io/zenml/pull/2914
• Embeddings finetuning guide for LLMOps guide by @strickvl in https://github.com/zenml-io/zenml/pull/2917

🥳 Community Contributions 🥳

We'd like to give a special thanks to @christianversloot who contributed to this release by bumping the mlflow version to 2.15.0

Full Changelog: https://github.com/zenml-io/zenml/compare/0.63.0...0.64.0

- Python
Published by bcdurak over 1 year ago

https://github.com/zenml-io/zenml - 0.63.0

Moving forward from the last two releases, we have further improved the 1-click deployment tool and the stack wizard by adding support for Azure.

Moreover, we implemented a new step operator that allows you to run individual steps of your pipeline in Kubernetes pods.

Lastly, we have simplified our pipeline models by removing their versions. No migration is required but if you were using the API, please take out references to pipeline versions, and reference runs directly from now on.

What's Changed

• Enable cloud build service in GCP stack deployment by @stefannica in https://github.com/zenml-io/zenml/pull/2864
• Adding a logo_url and the of the integration to component responses by @bcdurak in https://github.com/zenml-io/zenml/pull/2866
• Use REST in Model tests by @avishniakov in https://github.com/zenml-io/zenml/pull/2834
• Add Azure stack wizard by @avishniakov in https://github.com/zenml-io/zenml/pull/2841
• Migration testing for 0.62.0 by @schustmi in https://github.com/zenml-io/zenml/pull/2860
• Fix RBAC in combination with lazy loaders by @schustmi in https://github.com/zenml-io/zenml/pull/2869
• Misc cleanup after release by @schustmi in https://github.com/zenml-io/zenml/pull/2861
• Disable notebook error for Kubernetes orchestrator by @strickvl in https://github.com/zenml-io/zenml/pull/2870
• Added ability to add labels to k8s pod by @htahir1 in https://github.com/zenml-io/zenml/pull/2872
• Fix zenml pro links by @schustmi in https://github.com/zenml-io/zenml/pull/2875
• Fix mlstacks docs typo by @begoechavarren in https://github.com/zenml-io/zenml/pull/2878
• Fix requests vulnerability by @stefannica in https://github.com/zenml-io/zenml/pull/2843
• Fixed some minor docs things i noticed by @htahir1 in https://github.com/zenml-io/zenml/pull/2881
• Serialize source as Any to keep subclass attributes by @schustmi in https://github.com/zenml-io/zenml/pull/2880
• Fix node selectors for Vertex orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/2876
• Kubernetes step operator by @schustmi in https://github.com/zenml-io/zenml/pull/2883
• Automatically populate GCP/azure path when using wizard from the frontend by @schustmi in https://github.com/zenml-io/zenml/pull/2886
• Remove pipeline versioning and add run templates by @schustmi in https://github.com/zenml-io/zenml/pull/2830
• Implement the Azure 1-click stack deployment by @stefannica in https://github.com/zenml-io/zenml/pull/2887
• Better error message sagemaker, better documentation server env vars by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2885
• Azure Stack Wizard docs by @bcdurak in https://github.com/zenml-io/zenml/pull/2890
• Docs update mlflow deploy function call by @safoinme in https://github.com/zenml-io/zenml/pull/2863
• Fix databricks resource setting by @safoinme in https://github.com/zenml-io/zenml/pull/2889

New Contributors

• @begoechavarren made their first contribution in https://github.com/zenml-io/zenml/pull/2878

Full Changelog: https://github.com/zenml-io/zenml/compare/0.62.0...0.63.0

- Python
Published by bcdurak over 1 year ago

https://github.com/zenml-io/zenml - 0.62.0

Building on top of the last release, this release adds a new and easy way to deploy a GCP ZenML stack from the dashboard and the CLI. Give it a try by going to the Stacks section in the dashboard or running the zenml stack deploy command! For more information on this new feature, please do check out the video and blog from our previous release.

We also updated our Hugging Face integration to support the automatic display of an embedded datasets preview pane in the ZenML Dashboard whenever you return a Dataset from a step. This was recently released by the Hugging Face datasets team and it allows you to easily visualize and inspect your data from the comfort of the dashboard.

What's Changed

• Fix release action docker limit by @schustmi in https://github.com/zenml-io/zenml/pull/2837
• Upgrade ruff and yamlfix to latest versions before running formatting by @christianversloot in https://github.com/zenml-io/zenml/pull/2577
• Fixed edge-case where step run is stored incompletely by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2827
• Docs for stack registration + deployment wizards by @htahir1 in https://github.com/zenml-io/zenml/pull/2814
• Make upgrade checks in formatting script optional by @avishniakov in https://github.com/zenml-io/zenml/pull/2839
• Enable migration testing for version 0.61.0 by @schustmi in https://github.com/zenml-io/zenml/pull/2836
• One-click GCP stack deployments by @stefannica in https://github.com/zenml-io/zenml/pull/2833
• Only login to docker for PRs with secret access by @schustmi in https://github.com/zenml-io/zenml/pull/2842
• Add GCP Stack creation Wizard (CLI) by @avishniakov in https://github.com/zenml-io/zenml/pull/2826
• Update onboarding by @schustmi in https://github.com/zenml-io/zenml/pull/2794
• Merged log files in Step Ops steps might be not available on main process, due to merge in the step op by @avishniakov in https://github.com/zenml-io/zenml/pull/2795
• Fix some broken links, copy paste commands, and made secrets more visible by @htahir1 in https://github.com/zenml-io/zenml/pull/2848
• Update stack deployment docs and other small fixes by @stefannica in https://github.com/zenml-io/zenml/pull/2846
• Improved the StepInterfaceError message for missing inputs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2849
• add image pull secrets to k8s pod settings by @wjayesh in https://github.com/zenml-io/zenml/pull/2847
• Include apt installation of libgomp1 for docker images with lightgbm by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2813
• Patch filter mflow by stage by @whoknowsB in https://github.com/zenml-io/zenml/pull/2798
• Bump mlflow to version 2.14.2 by @christianversloot in https://github.com/zenml-io/zenml/pull/2825
• Fix Accelerate string arguments passing by @avishniakov in https://github.com/zenml-io/zenml/pull/2845
• Fix CI by @schustmi in https://github.com/zenml-io/zenml/pull/2850
• Added some visualizations for the HF dataset by @htahir1 in https://github.com/zenml-io/zenml/pull/2851
• Fix skypilot versioning for the lambda integration by @wjayesh in https://github.com/zenml-io/zenml/pull/2853
• Improve custom visualization docs by @htahir1 in https://github.com/zenml-io/zenml/pull/2855
• Fix list typo by @htahir1 in https://github.com/zenml-io/zenml/pull/2856
• Endpoint to get existing and prospective resources for service connector by @avishniakov in https://github.com/zenml-io/zenml/pull/2854
• Databricks integrations by @safoinme in https://github.com/zenml-io/zenml/pull/2823

New Contributors

• @whoknowsB made their first contribution in https://github.com/zenml-io/zenml/pull/2798

Full Changelog: https://github.com/zenml-io/zenml/compare/0.61.0...0.62.0

- Python
Published by schustmi over 1 year ago

https://github.com/zenml-io/zenml - 0.61.0

This release comes with a new and easy way to deploy a simple ZenML stack from the dashboard and the CLI. Give it a try by going to the Stacks section in the dashboard:

or by running the zenml stack deploy command!

We hope this makes it super easy for existing and new users to set up the infrastructure required to run ZenML pipelines on the cloud in one click.

Note: Only a simple AWS stack using Skypilot supported for now but GCP + Azure to come!

Additionally, this release includes improvements to our documentation and bugfixes for some integrations.

What's Changed

• Add latest zenml version to migration testing scripts by @htahir1 in https://github.com/zenml-io/zenml/pull/2811
• Add service connector support for Google Artifact Registry by @stefannica in https://github.com/zenml-io/zenml/pull/2771
• Update order in which requirements are installed by @schustmi in https://github.com/zenml-io/zenml/pull/2341
• Add installation instructions for Macs running on Apple Silicon by @strickvl in https://github.com/zenml-io/zenml/pull/2774
• Added docs for trigger interface by @htahir1 in https://github.com/zenml-io/zenml/pull/2806
• Update triggers docs with information on previously-run pipelines by @strickvl in https://github.com/zenml-io/zenml/pull/2820
• Bump kfp version in GCP integration for pydantic2.0 by @wjayesh in https://github.com/zenml-io/zenml/pull/2824
• Use shared cloud connection to reduce M2M token usage by @schustmi in https://github.com/zenml-io/zenml/pull/2817
• Fail pipeline run if error happens during deployment by @schustmi in https://github.com/zenml-io/zenml/pull/2818
• Login to dockerhub to solve rate limiting by @schustmi in https://github.com/zenml-io/zenml/pull/2828
• Stack wizard CLI + Endpoints by @avishniakov in https://github.com/zenml-io/zenml/pull/2808
• In-browser assisted full cloud stack deployments by @stefannica in https://github.com/zenml-io/zenml/pull/2816
• Fix Kubeflow v2 integration by @wjayesh in https://github.com/zenml-io/zenml/pull/2829
• fix skypilot jobs failing on VMs (sky bumped to 0.6.0) by @wjayesh in https://github.com/zenml-io/zenml/pull/2815
• Fix unicode decode errors in k8s pod logs read operation by @wjayesh in https://github.com/zenml-io/zenml/pull/2807
• Small improvements and bug fixes by @schustmi in https://github.com/zenml-io/zenml/pull/2821
• TF tests + various integration (un)install improvements by @avishniakov in https://github.com/zenml-io/zenml/pull/2791
• Fixed bug in the MacOS version check by @strickvl in https://github.com/zenml-io/zenml/pull/2819
• Remove prefix for analytics labels by @schustmi in https://github.com/zenml-io/zenml/pull/2831

Full Changelog: https://github.com/zenml-io/zenml/compare/0.60.0...0.61.0

- Python
Published by schustmi over 1 year ago

https://github.com/zenml-io/zenml - 0.60.0

ZenML now uses Pydantic v2. 🥳

This upgrade comes with a set of critical updates. While your user experience mostly remains unaffected, you might see unexpected behavior due to the changes in our dependencies. Moreover, since Pydantic v2 provides a slightly stricter validation process, you might end up bumping into some validation errors which was not caught before, but it is all for the better 🙂 If you run into any other errors, please let us know either on GitHub or on our Slack.

Changes in some of the critical dependencies

• SQLModel is one of the core dependencies of ZenML and prior to this upgrade, we were utilizing version 0.0.8. However, this version is relatively outdated and incompatible with Pydantic v2. Within the scope of this upgrade, we upgraded it to 0.0.18.
• Due to the change in the SQLModel version, we also had to upgrade our SQLAlchemy dependency from V1 to v2. While this does not affect the way that you are using ZenML, if you are using SQLAlchemy in your environment, you might have to migrate your code as well. For a detailed list of changes, feel free to check their migration guide.

Changes in pydantic

Pydantic v2 brings a lot of new and exciting changes to the table. The core logic now uses Rust, and it is much faster and more efficient in terms of performance. On top of it, the main concepts like model design, configuration, validation, or serialization now include a lot of new cool features. If you are using pydantic in your workflow and are interested in the new changes, you can check the brilliant migration guide provided by the pydantic team to see the full list of changes.

Changes in our integrations changes

Much like ZenML, pydantic is an important dependency in many other Python packages. That’s why conducting this upgrade helped us unlock a new version for several ZenML integration dependencies. Additionally, in some instances, we had to adapt the functionality of the integration to keep it compatible with pydantic. So, if you are using any of these integrations, please go through the changes.

Airflow

As mentioned above upgrading our pydantic dependency meant we had to upgrade our sqlmodel dependency. Upgrading our sqlmodel dependency meant we had to upgrade our sqlalchemy dependency as well. Unfortunately, apache-airflow is still using sqlalchemy v1 and is incompatible with pydantic v2. As a solution, we have removed the dependencies of the airflow integration. Now, you can use ZenML to create your Airflow pipelines and use a separate environment to run them with Airflow. You can check the updated docs right here.

AWS

Some of our integrations now require protobuf 4. Since our previous sagemaker version (2.117.0) did not support protobof 4, we could not pair it with these new integrations. Thankfully sagemaker started supporting protobuf 4 with version 2.172.0 and relaxing its dependency solved the compatibility issue.

Evidently

The old version of our evidently integration was not compatible with Pydantic v2. They started supporting it starting from version 0.4.16. As their latest version is 0.4.22, the new dependency of the integration is limited between these two versions.

Feast

Our previous implementation of the feast integration was not compatible with Pydantic v2 due to the extra redis dependency we were using. This extra dependency is now removed and the feast integration is working as intended.

GCP

The previous version of the Kubeflow dependency (kfp==1.8.22) in our GCP integration required Pydantic V1 to be installed. While we were upgrading our Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the kfp dependency to v2 (which has no dependencies on the Pydantic library). This is why you may see some functional changes in the vertex step operator and orchestrator. If you would like to go through the changes in the kfp library, you can find the migration guide here.

Great Expectations

Great Expectations started supporting Pydantic v2 starting from version 0.17.15 and they are closing in on their 1.0 release. Since this release might include a lot of big changes, we adjusted the dependency in our integration to great-expectations>=0.17.15,<1.0. We will try to keep it updated in the future once they release the 1.0 version

Kubeflow

Similar to the GCP integration, the previous version of the kubeflow dependency (kfp==1.8.22) in our kubeflow integration required Pydantic V1 to be installed. While we were upgrading our Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the kfp dependency to v2 (which has no dependencies on the Pydantic library). If you would like to go through the changes in the kfp library, you can find the migration guide here. ( We also are considering adding an alternative version of this integration so our users can keep using kfp V1 in their environment. Stay tuned for any updates.)

MLflow

mlflow is compatible with both Pydantic V1 and v2. However, due to a known issue, if you install zenml first and then do zenml integration install mlflow -y, it downgrades pydantic to V1. This is why we manually added the same duplicated pydantic requirement in the integration definition as well. Keep in mind that the mlflow library is still using some features of pydantic V1 which are deprecated. So, if the integration is installed in your environment, you might run into some deprecation warnings.

Label Studio

While we were working on updating our pydantic dependency, the label-studio-sdk has released its 1.0 version. In this new version, pydantic v2 is also supported. The implementation and documentation of our Label Studio integration have been updated accordingly.

Skypilot

With the switch to pydantic v2, the implementation of our skypilot integration mostly remained untouched. However, due to an incompatibility between the new version pydantic and the azurecli, the skypilot[azure] flavor can not be installed at the same time, thus our skypilot_azure integration is currently deactivated. We are working on fixing this issue and if you are using this integration in your workflows, we recommend staying on the previous version of ZenML until we can solve this issue.

Tensorflow

The new version of pydantic creates a drift between tensorflow and typing_extensions packages and relaxing the dependencies here resolves the issue. At the same time, the upgrade to kfp v2 (in integrations like kubeflow, tekton, or gcp) bumps our protobuf dependency from 3.X to 4.X. To stay compatible with this requirement, the installed version of tensorflow needs to be >=2.12.0. While this change solves the dependency issues in most settings, we have bumped into some errors while using tensorflow 2.12.0 on Python 3.8 on Ubuntu. If you would like to use this integration, please consider using a higher Python version.

Tekton

Similar to the gcp and kubeflow integrations, the old version of our tekton integration was not compatible with pydantic V1 due to its kfp dependency. With the switch from kfp V1 to v2, we have adapted our implementation to use the new version of kfp library and updated our documentation accordingly.

Additional Changes

• We have also released a new version of mlstacks with Pydantic v2 support. If you are using it in your development environment, you have to upgrade your mlstacks package as well.
• Added zenml.integrations.huggingface.steps.run_with_accelerate to enable running any step using accelerate. This function is supported by a utility that wraps any step function into a CLI script (which is required by most distributed training tools).
• Fixed a memory leak that was observed while using the ZenML dashboard to view pipeline logs or artifact visualizations logged through an S3 Artifact Store linked to an AWS Service Connector.
• Previously, we had an option called build_options that allowed users to pass arguments to the docker build command. However, these options were only applied when building the parent image. On macOS with ARM architecture, one needs to specify platform=linux/amd64 to the build command to leverage local caching of Docker image layers. We have added a way to specify these build options for the "main" ZenML build as well, not just the parent image build. Additionally, users can now specify a .dockerignore file for the parent image build, which was previously not possible.

What's Changed

• Extend migration testing by @avishniakov in https://github.com/zenml-io/zenml/pull/2768
• Add retry docs by @htahir1 in https://github.com/zenml-io/zenml/pull/2770
• Fix nightly Docker build by @strickvl in https://github.com/zenml-io/zenml/pull/2769
• Start CTA and Cloud -> Pro renaming by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2773
• Add star CTA to README by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2777
• Use build python version if available by @schustmi in https://github.com/zenml-io/zenml/pull/2775
• Introduced Legacy env var in docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2783
• Fixing the nlp template for the upcoming pydantic upgrade by @bcdurak in https://github.com/zenml-io/zenml/pull/2778
• Full renaming away from cloud to pro by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2782
• Adjust docs url for flavors by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2772
• Fixed broken unit test on develop and fixed duplicate / by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2785
• Added timeout by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2786
• Bump NLP template by @avishniakov in https://github.com/zenml-io/zenml/pull/2787
• Raise error if Dockerfile does not exist by @schustmi in https://github.com/zenml-io/zenml/pull/2776
• Pin numpy<2.0.0 by @avishniakov in https://github.com/zenml-io/zenml/pull/2789
• Fix partial logs loss in step operators with immutable FS in the backend by @avishniakov in https://github.com/zenml-io/zenml/pull/2788
• Upgrading to pydantic v2 by @bcdurak in https://github.com/zenml-io/zenml/pull/2543
• New CI/CD docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2784
• Improvements for running pipelines from the dashboard by @schustmi in https://github.com/zenml-io/zenml/pull/2781
• Accelerate runner helper method by @avishniakov in https://github.com/zenml-io/zenml/pull/2746
• Add --ignore-errors flag for zenml artifact prune by @strickvl in https://github.com/zenml-io/zenml/pull/2780
• Enable running a pipeline through the client by @schustmi in https://github.com/zenml-io/zenml/pull/2736
• Accelerated template LLMs by @avishniakov in https://github.com/zenml-io/zenml/pull/2797
• Separate actions from triggers by @schustmi in https://github.com/zenml-io/zenml/pull/2700
• Fix hook type definition and improve code completion for pipeline decorator by @schustmi in https://github.com/zenml-io/zenml/pull/2793
• Allow specifying build options for main image build by @schustmi in https://github.com/zenml-io/zenml/pull/2749
• Small improvements for yaml config files by @schustmi in https://github.com/zenml-io/zenml/pull/2796
• Docs for the pydantic migration guide by @bcdurak in https://github.com/zenml-io/zenml/pull/2801
• Bump mlflow to v2.14.1 by @christianversloot in https://github.com/zenml-io/zenml/pull/2779
• Bugfix fixing the installation script to use the right mlstacks branch by @bcdurak in https://github.com/zenml-io/zenml/pull/2803
• Fix S3 artifact store memory leak and other improvements by @stefannica in https://github.com/zenml-io/zenml/pull/2802

🥳 Community Contributions 🥳

We'd like to give a special thanks to @christianversloot who contributed to this release by bumping the mlflow version to 2.14.1

Full Changelog: https://github.com/zenml-io/zenml/compare/0.58.2...0.60.0

- Python
Published by htahir1 over 1 year ago

https://github.com/zenml-io/zenml -

The 0.58.2 minor release is packed with a set of improvements to the ZenML logging and ZenML Server.

With this release ZenML logging will: - Offer pagination of the logs during fetching via REST API - Store the full logs history on GCS Artifact Stores - Be performant running logging-heavy tasks, like TQDM logging or logging of training in any Deep Learning framework (also TQDM-backed)

What's Changed

• update test-migrations.sh with latest versions by @safoinme in https://github.com/zenml-io/zenml/pull/2757
• Fix overriding expiration date for api tokens by @schustmi in https://github.com/zenml-io/zenml/pull/2753
• Step logs pagination by @schustmi in https://github.com/zenml-io/zenml/pull/2731
• Fix broken links (round 2) by @strickvl in https://github.com/zenml-io/zenml/pull/2760
• Remove default system flag in docker UV by @avishniakov in https://github.com/zenml-io/zenml/pull/2764
• Another batch of small fixes and expansions by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2762
• Server scalability improvements by @stefannica in https://github.com/zenml-io/zenml/pull/2752
• Add option to start parallel kubernetes steps with delay by @schustmi in https://github.com/zenml-io/zenml/pull/2758
• Move thread_limiter to app startup event by @avishniakov in https://github.com/zenml-io/zenml/pull/2765
• Logging performance improvements and GCP logging fix by @avishniakov in https://github.com/zenml-io/zenml/pull/2755

Full Changelog: https://github.com/zenml-io/zenml/compare/0.58.1...0.58.2

- Python
Published by avishniakov over 1 year ago

https://github.com/zenml-io/zenml - 0.58.1

The 0.58.1 release brings a set of minor enhancements and bugfix to the ZenML framework, such as the ability to delete all pipeline versions using the Client/CLI, providing greater flexibility and control over pipeline management. Users can now specify Python package installer arguments. Furthermore, a fix has been implemented for the Sentencepiece tokenizer materializer.

We are also excited to introduce breadcrumbs to our dashboard to improve your navigation experience. This new feature allows you to easily visualize the path of your Pipelines, Models, and Artifacts, providing clear orientation, quick return to any section with a single click, and effortless navigation.

We’d like to give a special thanks to @eltociear for their first contribution.

Docs re-work

We reworked the structure of our documentation pages to make it easier to find answers to your practical questions. Please do let us know if you have any feedback on the structure or the new style of the 'How To' section!

What's Changed

• Add 0.58.0 to migration testing by @avishniakov in https://github.com/zenml-io/zenml/pull/2730
• Print step names in color, again by @avishniakov in https://github.com/zenml-io/zenml/pull/2728
• Workflow to create JIRA tickets when Github Issues are created by @strickvl in https://github.com/zenml-io/zenml/pull/2724
• Allow specifying python package installer args by @schustmi in https://github.com/zenml-io/zenml/pull/2727
• Send workflow dispatch event to Cloud Plugins repo on release by @wjayesh in https://github.com/zenml-io/zenml/pull/2633
• Fix Nightly Release by @safoinme in https://github.com/zenml-io/zenml/pull/2711
• Fix zenml go images visibility in notebook by @strickvl in https://github.com/zenml-io/zenml/pull/2742
• Handle error when using zenml info with missing dependencies by @strickvl in https://github.com/zenml-io/zenml/pull/2725
• Add Discord Alerter into TOC by @strickvl in https://github.com/zenml-io/zenml/pull/2735
• Allow deleting all versions of a pipeline using the Client/CLI by @schustmi in https://github.com/zenml-io/zenml/pull/2745
• Misc fixes by @schustmi in https://github.com/zenml-io/zenml/pull/2732
• Move full SQLite DB migration test to slow CI by @strickvl in https://github.com/zenml-io/zenml/pull/2743
• Add system flag as default for uv by @schustmi in https://github.com/zenml-io/zenml/pull/2748
• Add how-to section & restructure/update documentation by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2705
• Fix typo in help text by @eltociear in https://github.com/zenml-io/zenml/pull/2750
• Add support for function types in source utils by @schustmi in https://github.com/zenml-io/zenml/pull/2738
• Fix Sentencepiece tokenizer materializer by @safoinme in https://github.com/zenml-io/zenml/pull/2751

New Contributors

• @eltociear made their first contribution in https://github.com/zenml-io/zenml/pull/2750

Full Changelog: https://github.com/zenml-io/zenml/compare/0.58.0...0.58.1

- Python
Published by safoinme over 1 year ago

https://github.com/zenml-io/zenml - 0.58.0

New Annotators

This release brings in three new integrations for our annotator stack component: Prodigy, Argilla and Pigeon. * Pigeon works within Jupyter notebooks and supports a limited feature set but is great for experimentation and demos. * Argilla works both locally-deployed and when the annotation instance lives in the cloud (i.e. in the Hugging Face Spaces deployment which they recommend). * Prodigy is a powerful closed-source annotation tool that allows for efficient data labeling. With this integration, users can now connect ZenML with Prodigy and leverage its annotation capabilities in their ML pipelines.

Retry configuration for steps

This release also includes new retry configuration for the steps. The following parameters can be set:

• **maxretries**_: The maximum number of times the step should be retried in case of failure.
• delay: The initial delay in seconds before the first retry attempt.
• backoff: The factor by which the delay should be multiplied after each retry attempt.

To use this in your code:

```python from zenml.config.retry_config import StepRetryConfig

@step(retry=StepRetryConfig(maxretries=3, delay=10, backoff=2)) def step3() -> None: # Step implementation raise Exception("This is a test exception") ```

or using a config.yaml:

yaml steps: my_step: retry: max_retries: 3 delay: 10 backoff: 2

In addition, this release includes a number of bug fixes and documentation updates, such as a new LLM finetuning template powered by PEFT and BitsAndBytes and instructions for the new annotators.

Breaking changes

• The interface for the base class of the annotator stack component has been updated to account for the fact that not all annotators will launch with a specific URL. So there is no longer an url argument passed in.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @christianversloot who contributed to this release by bumping the mlflow version to 2.12.2

What's Changed

• Add more failure logs for code repositories and build reuse by @schustmi in https://github.com/zenml-io/zenml/pull/2697
• Prodigy annotator by @strickvl in https://github.com/zenml-io/zenml/pull/2655
• Bump mlflow support to version 2.12.2 by @christianversloot in https://github.com/zenml-io/zenml/pull/2693
• add 0.57.1 to migration test scripts by @safoinme in https://github.com/zenml-io/zenml/pull/2702
• Pigeon annotator by @strickvl in https://github.com/zenml-io/zenml/pull/2641
• Allow credentials expiry to be configured for service connectors by @stefannica in https://github.com/zenml-io/zenml/pull/2704
• Argilla annotator by @strickvl in https://github.com/zenml-io/zenml/pull/2687
• Add MySQL and mariadb migration tests to Slow CI by @safoinme in https://github.com/zenml-io/zenml/pull/2686
• Misc small fixes by @schustmi in https://github.com/zenml-io/zenml/pull/2712
• Allow resetting server and user metadata by @schustmi in https://github.com/zenml-io/zenml/pull/2666
• Fix Docker failures in the CI by @avishniakov in https://github.com/zenml-io/zenml/pull/2716
• Add note about helm dependencies by @strickvl in https://github.com/zenml-io/zenml/pull/2709
• Add retry config for failing steps by @safoinme in https://github.com/zenml-io/zenml/pull/2627
• Update pyparsing version by @strickvl in https://github.com/zenml-io/zenml/pull/2710
• New ruff issue by @avishniakov in https://github.com/zenml-io/zenml/pull/2718
• PEFT LLM Template by @avishniakov in https://github.com/zenml-io/zenml/pull/2719
• Add model_version_id as part of the Model config by @avishniakov in https://github.com/zenml-io/zenml/pull/2703
• Add more runners to fast CI by @safoinme in https://github.com/zenml-io/zenml/pull/2706
• Fail faster on notebook installation and only clone / download the branch we need for zenml go by @strickvl in https://github.com/zenml-io/zenml/pull/2721
• Make a clear separation between server and dashboard API in the server configuration by @stefannica in https://github.com/zenml-io/zenml/pull/2722
• Update pymysql to fix CVE-2024-36039 by @stefannica in https://github.com/zenml-io/zenml/pull/2714
• Allow specifying privileged mode for Kubernetes orchestrator containers by @schustmi in https://github.com/zenml-io/zenml/pull/2717
• Don't use pod resources/affinity for kubernetes orchestrator pod by @schustmi in https://github.com/zenml-io/zenml/pull/2707
• Extra test for artifact listing by @avishniakov in https://github.com/zenml-io/zenml/pull/2715
• Pipeline run not tracked in cached artifact version by @avishniakov in https://github.com/zenml-io/zenml/pull/2713

Full Changelog: https://github.com/zenml-io/zenml/compare/0.57.1...0.58.0

- Python
Published by avishniakov over 1 year ago

https://github.com/zenml-io/zenml - 0.57.1

This a minor release that brings a variety of enhancements for the new dashboard release, a new update to the LLMOps guide (covering the use of rerankers in RAG pipelines), and an updated README. It also introduces some new improvements to the service connectors.

We'd like to give a special thanks to @ruvilonix for their first contribution.

What's Changed

• Add new versions to migration testing by @avishniakov in https://github.com/zenml-io/zenml/pull/2663
• Resource settings import fix by @htahir1 in https://github.com/zenml-io/zenml/pull/2664
• Fix env variable for legacy dashboard by @schustmi in https://github.com/zenml-io/zenml/pull/2668
• Fix broken links in code examples by @strickvl in https://github.com/zenml-io/zenml/pull/2672
• Improve error message when trying to unpack a step artifact by @schustmi in https://github.com/zenml-io/zenml/pull/2674
• Prevent special whitespaces in the names of entities by @avishniakov in https://github.com/zenml-io/zenml/pull/2665
• Ensure extra flags aren't passed into uv integration install command by @strickvl in https://github.com/zenml-io/zenml/pull/2670
• enable_cache option shouldn't be set to False for one of the steps by @ruvilonix in https://github.com/zenml-io/zenml/pull/2574
• Add new dashboard links to create/deactivate CLI commands by @avishniakov in https://github.com/zenml-io/zenml/pull/2678
• Add reranking section to LLMOps guide by @strickvl in https://github.com/zenml-io/zenml/pull/2679
• Updated Readme by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2675
• Added Thumbnail by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2684
• [k8s orchestrator] Fix credentials refresh and don't use service connector for incluster auth by @wjayesh in https://github.com/zenml-io/zenml/pull/2671
• Prepare Release 0.57.1 by @safoinme in https://github.com/zenml-io/zenml/pull/2683
• Include email in event by @schustmi in https://github.com/zenml-io/zenml/pull/2692
• Set newsletter flag from email opted in by @schustmi in https://github.com/zenml-io/zenml/pull/2694
• Only report usage once pipeline run starts by @schustmi in https://github.com/zenml-io/zenml/pull/2680
• Reduced thumbnail size by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2689
• Fix intermittent timeout issues with service connector sessions by @stefannica in https://github.com/zenml-io/zenml/pull/2690
• Include unique constraints in the database backup by @stefannica in https://github.com/zenml-io/zenml/pull/2695
• [k8s orch] Add option to specify separate service account for step pods by @wjayesh in https://github.com/zenml-io/zenml/pull/2688
• Update GCP registry docs by @safoinme in https://github.com/zenml-io/zenml/pull/2676
• Use service connector for boto session if possible by @schustmi in https://github.com/zenml-io/zenml/pull/2682
• Send missing user enriched events by @schustmi in https://github.com/zenml-io/zenml/pull/2696

New Contributors

• @ruvilonix made their first contribution in https://github.com/zenml-io/zenml/pull/2574

Full Changelog: https://github.com/zenml-io/zenml/compare/0.57.0...0.57.1

- Python
Published by safoinme almost 2 years ago

https://github.com/zenml-io/zenml - 0.57.0

We're excited to announce that we're open-sourcing our new and improved dashboard. This unifies the experience for OSS and cloud users, though OSS users will initially see some dashboard features unavailable in this launch release.

We're open-sourcing our dashboard for a few reasons:

• to ensure that the dashboard experience is consistent across all users, for both the open-source and cloud versions
• to make it easier for us to maintain and develop the dashboard, as we can share components between the two versions
• to allow OSS contributions (and self-hosting and modifications) to the new dashboard
• to open up possibilities for future features, particularly for our OSS users

New users of the ZenML in the dashboard will have a better experience thanks to a much-improved onboarding sequence:

The dashboard will guide you through connecting to your server, setting up a stack, connecting to service connectors as well as running a pipeline.

We’ve also improved the ‘Settings’ section of the dashboard and this is the new home for configuration of your repositories, secrets, and connectors, along with some other options.

What It Means for You

If you're already a cloud user, not much will change for you. You're already using the new dashboard for pipelines, models and artifacts. Your experience won’t change and for the moment you’ll continue using the old dashboard for certain components (notably for stacks and components).

If you're an open-source user, the new dashboard is now available to you as part of our latest release (0.57.0). You'll notice a completely refreshed design and a new DAG visualizer.

Unfortunately, some dashboard features are not yet ready so you'll see instructions on how to access them via the CLI. We hope to have these features returned into the product soon. (If you have a strong opinion as to which you'd like to see first, please let us know!) Specifically, secrets, stacks, and service connectors are not yet implemented in the new dashboard.

How to use the legacy dashboard

The old dashboard is still available to you. To run with the legacy dashboard pass the --legacy flag when spinning it up:

bash zenml up --legacy

Note that you can’t use both the new and old dashboard at the same time.

If you’re self-hosting ZenML instead of using ZenML Cloud, you can specify which dashboard you want to use by setting the ZEN_SERVER_USE_LEGACY_DASHBOARD environment variable pre-deployment. Specifying a boolean value for this variable will determine which dashboard gets served for your deployment. (There’s no dynamic switching between dashboards allowed, so if you wish to change which dashboard is used for a deployed server, you’ll need to redeploy the server after updating the environment variable.)

If you’re using ZenML Cloud, your experience won’t change with this release and your use of the dashboard remains the same.

What's Changed

• Add Comet to Experiment Trackers in TOC by @strickvl in https://github.com/zenml-io/zenml/pull/2637
• Fix Comet docs formatting by @strickvl in https://github.com/zenml-io/zenml/pull/2639
• ZenML Server activation and user on-boarding by @stefannica in https://github.com/zenml-io/zenml/pull/2630
• Slimmer and more secure Docker container images by @stefannica in https://github.com/zenml-io/zenml/pull/2617
• Add dashboard v2 source context by @schustmi in https://github.com/zenml-io/zenml/pull/2642
• Support New Dashboard release by @avishniakov in https://github.com/zenml-io/zenml/pull/2635
• Fix CI by @strickvl in https://github.com/zenml-io/zenml/pull/2645
• Misc/prepare release 0.57.0rc1 by @avishniakov in https://github.com/zenml-io/zenml/pull/2646
• Add rate limiting to user password reset operations by @stefannica in https://github.com/zenml-io/zenml/pull/2643
• Set zenml server name to default if not customized by @stefannica in https://github.com/zenml-io/zenml/pull/2647
• Docker release fix by @avishniakov in https://github.com/zenml-io/zenml/pull/2649
• Fix dashboard urls by @schustmi in https://github.com/zenml-io/zenml/pull/2648
• Enable analytics during db initialization if specified by @schustmi in https://github.com/zenml-io/zenml/pull/2652
• Better checks for user account updates to avoid Mass Assignment attacks by @stefannica in https://github.com/zenml-io/zenml/pull/2622
• Prepare 0.57.0-rc2 by @avishniakov in https://github.com/zenml-io/zenml/pull/2651
• Fix frontend analytics calls by @schustmi in https://github.com/zenml-io/zenml/pull/2653
• Label studio settings and optional port by @htahir1 in https://github.com/zenml-io/zenml/pull/2628
• Introduce default value fro enable_analytics by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2654
• Fix helm chart notes syntax by @wjayesh in https://github.com/zenml-io/zenml/pull/2656
• Add server env variable to fix activation by @schustmi in https://github.com/zenml-io/zenml/pull/2657
• Respect analytic ENV in local servers by @avishniakov in https://github.com/zenml-io/zenml/pull/2658
• Small fixes in helm docs by @schustmi in https://github.com/zenml-io/zenml/pull/2659

Full Changelog: https://github.com/zenml-io/zenml/compare/0.56.4...0.57.0

- Python
Published by avishniakov almost 2 years ago

https://github.com/zenml-io/zenml - 0.56.4

This release brings a variety of bug fixes and enhancements, including a new Comet Experiment Tracker integration, additional support for the uv package installer for zenml integration ... commands which significantly improves the speed of integration installations and dependency management, and a new evaluation section in the LLMOps guide.

In addition, it includes a number of bug fixes and documentation updates, such as a fix for cached artifacts produced via save_artifact inside steps linkage to the MCP.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @christianversloot who contributed to this release by bumping the mlflow version to 2.12.1

What's Changed

• Fix mariadb test script by @avishniakov in https://github.com/zenml-io/zenml/pull/2599
• Disable CSP headers for the openAPI docs pages and fix API docs building by @stefannica in https://github.com/zenml-io/zenml/pull/2598
• Add short motivating example for RAG pipeline by @strickvl in https://github.com/zenml-io/zenml/pull/2596
• Fix DB backup and restore and add database upgrade testing improvements by @stefannica in https://github.com/zenml-io/zenml/pull/2607
• Fix for #2556 by @avishniakov in https://github.com/zenml-io/zenml/pull/2603
• Fix AWS service connector resource ID regexp by @stefannica in https://github.com/zenml-io/zenml/pull/2611
• Add dry run for docs CI by @avishniakov in https://github.com/zenml-io/zenml/pull/2612
• Completing and refining the CLI documentation by @bcdurak in https://github.com/zenml-io/zenml/pull/2605
• Allow DB backup failures if the database version is 0.56.3 or earlier by @stefannica in https://github.com/zenml-io/zenml/pull/2613
• Mixpanel grouping improvements by @schustmi in https://github.com/zenml-io/zenml/pull/2610
• Add support for uv package installer for zenml integration ... commands by @strickvl in https://github.com/zenml-io/zenml/pull/2609
• Add evaluation section to LLMOps guide by @strickvl in https://github.com/zenml-io/zenml/pull/2614
• Fix GCP commands in docs for project_id by @strickvl in https://github.com/zenml-io/zenml/pull/2616
• Minor fix for GitGuardian warnings. by @bcdurak in https://github.com/zenml-io/zenml/pull/2621
• Bump mlflow to version 2.12.1 by @christianversloot in https://github.com/zenml-io/zenml/pull/2618
• Updated security email by @htahir1 in https://github.com/zenml-io/zenml/pull/2625
• Add Comet Experiment Tracker integration by @strickvl in https://github.com/zenml-io/zenml/pull/2620
• Fix cached artifacts produced via save_artifact inside steps linkage to MCP by @avishniakov in https://github.com/zenml-io/zenml/pull/2619
• Update MCP instructions by @avishniakov in https://github.com/zenml-io/zenml/pull/2632
• Replace parse_obj by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2623
• Fix imports in for Model in documentation by @strickvl in https://github.com/zenml-io/zenml/pull/2631
• Return up-to-date PipelineRunResponse from pipeline run by @avishniakov in https://github.com/zenml-io/zenml/pull/2624

Full Changelog: https://github.com/zenml-io/zenml/compare/0.56.3...0.56.4

- Python
Published by safoinme almost 2 years ago

https://github.com/zenml-io/zenml - 0.56.3

This release comes with a number of bug fixes and enhancements.

With this release you can benefit from new Lambda Labs GPU orchestrator integration in your pipelines. Lambda Labs is a cloud provider that offers GPU instances for machine learning workloads.

In this release we have also implemented a few important security improvements to ZenML Server mostly around Content Security Policies. Also users are from now on mandated to provide previous password during the password change process.

Also the documentation was significantly improved with the new AWS Cloud guide and the LLMOps guide covering various aspects of the LLM lifecycle.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @christianversloot who contributed to this release by adding support for Schedule.start_time to the HyperAI orchestrator.

What's Changed

• Really run migration testing by @avishniakov in https://github.com/zenml-io/zenml/pull/2562
• Interact with feature gate by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2492
• Allow for logs to be unformatted / without colours by @strickvl in https://github.com/zenml-io/zenml/pull/2544
• Add VS Code extension to README / docs by @strickvl in https://github.com/zenml-io/zenml/pull/2568
• Allow loading of artifacts without needing to activate the artifact store (again) by @avishniakov in https://github.com/zenml-io/zenml/pull/2545
• Minor fix by @htahir1 in https://github.com/zenml-io/zenml/pull/2578
• [DOCS] Fix code block in Vertex docs by @wjayesh in https://github.com/zenml-io/zenml/pull/2580
• Added an AWS cloud guide by @htahir1 in https://github.com/zenml-io/zenml/pull/2570
• Update AWS cloud guide by @strickvl in https://github.com/zenml-io/zenml/pull/2581
• More docs fixes by @htahir1 in https://github.com/zenml-io/zenml/pull/2585
• Bugfix for the pyyaml_include version for copier by @bcdurak in https://github.com/zenml-io/zenml/pull/2586
• Update fastapi and orjson to fix python-multipart and orjson vulnerabilities by @stefannica in https://github.com/zenml-io/zenml/pull/2582
• Add security headers to the ZenML server by @stefannica in https://github.com/zenml-io/zenml/pull/2583
• Fix and update AWS cloud guide by @strickvl in https://github.com/zenml-io/zenml/pull/2591
• Add start_time support to HyperAI orchestrator scheduled pipelines by @christianversloot in https://github.com/zenml-io/zenml/pull/2572
• Make secure an optional import by @stefannica in https://github.com/zenml-io/zenml/pull/2592
• RAG guide for docs by @strickvl in https://github.com/zenml-io/zenml/pull/2525
• Update test-migrations scripts with new versions 0.56.2 by @safoinme in https://github.com/zenml-io/zenml/pull/2565
• Check old password during password change and add missing CLI commands by @stefannica in https://github.com/zenml-io/zenml/pull/2587
• Add a note about the f prefix being needed for template strings by @strickvl in https://github.com/zenml-io/zenml/pull/2593
• Skypilot: Lambda Edition by @safoinme in https://github.com/zenml-io/zenml/pull/2526
• Use the correct validity for EKS API tokens and handle long-running Kubernetes pipelines by @stefannica in https://github.com/zenml-io/zenml/pull/2589
• Catch missing jupyter installation for zenml go by @strickvl in https://github.com/zenml-io/zenml/pull/2571
• Allow resources required for the fastapi OpenAPI docs in the CSP header by @stefannica in https://github.com/zenml-io/zenml/pull/2595

Full Changelog: https://github.com/zenml-io/zenml/compare/0.56.2...0.56.3

- Python
Published by avishniakov almost 2 years ago

https://github.com/zenml-io/zenml - 0.56.2

This release introduces a wide array of new features, enhancements, and bug fixes, with a strong emphasis on elevating the user experience and streamlining machine learning workflows. Most notably, you can now deploy models using Hugging Face inference endpoints thanks to an open-source community contribution of this model deployer stack component!

Note that 0.56.0 and 0.56.1 were yanked and removed from PyPI due to an issue with the alembic versions + migration which could affect the database state. This release fixes that issue.

This release also comes with a breaking change to the services architecture.

Breaking Change

A significant change in this release is the migration of the Service (ZenML's technical term for deployment) registration and deployment from local or remote environments to the ZenML server. This change will be reflected in an upcoming tab in the dashboard which will allow users to explore and see the deployed models in the dashboard with their live status and metadata. This architectural shift also simplifies the model deployer abstraction and streamlines the model deployment process for users by moving from limited built-in steps to a more documented and flexible approach.

Important note: If you have models that you previously deployed with ZenML, you might want to redeploy them to have them stored in the ZenML server and tracked by ZenML, ensuring they appear in the dashboard.

Additionally, the find_model_server method now retrieves models (services) from the ZenML server instead of local or remote deployment environments. As a result, any usage of find_model_server will only return newly deployed models stored in the server.

It is also no longer recommended to call service functions like service.start(). Instead, use model_deployer.start_model_server(service_id), which will allow ZenML to update the changed status of the service in the server.

Starting a service

Old syntax: ```python from zenml import pipeline, from zenml.integrations.bentoml.services.bentoml_deployment import BentoMLDeploymentService

@step def predictor( service: BentoMLDeploymentService, ) -> None: # starting the service service.start(timeout=10) ```

New syntax: ```python from zenml import pipeline from zenml.integrations.bentoml.modeldeployers import BentoMLModelDeployer from zenml.integrations.bentoml.services.bentomldeployment import BentoMLDeploymentService

@step def predictor( service: BentoMLDeploymentService, ) -> None: # starting the service modeldeployer = BentoMLModelDeployer.getactivemodeldeployer() modeldeployer.startmodelserver(serviceid=service.service_id, timeout=10) ```

Enabling continuous deployment

Instead of replacing the parameter that was used in the deploy_model method to replace the existing service (if it matches the exact same pipeline name and step name without taking into accounts other parameters or configurations), we now have a new parameter, continuous_deployment_mode, that allows you to enable continuous deployment for the service. This will ensure that the service is updated with the latest version if it's on the same pipeline and step and the service is not already running. Otherwise, any new deployment with different configurations will create a new service.

```python from zenml import pipeline, step, getstepcontext from zenml.client import Client

@step def deploymodel() -> Optional[MLFlowDeploymentService]: # Deploy a model using the MLflow Model Deployer zenmlclient = Client() modeldeployer = zenmlclient.activestack.modeldeployer mlflowdeploymentconfig = MLFlowDeploymentConfig( name: str = "mlflow-model-deployment-example", description: str = "An example of deploying a model using the MLflow Model Deployer", pipelinename: str = getstepcontext().pipelinename, pipelinestepname: str = getstepcontext().stepname, modeluri: str = "runs://model" or "models://", modelname: str = "model", workers: int = 1 mlserver: bool = False timeout: int = DEFAULTSERVICESTARTSTOPTIMEOUT ) service = modeldeployer.deploymodel(mlflowdeploymentconfig, continuousdeploymentmode=True) logger.info(f"The deployed service info: {modeldeployer.getmodelserver_info(service)}") return service ```

Major Features and Enhancements:

• A new Huggingface Model Deployer has been introduced, allowing you to seamlessly deploy your Huggingface models using ZenML. (Thank you so much @dudeperf3ct for the contribution!)
• Faster Integration and Dependency Management ZenML now leverages the uv library, significantly improving the speed of integration installations and dependency management, resulting in a more streamlined and efficient workflow.
• Enhanced Logging and Status Tracking Logging have been improved, providing better visibility into the state of your ZenML services.
• Improved Artifact Store Isolation: ZenML now prevents unsafe operations that access data outside the scope of the artifact store, ensuring better isolation and security.
• Adding admin user notion for the user accounts and added protection to certain operations performed via the REST interface to ADMIN-allowed only.
• Rate limiting for login API to prevent abuse and protect the server from potential security threats.
• The LLM template is now supported in ZenML, allowing you to use the LLM template for your pipelines.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @dudeperf3ct he contributed to this release by introducing the Huggingface Model Deployer. We'd also like to thank @moesio-f for their contribution to this release by adding a new attribute to the Kaniko image builder. Additionally, we'd like to thank @christianversloot for his contributions to this release.

What's Changed

• Upgrading SQLModel to the latest version by @bcdurak in https://github.com/zenml-io/zenml/pull/2452
• Remove KServe integration by @safoinme in https://github.com/zenml-io/zenml/pull/2495
• Upgrade migration testing with 0.55.5 by @avishniakov in https://github.com/zenml-io/zenml/pull/2501
• Relax azure, gcfs and s3 dependencies by @strickvl in https://github.com/zenml-io/zenml/pull/2498
• Use HTTP forwarded headers to detect the real origin of client devices by @stefannica in https://github.com/zenml-io/zenml/pull/2499
• Update README.md for quickstart colab link by @strickvl in https://github.com/zenml-io/zenml/pull/2505
• Add sequential migration tests for MariaDB and MySQL by @strickvl in https://github.com/zenml-io/zenml/pull/2502
• Huggingface Model Deployer by @dudeperf3ct in https://github.com/zenml-io/zenml/pull/2376
• Use uv to speed up pip installs & the CI in general by @strickvl in https://github.com/zenml-io/zenml/pull/2442
• Handle corrupted or empty global configuration file by @stefannica in https://github.com/zenml-io/zenml/pull/2508
• Add admin users notion by @avishniakov in https://github.com/zenml-io/zenml/pull/2494
• Remove dashboard from gitignore by @safoinme in https://github.com/zenml-io/zenml/pull/2517
• Colima / Homebrew fix by @strickvl in https://github.com/zenml-io/zenml/pull/2512
• [HELM] Remove extra environment variable assignment by @wjayesh in https://github.com/zenml-io/zenml/pull/2518
• Allow installing packages using UV by @schustmi in https://github.com/zenml-io/zenml/pull/2510
• Additional fields for track events by @bcdurak in https://github.com/zenml-io/zenml/pull/2507
• Check if environment key is set before deleting in HyperAI orchestrator by @christianversloot in https://github.com/zenml-io/zenml/pull/2511
• Fix the pagination in the database backup by @stefannica in https://github.com/zenml-io/zenml/pull/2522
• Bump mlflow to version 2.11.1 by @christianversloot in https://github.com/zenml-io/zenml/pull/2524
• Add docs for uv installation by @schustmi in https://github.com/zenml-io/zenml/pull/2527
• Fix bug in HyperAI orchestrator depends_on parallelism by @christianversloot in https://github.com/zenml-io/zenml/pull/2523
• Upgrade pip in docker images by @schustmi in https://github.com/zenml-io/zenml/pull/2528
• Fix node selector and other fields for DB job in helm chart by @stefannica in https://github.com/zenml-io/zenml/pull/2531
• Revert "Upgrading SQLModel to the latest version" by @bcdurak in https://github.com/zenml-io/zenml/pull/2515
• Add pod_running_timeout attribute to Kaniko image builder by @moesio-f in https://github.com/zenml-io/zenml/pull/2509
• Add test to install dashboard script by @strickvl in https://github.com/zenml-io/zenml/pull/2521
• Sort pipeline namespaces by last run by @schustmi in https://github.com/zenml-io/zenml/pull/2514
• Add support for LLM template by @schustmi in https://github.com/zenml-io/zenml/pull/2519
• Rate limiting for login API by @avishniakov in https://github.com/zenml-io/zenml/pull/2484
• Try/catch for Docker client by @christianversloot in https://github.com/zenml-io/zenml/pull/2513
• Fix config file in starter guide by @schustmi in https://github.com/zenml-io/zenml/pull/2534
• Log URL for pipelines and model versions when running a pipeline by @wjayesh in https://github.com/zenml-io/zenml/pull/2506
• Add security exclude by @schustmi in https://github.com/zenml-io/zenml/pull/2541
• Update error message around notebook use by @strickvl in https://github.com/zenml-io/zenml/pull/2536
• Cap fsspec for Huggingface integration by @avishniakov in https://github.com/zenml-io/zenml/pull/2542
• Fix integration materializers' URLs in docs by @strickvl in https://github.com/zenml-io/zenml/pull/2538
• Bug fix HyperAI orchestrator: Offload scheduled pipeline execution to bash script by @christianversloot in https://github.com/zenml-io/zenml/pull/2535
• Update pip check command to use uv by @strickvl in https://github.com/zenml-io/zenml/pull/2520
• Implemented bitbucket webhook event source by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2481
• Add ZenMLServiceType and update service registration by @safoinme in https://github.com/zenml-io/zenml/pull/2471
• Prepare release 0.56.0 by @safoinme in https://github.com/zenml-io/zenml/pull/2546
• Fix formatting and release workflow by @strickvl in https://github.com/zenml-io/zenml/pull/2549
• Fix release workflow by @strickvl in https://github.com/zenml-io/zenml/pull/2550
• Fix pipelines and model links for the cloud dashboard by @wjayesh in https://github.com/zenml-io/zenml/pull/2554
• Make starlette non-must for client by @avishniakov in https://github.com/zenml-io/zenml/pull/2553
• Bump MLFlow to version 2.11.2 by @christianversloot in https://github.com/zenml-io/zenml/pull/2552
• Prepare release 0.56.1 by @avishniakov in https://github.com/zenml-io/zenml/pull/2555
• Updated neptune documentation by @SiddhantSadangi in https://github.com/zenml-io/zenml/pull/2548
• 0.56.0 and 0.56.1 in testing by @avishniakov in https://github.com/zenml-io/zenml/pull/2557
• Only install uv once by @schustmi in https://github.com/zenml-io/zenml/pull/2558
• Bump MLFlow to version 2.11.3 by @christianversloot in https://github.com/zenml-io/zenml/pull/2559
• Update docs with warning about pickle materializer insecurity by @avishniakov in https://github.com/zenml-io/zenml/pull/2561
• Add service table migration by @safoinme in https://github.com/zenml-io/zenml/pull/2563

New Contributors

• @dudeperf3ct made their first contribution in https://github.com/zenml-io/zenml/pull/2376
• @moesio-f made their first contribution in https://github.com/zenml-io/zenml/pull/2509
• @SiddhantSadangi made their first contribution in https://github.com/zenml-io/zenml/pull/2548

Full Changelog: https://github.com/zenml-io/zenml/compare/0.55.5...0.56.2

- Python
Published by safoinme almost 2 years ago

https://github.com/zenml-io/zenml - 0.56.1 [YANKED]

[NOTICE] This version introduced the services class that is causing a bug for those users who are migrating from older versions. 0.56.3 will be out shortly in place of this release. For now, this release has been yanked.

This is a patch release aiming to solve a dependency problem that was brought in with the new rate-limiting functionality. With 0.56.1 you no longer need starlette to run client code or to run ZenML CLI commands.

🥳 Community Contributions 🥳

We'd like to thank @christianversloot for his contribution to this release.

What's Changed

• Fix pipelines and model links for the cloud dashboard by @wjayesh in https://github.com/zenml-io/zenml/pull/2554
• Make starlette non-must for client by @avishniakov in https://github.com/zenml-io/zenml/pull/2553
• Bump MLFlow to version 2.11.2 by @christianversloot in https://github.com/zenml-io/zenml/pull/2552

Full Changelog: https://github.com/zenml-io/zenml/compare/0.56.0...0.56.1

- Python
Published by avishniakov almost 2 years ago

https://github.com/zenml-io/zenml - 0.56.0 [YANKED]

[NOTICE] This version introduced the services class that is causing a bug for those users who are migrating from older versions. 0.56.3 will be out shortly in place of this release. For now, this release has been yanked.

ZenML 0.56.0 introduces a wide array of new features, enhancements, and bug fixes, with a strong emphasis on elevating the user experience and streamlining machine learning workflows. Most notably, you can now deploy models using Hugging Face inference endpoints thanks for an open-source community contribution of this model deployer stack component!

This release also comes with a breaking change to the services architecture.

Breaking Change

A significant change in this release is the migration of the Service (ZenML's technical term for deployment) registration and deployment from local or remote environments to the ZenML server. This change will be reflected in an upcoming tab in the dashboard which will allow users to explore and see the deployed models in the dashboard with their live status and metadata. This architectural shift also simplifies the model deployer abstraction and streamlines the model deployment process for users by moving from limited built-in steps to a more documented and flexible approach.

Important note: If you have models that you previously deployed with ZenML, you might want to redeploy them to have them stored in the ZenML server and tracked by ZenML, ensuring they appear in the dashboard.

Additionally, the find_model_server method now retrieves models (services) from the ZenML server instead of local or remote deployment environments. As a result, any usage of find_model_server will only return newly deployed models stored in the server.

It is also no longer recommended to call service functions like service.start(). Instead, use model_deployer.start_model_server(service_id), which will allow ZenML to update the changed status of the service in the server.

Starting a service

Old syntax: ```python from zenml import pipeline, from zenml.integrations.bentoml.services.bentoml_deployment import BentoMLDeploymentService

@step def predictor( service: BentoMLDeploymentService, ) -> None: # starting the service service.start(timeout=10) ```

New syntax: ```python from zenml import pipeline from zenml.integrations.bentoml.modeldeployers import BentoMLModelDeployer from zenml.integrations.bentoml.services.bentomldeployment import BentoMLDeploymentService

@step def predictor( service: BentoMLDeploymentService, ) -> None: # starting the service modeldeployer = BentoMLModelDeployer.getactivemodeldeployer() modeldeployer.startmodelserver(serviceid=service.service_id, timeout=10) ```

Enabling continuous deployment

Instead of replacing the parameter that was used in the deploy_model method to replace the existing service (if it matches the exact same pipeline name and step name without taking into accounts other parameters or configurations), we now have a new parameter, continuous_deployment_mode, that allows you to enable continuous deployment for the service. This will ensure that the service is updated with the latest version if it's on the same pipeline and step and the service is not already running. Otherwise, any new deployment with different configurations will create a new service.

```python from zenml import pipeline, step, getstepcontext from zenml.client import Client

@step def deploymodel() -> Optional[MLFlowDeploymentService]: # Deploy a model using the MLflow Model Deployer zenmlclient = Client() modeldeployer = zenmlclient.activestack.modeldeployer mlflowdeploymentconfig = MLFlowDeploymentConfig( name: str = "mlflow-model-deployment-example", description: str = "An example of deploying a model using the MLflow Model Deployer", pipelinename: str = getstepcontext().pipelinename, pipelinestepname: str = getstepcontext().stepname, modeluri: str = "runs://model" or "models://", modelname: str = "model", workers: int = 1 mlserver: bool = False timeout: int = DEFAULTSERVICESTARTSTOPTIMEOUT ) service = modeldeployer.deploymodel(mlflowdeploymentconfig, continuousdeploymentmode=True) logger.info(f"The deployed service info: {modeldeployer.getmodelserver_info(service)}") return service ```

Major Features and Enhancements:

• A new Huggingface Model Deployer has been introduced, allowing you to seamlessly deploy your Huggingface models using ZenML. (Thank you so much @dudeperf3ct for the contribution!)
• Faster Integration and Dependency Management ZenML now leverages the uv library, significantly improving the speed of integration installations and dependency management, resulting in a more streamlined and efficient workflow.
• Enhanced Logging and Status Tracking Logging have been improved, providing better visibility into the state of your ZenML services.
• Improved Artifact Store Isolation: ZenML now prevents unsafe operations that access data outside the scope of the artifact store, ensuring better isolation and security.
• Adding admin user notion for the user accounts and added protection to certain operations performed via the REST interface to ADMIN-allowed only.
• Rate limiting for login API to prevent abuse and protect the server from potential security threats.
• The LLM template is now supported in ZenML, allowing you to use the LLM template for your pipelines.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @dudeperf3ct he contributed to this release by introducing the Huggingface Model Deployer. We'd also like to thank @moesio-f for their contribution to this release by adding a new attribute to the Kaniko image builder. Additionally, we'd like to thank @christianversloot for his contributions to this release.

All changes:

• Upgrading SQLModel to the latest version by @bcdurak in https://github.com/zenml-io/zenml/pull/2452
• Remove KServe integration by @safoinme in https://github.com/zenml-io/zenml/pull/2495
• Upgrade migration testing with 0.55.5 by @avishniakov in https://github.com/zenml-io/zenml/pull/2501
• Relax azure, gcfs and s3 dependencies by @strickvl in https://github.com/zenml-io/zenml/pull/2498
• Use HTTP forwarded headers to detect the real origin of client devices by @stefannica in https://github.com/zenml-io/zenml/pull/2499
• Update README.md for quickstart colab link by @strickvl in https://github.com/zenml-io/zenml/pull/2505
• Add sequential migration tests for MariaDB and MySQL by @strickvl in https://github.com/zenml-io/zenml/pull/2502
• Huggingface Model Deployer by @dudeperf3ct in https://github.com/zenml-io/zenml/pull/2376
• Use uv to speed up pip installs & the CI in general by @strickvl in https://github.com/zenml-io/zenml/pull/2442
• Handle corrupted or empty global configuration file by @stefannica in https://github.com/zenml-io/zenml/pull/2508
• Add admin users notion by @avishniakov in https://github.com/zenml-io/zenml/pull/2494
• Remove dashboard from gitignore by @safoinme in https://github.com/zenml-io/zenml/pull/2517
• Colima / Homebrew fix by @strickvl in https://github.com/zenml-io/zenml/pull/2512
• [HELM] Remove extra environment variable assignment by @wjayesh in https://github.com/zenml-io/zenml/pull/2518
• Allow installing packages using UV by @schustmi in https://github.com/zenml-io/zenml/pull/2510
• Additional fields for track events by @bcdurak in https://github.com/zenml-io/zenml/pull/2507
• Check if environment key is set before deleting in HyperAI orchestrator by @christianversloot in https://github.com/zenml-io/zenml/pull/2511
• Fix the pagination in the database backup by @stefannica in https://github.com/zenml-io/zenml/pull/2522
• Bump mlflow to version 2.11.1 by @christianversloot in https://github.com/zenml-io/zenml/pull/2524
• Add docs for uv installation by @schustmi in https://github.com/zenml-io/zenml/pull/2527
• Fix bug in HyperAI orchestrator depends_on parallelism by @christianversloot in https://github.com/zenml-io/zenml/pull/2523
• Upgrade pip in docker images by @schustmi in https://github.com/zenml-io/zenml/pull/2528
• Fix node selector and other fields for DB job in helm chart by @stefannica in https://github.com/zenml-io/zenml/pull/2531
• Revert "Upgrading SQLModel to the latest version" by @bcdurak in https://github.com/zenml-io/zenml/pull/2515
• Add pod_running_timeout attribute to Kaniko image builder by @moesio-f in https://github.com/zenml-io/zenml/pull/2509
• Add test to install dashboard script by @strickvl in https://github.com/zenml-io/zenml/pull/2521
• Sort pipeline namespaces by last run by @schustmi in https://github.com/zenml-io/zenml/pull/2514
• Add support for LLM template by @schustmi in https://github.com/zenml-io/zenml/pull/2519
• Rate limiting for login API by @avishniakov in https://github.com/zenml-io/zenml/pull/2484
• Try/catch for Docker client by @christianversloot in https://github.com/zenml-io/zenml/pull/2513
• Fix config file in starter guide by @schustmi in https://github.com/zenml-io/zenml/pull/2534
• Log URL for pipelines and model versions when running a pipeline by @wjayesh in https://github.com/zenml-io/zenml/pull/2506
• Add security exclude by @schustmi in https://github.com/zenml-io/zenml/pull/2541
• Update error message around notebook use by @strickvl in https://github.com/zenml-io/zenml/pull/2536
• Cap fsspec for Huggingface integration by @avishniakov in https://github.com/zenml-io/zenml/pull/2542
• Fix integration materializers' URLs in docs by @strickvl in https://github.com/zenml-io/zenml/pull/2538
• Bug fix HyperAI orchestrator: Offload scheduled pipeline execution to bash script by @christianversloot in https://github.com/zenml-io/zenml/pull/2535
• Update pip check command to use uv by @strickvl in https://github.com/zenml-io/zenml/pull/2520
• Implemented bitbucket webhook event source by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2481
• Add ZenMLServiceType and update service registration by @safoinme in https://github.com/zenml-io/zenml/pull/2471

New Contributors

• @dudeperf3ct made their first contribution in https://github.com/zenml-io/zenml/pull/2376
• @moesio-f made their first contribution in https://github.com/zenml-io/zenml/pull/2509

Full Changelog: https://github.com/zenml-io/zenml/compare/0.55.5...0.56.0

- Python
Published by safoinme almost 2 years ago

https://github.com/zenml-io/zenml - 0.55.5

This patch contains a number of bug fixes and security improvements.

We improved the isolation of artifact stores so that various artifacts cannot be stored or accessed outside of the configured artifact store scope. Such unsafe operations are no longer allowed. This may have an impact on existing codebases if you have used unsafe file operations in the past.

To illustrate such a side effect, let's consider a remote S3 artifact store is configured for the path s3://some_bucket/some_sub_folder and in the code you use artifact_store.open("s3://some_bucket/some_other_folder/dummy.txt","w") -> this operation is considered unsafe as it accesses the data outside the scope of the artifact store. If you really need this to achieve your goals, consider switching to s3fs or similar libraries for such cases.

Also with this release, the server global configuration is no longer stored on the server file system to prevent exposure of sensitive information.

User entities are now uniquely constrained to prevent the creation of duplicate users under certain race conditions.

What's Changed

• Change runnerset name to ubuntu-runners by @safoinme in https://github.com/zenml-io/zenml/pull/2489
• Allow latest ruff versions by @strickvl in https://github.com/zenml-io/zenml/pull/2487
• Uniquely constrained users table by @avishniakov in https://github.com/zenml-io/zenml/pull/2483
• Add option to add base URL for zenml server (with support for cloud) by @wjayesh in https://github.com/zenml-io/zenml/pull/2464
• Improve Artifact Store isolation by @avishniakov in https://github.com/zenml-io/zenml/pull/2490
• Don't write the global config to file on server by @stefannica in https://github.com/zenml-io/zenml/pull/2491
• Add versions for DB migration testing by @strickvl in https://github.com/zenml-io/zenml/pull/2486

Full Changelog: https://github.com/zenml-io/zenml/compare/0.55.4...0.55.5

- Python
Published by avishniakov almost 2 years ago

https://github.com/zenml-io/zenml - 0.55.4

This release brings a host of enhancements and fixes across the board, including significant improvements to our services logging and status, the integration of model saving to the registry via CLI methods, and more robust handling of parallel pipelines and database entities. We've also made strides in optimizing MLflow interactions, enhancing our documentation, and ensuring our CI processes are more robust.

Additionally, we've tackled several bug fixes and performance improvements, making our platform even more reliable and user-friendly.

We'd like to give a special thanks to @christianversloot and @francoisserra for their contributions.

What's Changed

• Bump mlflow to 2.10.2 by @christianversloot in https://github.com/zenml-io/zenml/pull/2444
• Improve services logging and status by @safoinme in https://github.com/zenml-io/zenml/pull/2436
• Add save models to registry setting of a model to CLI methods by @avishniakov in https://github.com/zenml-io/zenml/pull/2447
• Parallel pipelines can create entities in DB by @avishniakov in https://github.com/zenml-io/zenml/pull/2446
• Fix MlFlow TF autlogging excessive warnings by @avishniakov in https://github.com/zenml-io/zenml/pull/2449
• Fix and improve integration deps checker by @stefannica in https://github.com/zenml-io/zenml/pull/2455
• Add migration test version + use self-hosted runners for release by @strickvl in https://github.com/zenml-io/zenml/pull/2450
• Enable running pipeline via REST by @schustmi in https://github.com/zenml-io/zenml/pull/2389
• Faster mlflow list_model_versions by @avishniakov in https://github.com/zenml-io/zenml/pull/2460
• Avoid exposure of tracking uri to metadata by @avishniakov in https://github.com/zenml-io/zenml/pull/2458
• Some important docs updates by @htahir1 in https://github.com/zenml-io/zenml/pull/2463
• Fix CI by @strickvl in https://github.com/zenml-io/zenml/pull/2467
• Fix local Airflow install + docs instructions by @strickvl in https://github.com/zenml-io/zenml/pull/2459
• Update .coderabbit.yaml by @strickvl in https://github.com/zenml-io/zenml/pull/2470
• Prevent templates update from formatting the whole codebase by @avishniakov in https://github.com/zenml-io/zenml/pull/2469
• Telemetry guarding for CI & editable installs by @strickvl in https://github.com/zenml-io/zenml/pull/2468
• Add Vertex Step Operator network parameter by @francoisserra in https://github.com/zenml-io/zenml/pull/2398
• Allow integration export to overwrite a pre-existing file by @strickvl in https://github.com/zenml-io/zenml/pull/2466
• Fix log_model_metadata with explicit name and version by @avishniakov in https://github.com/zenml-io/zenml/pull/2465
• Triggers, actions, event sources - base abstractions and github and pipeline run implementations by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2312
• Mount zenml config path as empty dir by @stefannica in https://github.com/zenml-io/zenml/pull/2472
• Fix broken docs links by @strickvl in https://github.com/zenml-io/zenml/pull/2473
• Use uv pip compile for environment setup in CI by @strickvl in https://github.com/zenml-io/zenml/pull/2474
• MLflow fix for tests on Mac Python 3.9 and 3.10 by @strickvl in https://github.com/zenml-io/zenml/pull/2462
• Improve custom data types docs by @avishniakov in https://github.com/zenml-io/zenml/pull/2476
• Reflect env variables on global configuration by @safoinme in https://github.com/zenml-io/zenml/pull/2371
• Fix zenml deploy secret stores by @safoinme in https://github.com/zenml-io/zenml/pull/2454
• Don't fail when workload manager source fails to load by @schustmi in https://github.com/zenml-io/zenml/pull/2478
• Add analytics events for cloud onboarding by @schustmi in https://github.com/zenml-io/zenml/pull/2456
• Race condition on creating new users allows duplicate usernames by @avishniakov in https://github.com/zenml-io/zenml/pull/2479

Full Changelog: https://github.com/zenml-io/zenml/compare/0.55.3...0.55.4

- Python
Published by strickvl almost 2 years ago

https://github.com/zenml-io/zenml - 0.55.3

This patch comes with a variety of bug fixes and documentation updates.

With this release you can now download files directly from artifact versions that you get back from the client without the need to materialize them. If you would like to bypass materialization entirely and just download the data or files associated with a particular artifact version, you can use the download_files method:

```python from zenml.client import Client

client = Client() artifact = client.getartifactversion(nameidorprefix="irisdataset") artifact.download_files("path/to/save.zip") ```

What's Changed

• Backport: Add HyperAI to TOC (#2406) by @strickvl in https://github.com/zenml-io/zenml/pull/2407
• Fix conditional statements in GitHub workflows by @strickvl in https://github.com/zenml-io/zenml/pull/2404
• Ensure proper spacing in error messages by @christianversloot in https://github.com/zenml-io/zenml/pull/2399
• Fix hyperai markdown table by @strickvl in https://github.com/zenml-io/zenml/pull/2426
• Upgrade Vertex integration google-cloud-aiplatform minimum required version to 1.34.0 by @francoisserra in https://github.com/zenml-io/zenml/pull/2428
• Close code block left open in the docs by @jlopezpena in https://github.com/zenml-io/zenml/pull/2432
• Simplify HF example and notify when cache is down by @safoinme in https://github.com/zenml-io/zenml/pull/2300
• Adding the latest version id and name to the artifact response by @bcdurak in https://github.com/zenml-io/zenml/pull/2430
• Adding the ID of the producer pipeline run to artifact versions by @bcdurak in https://github.com/zenml-io/zenml/pull/2431
• Add vulnerability notice to README by @strickvl in https://github.com/zenml-io/zenml/pull/2437
• REVERTED: Allow more recent adlfs and s3fs versions by @strickvl in https://github.com/zenml-io/zenml/pull/2402
• Add new property for filtering service account events by @strickvl in https://github.com/zenml-io/zenml/pull/2405
• Add download_files method for ArtifactVersion by @strickvl in https://github.com/zenml-io/zenml/pull/2434
• Fixing update_models and revert #2402 by @bcdurak in https://github.com/zenml-io/zenml/pull/2440

Full Changelog: https://github.com/zenml-io/zenml/compare/0.55.2...0.55.3

- Python
Published by strickvl about 2 years ago

https://github.com/zenml-io/zenml - 0.55.2

This patch comes with a variety of new features, bug-fixes, and documentation updates.

Some of the most important changes include:

• The ability to add tags to outputs through the step context
• Allowing the secret stores to utilize the implicit authentication method of AWS/GCP/Azure Service Connectors
• Lazy loading client methods in a pipeline context
• Updates on the Vertex orchestrator to switch to the native VertexAI scheduler
• The new HyperAI integration featuring a new orchestrator and service connector
• Bumping the mlflow version to 2.10.0

We'd like to give a special thanks to @christianversloot and @francoisserra for their contributions.

What's Changed

• 0.55.1 in migration testing by @avishniakov in https://github.com/zenml-io/zenml/pull/2368
• Credential-less AWS/GCP/Azure Secrets Store support by @stefannica in https://github.com/zenml-io/zenml/pull/2365
• Small docs updates by @strickvl in https://github.com/zenml-io/zenml/pull/2359
• generic Client() getters lazy loading by @avishniakov in https://github.com/zenml-io/zenml/pull/2323
• Added slack settings OSSK-382 by @htahir1 in https://github.com/zenml-io/zenml/pull/2378
• Label triggered slow ci by @avishniakov in https://github.com/zenml-io/zenml/pull/2379
• Remove unused is-slow-ci input from fast and slow integration testing by @strickvl in https://github.com/zenml-io/zenml/pull/2382
• Add deprecation warning for ExternalArtifact non-value features by @avishniakov in https://github.com/zenml-io/zenml/pull/2375
• Add telemetry pipeline run ends by @htahir1 in https://github.com/zenml-io/zenml/pull/2377
• Updating the update_model decorator by @bcdurak in https://github.com/zenml-io/zenml/pull/2136
• Mocked API docs building by @avishniakov in https://github.com/zenml-io/zenml/pull/2360
• Add outputs tags function by @avishniakov in https://github.com/zenml-io/zenml/pull/2383
• Bump mlflow to v2.10.0 by @christianversloot in https://github.com/zenml-io/zenml/pull/2374
• Fix sharing of model versions by @schustmi in https://github.com/zenml-io/zenml/pull/2380
• Fix GCP service connector login to overwrite existing valid credentials by @stefannica in https://github.com/zenml-io/zenml/pull/2392
• Update has_custom_name for legacy artifacts by @avishniakov in https://github.com/zenml-io/zenml/pull/2384
• Use native VertexAI scheduler capability instead of old GCP official workaround by @francoisserra in https://github.com/zenml-io/zenml/pull/2310
• HyperAI integration: orchestrator and service connector by @christianversloot in https://github.com/zenml-io/zenml/pull/2372

Full Changelog: https://github.com/zenml-io/zenml/compare/0.55.1...0.55.2

- Python
Published by bcdurak about 2 years ago

https://github.com/zenml-io/zenml - 0.55.1

If you are actively using the Model Control Plane features, we suggest that you directly upgrade to 0.55.1, bypassing 0.55.0.

This is a patch release bringing backward compatibility for breaking changes introduced in 0.55.0, so that appropriate migration actions can be performed at the desired pace. Please refer to the 0.55.0 release notes for specific information on breaking changes and how to update your code to align with the new way of doing things. We also have updated our documentation to serve you better and introduced PipelineNamespace models in our API.

Also, this release is packed with Database recovery in case the upgrade failed to migrate the Database to a newer version of ZenML.

What's Changed

• Update skypilot docs by @safoinme in https://github.com/zenml-io/zenml/pull/2344
• Fast CI / Slow CI by @strickvl in https://github.com/zenml-io/zenml/pull/2268
• Add repeating tests and instafail error logging to testing in CI by @strickvl in https://github.com/zenml-io/zenml/pull/2334
• Added more info about metadata by @htahir1 in https://github.com/zenml-io/zenml/pull/2346
• Use GitHub as trusted publisher for PyPI publication by @strickvl in https://github.com/zenml-io/zenml/pull/2343
• Fix code in docs/questions about MCP by @wjayesh in https://github.com/zenml-io/zenml/pull/2340
• Update release notes for 0.55.0 by @avishniakov in https://github.com/zenml-io/zenml/pull/2351
• Fixed metadata docs by @htahir1 in https://github.com/zenml-io/zenml/pull/2350
• Add generate test duration file cron by @safoinme in https://github.com/zenml-io/zenml/pull/2347
• CI comments for slow CI and more conditional membership checking by @strickvl in https://github.com/zenml-io/zenml/pull/2356
• Backward compatible ModelVersion by @avishniakov in https://github.com/zenml-io/zenml/pull/2357
• Add model version created to analytics by @avishniakov in https://github.com/zenml-io/zenml/pull/2352
• Make CI run on the appropriate branch by @strickvl in https://github.com/zenml-io/zenml/pull/2358
• Add MVP pipeline namespace support by @schustmi in https://github.com/zenml-io/zenml/pull/2353
• Apply docker run args to skypilot orchestrator VM by @schustmi in https://github.com/zenml-io/zenml/pull/2342
• 📝 Minor docs improvements (basic step example) by @plattenschieber in https://github.com/zenml-io/zenml/pull/2348
• Add DB backup and recovery during DB schema migrations by @wjayesh in https://github.com/zenml-io/zenml/pull/2158
• Fix CI issues by @strickvl in https://github.com/zenml-io/zenml/pull/2363

New Contributors

• @plattenschieber made their first contribution in https://github.com/zenml-io/zenml/pull/2348

Full Changelog: https://github.com/zenml-io/zenml/compare/0.55.0...0.55.1

- Python
Published by avishniakov about 2 years ago

https://github.com/zenml-io/zenml - 0.43.1

Backports some important fixes that have been introduced in more recent versions of ZenML to the 0.43.x release line.

Full Changelog: https://github.com/zenml-io/zenml/compare/0.43.0...0.43.1

- Python
Published by strickvl about 2 years ago

https://github.com/zenml-io/zenml - 0.42.2

Backports some important fixes that have been introduced in more recent versions of ZenML to the 0.42.x release line.

Full Changelog: https://github.com/zenml-io/zenml/compare/0.42.1...0.42.2

- Python
Published by strickvl about 2 years ago

https://github.com/zenml-io/zenml - 0.55.0

This release comes with a range of new features, bug fixes and documentation updates. The most notable changes are the ability to do lazy loading of Artifacts and their Metadata and Model and its Metadata inside the pipeline code using pipeline context object, and the ability to link Artifacts to Model Versions implicitly via the save_artifact function.

Additionally, we've updated the documentation to include a new starter guide on how to manage artifacts, and a new production guide that walks you through how to configure your pipelines to run in production.

Breaking Change

The ModelVersion concept was renamed to Model going forward, which affects code bases using the Model Control Plane feature. This change is not backward compatible.

Pipeline decorator

@pipeline(model_version=ModelVersion(...)) -> @pipeline(model=Model(...))

Old syntax: ```python from zenml import pipeline, ModelVersion

@pipeline(modelversion=ModelVersion(name="modelname",version="v42")) def p(): ... ```

New syntax: ```python from zenml import pipeline, Model

@pipeline(model=Model(name="model_name",version="v42")) def p(): ... ```

Step decorator

@step(model_version=ModelVersion(...)) -> @step(model=Model(...))

Old syntax: ```python from zenml import step, ModelVersion

@step(modelversion=ModelVersion(name="modelname",version="v42")) def s(): ... ```

New syntax: ```python from zenml import step, Model

@step(model=Model(name="model_name",version="v42")) def s(): ... ```

Acquiring model configuration from pipeline/step context

Old syntax: ```python from zenml import pipeline, step, ModelVersion, getstepcontext, getpipelinecontext

@pipeline(modelversion=ModelVersion(name="modelname",version="v42")) def p(): modelversion = getpipelinecontext().modelversion ...

@step(modelversion=ModelVersion(name="modelname",version="v42")) def s(): modelversion = getstepcontext().modelversion ... ```

New syntax: ```python from zenml import pipeline, step, Model, getstepcontext, getpipelinecontext

@pipeline(model=Model(name="modelname",version="v42")) def p(): model = getpipeline_context().model ...

@step(model=Model(name="modelname",version="v42")) def s(): model = getstep_context().model ... ```

Usage of model configuration inside pipeline YAML config file

Old syntax: yaml model_version: name: model_name version: v42 ...

New syntax: yaml model: name: model_name version: v42 ...

ModelVersion.metadata -> Model.run_metadata

Old syntax: ```python from zenml import ModelVersion

def s(): modelversion = ModelVersion(name="modelname",version="production") somemetadata = modelversion.metadata["some_metadata"].value ... ```

New syntax: ```python from zenml import Model

def s(): model = Model(name="modelname",version="production") somemetadata = model.runmetadata["somemetadata"].value ... ```

Those using the older syntax are requested to update their code accordingly.

Full set of changes are highlighted here: https://github.com/zenml-io/zenml/pull/2267

What's Changed

• Remove --name from service account creation in docs by @christianversloot in https://github.com/zenml-io/zenml/pull/2295
• Secrets store hot backup and restore by @stefannica in https://github.com/zenml-io/zenml/pull/2277
• Updating the README of the e2e template by @bcdurak in https://github.com/zenml-io/zenml/pull/2299
• Add missing docstring for Skypilot setting by @schustmi in https://github.com/zenml-io/zenml/pull/2305
• Update Manage artifacts starter guide docs by @JonathanLoscalzo in https://github.com/zenml-io/zenml/pull/2301
• Add some tiny details and move around a page by @htahir1 in https://github.com/zenml-io/zenml/pull/2297
• Model links lazy evaluation in pipeline code by @avishniakov in https://github.com/zenml-io/zenml/pull/2205
• Link artifact to MCP entity via function call or implicitly in save_artifact by @avishniakov in https://github.com/zenml-io/zenml/pull/2298
• Extend MCP/ACP listing capabilities by @avishniakov in https://github.com/zenml-io/zenml/pull/2285
• Add the latest zenml version to migration testing scripts by @strickvl in https://github.com/zenml-io/zenml/pull/2294
• Remove Python 3.7 check for Langchain Integration by @strickvl in https://github.com/zenml-io/zenml/pull/2308
• Allow spellcheck to run for docs changes by @strickvl in https://github.com/zenml-io/zenml/pull/2307
• Add helper message for zenml up --blocking login by @strickvl in https://github.com/zenml-io/zenml/pull/2290
• Fix secret migration from an external store in helm deployment by @stefannica in https://github.com/zenml-io/zenml/pull/2315
• Small docs fixes by @htahir1 in https://github.com/zenml-io/zenml/pull/2314
• Rename model version to a model by @avishniakov in https://github.com/zenml-io/zenml/pull/2267
• Updating the docs after the Skypilot tests by @bcdurak in https://github.com/zenml-io/zenml/pull/2311
• Remove unused Segment / Mixpanel generation workflow and script by @strickvl in https://github.com/zenml-io/zenml/pull/2319
• Add log_step_metadata utility function by @strickvl in https://github.com/zenml-io/zenml/pull/2322
• Add conditional checks to prevent scheduled actions running inside forked repositories by @strickvl in https://github.com/zenml-io/zenml/pull/2317
• RBAC resource sharing by @schustmi in https://github.com/zenml-io/zenml/pull/2320
• Fix typo in migration downgrade by @avishniakov in https://github.com/zenml-io/zenml/pull/2337
• Separate skypilot flavors into different folders by @safoinme in https://github.com/zenml-io/zenml/pull/2332
• Add warning for GCP integration when using Python >=3.11 by @strickvl in https://github.com/zenml-io/zenml/pull/2333

New Contributors

• @JonathanLoscalzo made their first contribution at https://github.com/zenml-io/zenml/pull/2301

Full Changelog: https://github.com/zenml-io/zenml/compare/0.54.1...0.55.0

- Python
Published by safoinme about 2 years ago

https://github.com/zenml-io/zenml - 0.54.1

Release 0.54.1, includes a mix of updates new additions, and bug fixes. The most notable changes are the new production guide, allowing multi-step VMs for the Skypilot orchestrator which allows you to configure a step to run on a specific VM or run the entire pipeline on a single VM, and some improvements to the Model Control Plane.

What's Changed

• Bump aquasecurity/trivy-action from 0.16.0 to 0.16.1 by @dependabot in https://github.com/zenml-io/zenml/pull/2244
• Bump crate-ci/typos from 1.16.26 to 1.17.0 by @dependabot in https://github.com/zenml-io/zenml/pull/2245
• Add YAML formatting standardisation to formatting & linting scripts by @strickvl in https://github.com/zenml-io/zenml/pull/2224
• Remove text annotation by @strickvl in https://github.com/zenml-io/zenml/pull/2246
• Add MariaDB migration testing by @strickvl in https://github.com/zenml-io/zenml/pull/2170
• Delete artifact links from model version via Client, ModelVersion and API by @avishniakov in https://github.com/zenml-io/zenml/pull/2191
• Default/Non-Default step params produce conflict with yaml ones as defaults are set in code by @avishniakov in https://github.com/zenml-io/zenml/pull/2247
• Prune of unused artifacts links via client by @avishniakov in https://github.com/zenml-io/zenml/pull/2192
• Rename nlp example by @safoinme in https://github.com/zenml-io/zenml/pull/2221
• Support refreshing service connector credentials in the Vertex step operator to support long-running jobs by @stefannica in https://github.com/zenml-io/zenml/pull/2198
• Refactor secrets stores to store all secret metadata in the DB by @stefannica in https://github.com/zenml-io/zenml/pull/2193
• Add latest_version_id to the ModelResponse by @avishniakov in https://github.com/zenml-io/zenml/pull/2266
• Remove link_artifact from docs for MCP by @strickvl in https://github.com/zenml-io/zenml/pull/2272
• Improve action by adding advice to KeyError when configured steps are not present in pipeline by @christianversloot in https://github.com/zenml-io/zenml/pull/2265
• Allow multi step configuration for skypilot by @safoinme in https://github.com/zenml-io/zenml/pull/2166
• Reworking the examples by @bcdurak in https://github.com/zenml-io/zenml/pull/2259
• A docs update for incorrect import in docs/book/user-guide/starter-guide/track-ml-models.md by @yo-harsh in https://github.com/zenml-io/zenml/pull/2279
• Allow sklearn versions > 1.3 by @Vishal-Padia in https://github.com/zenml-io/zenml/pull/2271
• Free sklearn dependency to allow all versions by @strickvl in https://github.com/zenml-io/zenml/pull/2281
• Misc CI bugfixes by @strickvl in https://github.com/zenml-io/zenml/pull/2260
• Fix yamlfix script to use --no-yamlfix flag by @strickvl in https://github.com/zenml-io/zenml/pull/2280
• Fix dependabot settings autoformatting by yamlfix by @strickvl in https://github.com/zenml-io/zenml/pull/2282
• Add advice for next step to error on AuthorizationException by @christianversloot in https://github.com/zenml-io/zenml/pull/2264
• Allow skypilot to configure step or run full pipeline in one VM by @safoinme in https://github.com/zenml-io/zenml/pull/2276
• A docs update with production guide + restructured advanced guide by @htahir1 in https://github.com/zenml-io/zenml/pull/2232
• Prepare release 0.54.1 by @safoinme in https://github.com/zenml-io/zenml/pull/2284
• Add Escan to teams.yml by @strickvl in https://github.com/zenml-io/zenml/pull/2283
• Fix typos from docs updates by @strickvl in https://github.com/zenml-io/zenml/pull/2287
• Fix to file path for notebook quickstart by @jacksandom in https://github.com/zenml-io/zenml/pull/2288
• Fix zenml-tutorial path assignment in cli.go() by @safoinme in https://github.com/zenml-io/zenml/pull/2291

New Contributors

• @yo-harsh made their first contribution in https://github.com/zenml-io/zenml/pull/2279
• @Vishal-Padia made their first contribution in https://github.com/zenml-io/zenml/pull/2271
• @jacksandom made their first contribution in https://github.com/zenml-io/zenml/pull/2288

Full Changelog: https://github.com/zenml-io/zenml/compare/0.54.0...0.54.1

- Python
Published by safoinme about 2 years ago

https://github.com/zenml-io/zenml - 0.54.0

This release brings a range of new features, bug fixes and documentation updates. The Model Control Plane has received a number of small bugfixes and improvements, notably the ability to change model and model version names.

We've also added a whole new starter guide that walks you through how to get started with ZenML, from creating your first pipeline to fetching objects once your pipelines have run and much more. Be sure to check it out if you're new to ZenML!

Speaking of documentation improvements, the Model Control Plane now has its own dedicated documentation section introducing the concepts and features of the Model Control Plane.

As always, this release comes with number of bug fixes, docs additions and smaller improvements to our internal processes.

Breaking Change

This PR introduces breaking changes in the areas of the REST API concerning secrets and tags. As a consequence, the ZenML Client running the previous ZenML version is no longer compatible with a ZenML Server running the new version and vice-versa. To address this, simply ensure that all your ZenML clients use the same version as the server(s) they connect to.

🥳 Community Contributions 🥳

We'd like to give a special thanks to @christianversloot for two PRs he contributed to this release. One of them fixes a bug that prevented ZenML from running on Windows and the other one adds a new materializer for the Polars library.

Also many thanks to @sean-hickey-wf for his contribution of an improvement to the Slack Alerter stack component which allows you to define custom blocks for the Slack message.

What's Changed

• Completing the hydration story with the remaining models by @bcdurak in https://github.com/zenml-io/zenml/pull/2151
• Remove secrets manager flavors from DB by @stefannica in https://github.com/zenml-io/zenml/pull/2182
• Prepare 0.53.1 release by @stefannica in https://github.com/zenml-io/zenml/pull/2183
• Update package name for nightly build by @strickvl in https://github.com/zenml-io/zenml/pull/2172
• Remove space saver action + upgrade other actions by @strickvl in https://github.com/zenml-io/zenml/pull/2174
• mutable names in Model and MV by @avishniakov in https://github.com/zenml-io/zenml/pull/2185
• Fix image building for nightly container builds by @strickvl in https://github.com/zenml-io/zenml/pull/2189
• Test that artifacts not get linked to model version not from context by @avishniakov in https://github.com/zenml-io/zenml/pull/2188
• Warn if Model(Version) config fluctuates from DB state by @avishniakov in https://github.com/zenml-io/zenml/pull/2144
• Add blocks field to SlackAlerterParameters for custom slack blocks by @sean-hickey-wf in https://github.com/zenml-io/zenml/pull/2153
• Model control plane technical documentation by @strickvl in https://github.com/zenml-io/zenml/pull/2111
• Alembic branching issue fix by @avishniakov in https://github.com/zenml-io/zenml/pull/2197
• Bump github/codeql-action from 2 to 3 by @dependabot in https://github.com/zenml-io/zenml/pull/2201
• Bump google-github-actions/get-gke-credentials from 0 to 2 by @dependabot in https://github.com/zenml-io/zenml/pull/2202
• Bump google-github-actions/auth from 1 to 2 by @dependabot in https://github.com/zenml-io/zenml/pull/2203
• Bump aws-actions/amazon-ecr-login from 1 to 2 by @dependabot in https://github.com/zenml-io/zenml/pull/2200
• Bump crate-ci/typos from 1.16.25 to 1.16.26 by @dependabot in https://github.com/zenml-io/zenml/pull/2207
• Fix unreliable test behaviour when using hypothesis by @strickvl in https://github.com/zenml-io/zenml/pull/2208
• Added more pod spec properties for k8s orchestrator by @htahir1 in https://github.com/zenml-io/zenml/pull/2097
• Fix API docs environment setup by @strickvl in https://github.com/zenml-io/zenml/pull/2190
• Use placeholder runs to show pipeline runs in the dashboard without delay by @schustmi in https://github.com/zenml-io/zenml/pull/2048
• Update README and CONTRIBUTING.md docs with links to good first issues for contribution by @strickvl in https://github.com/zenml-io/zenml/pull/2220
• Bump supported mlstacks version to 0.8.0 by @strickvl in https://github.com/zenml-io/zenml/pull/2196
• Misc cleanup by @schustmi in https://github.com/zenml-io/zenml/pull/2126
• Refactor pipeline run updates by @schustmi in https://github.com/zenml-io/zenml/pull/2117
• Rename logmodelversionmetadata to logmodel_metadata by @htahir1 in https://github.com/zenml-io/zenml/pull/2215
• Update starter and create new production guide by @htahir1 in https://github.com/zenml-io/zenml/pull/2143
• Fix typo by @strickvl in https://github.com/zenml-io/zenml/pull/2223
• Consolidate Custom Filter Logic by @fa9r in https://github.com/zenml-io/zenml/pull/2116
• Force forward slashes when saving artifacts by @christianversloot in https://github.com/zenml-io/zenml/pull/2195
• Temporarily disable two MLflow tests for MacOS with Python 3.9 and 3.10 by @strickvl in https://github.com/zenml-io/zenml/pull/2186
• Disable template updates for forked repositories by @strickvl in https://github.com/zenml-io/zenml/pull/2222
• Remove Label Studio text annotation example by @strickvl in https://github.com/zenml-io/zenml/pull/2225
• Add scarf checker script and CI workflow by @strickvl in https://github.com/zenml-io/zenml/pull/2227
• Add mlstacks installation instructions to docs by @strickvl in https://github.com/zenml-io/zenml/pull/2228
• Adding the hydrate flag to the client methods by @bcdurak in https://github.com/zenml-io/zenml/pull/2120
• Fixing the remaining docs pages for run_metadata by @bcdurak in https://github.com/zenml-io/zenml/pull/2230
• Fix CI check to disallow template testing on forked repositories by @strickvl in https://github.com/zenml-io/zenml/pull/2231
• Fix fork check syntax by @strickvl in https://github.com/zenml-io/zenml/pull/2237
• Add missing annotations section to zenml service account by @wjayesh in https://github.com/zenml-io/zenml/pull/2234
• Allow filtering artifacts with/without custom names by @schustmi in https://github.com/zenml-io/zenml/pull/2226
• Adjust migration settings based on database engine by @strickvl in https://github.com/zenml-io/zenml/pull/2236
• Added one more chapter to starter guide by @htahir1 in https://github.com/zenml-io/zenml/pull/2238
• Add Polars materializer by @christianversloot in https://github.com/zenml-io/zenml/pull/2229

New Contributors

• @sean-hickey-wf made their first contribution in https://github.com/zenml-io/zenml/pull/2153
• @dependabot 🤖 made their first contribution in https://github.com/zenml-io/zenml/pull/2201

Full Changelog: https://github.com/zenml-io/zenml/compare/0.53.1...0.54.0

- Python
Published by strickvl about 2 years ago

https://github.com/zenml-io/zenml - 0.53.1

This minor release contains a hot fix for a bug that was introduced in 0.53.0 where the secrets manager flavors were not removed from the database properly. This release fixes that issue.

What's Changed

• Remove secrets manager flavors from DB by @stefannica in https://github.com/zenml-io/zenml/pull/2182

Full Changelog: https://github.com/zenml-io/zenml/compare/0.53.0...0.53.1

- Python
Published by stefannica about 2 years ago

https://github.com/zenml-io/zenml - 0.53.0

This release is packed with a deeply reworked quickstart example and starter template, the removal of secret manager stack component, improved experience with Cloud Secret Stores, support for tags and metadata directly in Model Versions, some breaking changes for Model Control Plane and a few bugfixes.

New features

Try out the new quickstart and starter template featuring the Model Control Plane

Maybe it slipped you by, but ZenML has a brand new feature, and it's called the Model Control Plane (MCP)! The MCP allows users to register their Models directly into their zenml servers, and associate metadata, runs, and artifacts with it. This gives users a model-centric view into ZenML, rather than a pipeline-centric one. Try it out now with the quickstart:

``` pip install "zenml[server,templates]" mkdir zenmlstartertemplate cd zenmlstartertemplate copier copy --trust -r 2023.12.18 https://github.com/zenml-io/template-starter.git .

or zenml init --template starter

Now either read the README or run quickstart.ipynb

```

Alternatively, just clone the repo and run the quickstart.

Please note that while frontend features are Cloud Only, OSS users can still use the MCP via the CLI and Python SDK.

Breaking changes

Secret Manager stack components sunset

Upon upgrading, all Secrets Manager stack components will be removed from the Stacks that still contain them and from the database. This also implies that access to any remaining secrets managed through Secrets Manager stack components will be lost. If you still have secrets configured and managed through Secrets Manager stack components, please consider migrating all your existing secrets to the centralized secrets store before upgrading by means of the zenml secrets-manager secret migrate CLI command. Also see the zenml secret --help command for more information.

Renaming "endpoints" to "deployments" in Model Control Plane

This is just a renaming to provide better alignment with industry standards. Though, it will affect: - ArtifactConfig(..., is_endpoint_artifact=True) now is ArtifactConfig(..., is_deployment_artifact=True) - CLI command zenml model endpoint_artifacts ... now is zenml model deployment_artifacts ... - Client().list_model_version_artifact_links(..., only_endpoint_artifacts=True) now is Client().list_model_version_artifact_links(..., only_deployment_artifacts=True) - ModelVersion(...).get_endpoint_artifact(...) now is ModelVersion(...).get_deployment_artifact(...)

Major bugfixes

• Fix various bugs by @stefannica in https://github.com/zenml-io/zenml/pull/2147
• Adding a link from pipeline runs to code repositories by @bcdurak in https://github.com/zenml-io/zenml/pull/2146
• Fix Client doesn't recover from remote connection resets by @avishniakov in https://github.com/zenml-io/zenml/pull/2129
• Bugfix: run_metadata value returns string instead of other types by @avishniakov in https://github.com/zenml-io/zenml/pull/2149
• KubernetesSparkStepOperator imports fails by @avishniakov in https://github.com/zenml-io/zenml/pull/2159
• Fix get_pipeline_context().model_version.get_artifact(...) flow by @avishniakov in https://github.com/zenml-io/zenml/pull/2162

What's Changed

• Model Versions are taggable by @avishniakov in https://github.com/zenml-io/zenml/pull/2102
• Adding a condition to the PR template by @bcdurak in https://github.com/zenml-io/zenml/pull/2140
• trying local caching for custom runners by @safoinme in https://github.com/zenml-io/zenml/pull/2148
• make template tests runs on ubuntu latest instead of custom runners by @safoinme in https://github.com/zenml-io/zenml/pull/2150
• Fix various bugs by @stefannica in https://github.com/zenml-io/zenml/pull/2147
• Fix importlib calling to importlib.metadata by @safoinme in https://github.com/zenml-io/zenml/pull/2160
• Debugging zenml clean by @bcdurak in https://github.com/zenml-io/zenml/pull/2119
• Add metadata to model versions by @avishniakov in https://github.com/zenml-io/zenml/pull/2109
• Adding a link from pipeline runs to code repositories by @bcdurak in https://github.com/zenml-io/zenml/pull/2146
• Moving tags to the body for artifacts and artifact versions by @bcdurak in https://github.com/zenml-io/zenml/pull/2138
• Fix MLFlow test by @avishniakov in https://github.com/zenml-io/zenml/pull/2161
• Fix Client doesn't recover from remote connection resets by @avishniakov in https://github.com/zenml-io/zenml/pull/2129
• Bugfix: run_metadata value returns string instead of other types by @avishniakov in https://github.com/zenml-io/zenml/pull/2149
• KubernetesSparkStepOperator imports fails by @avishniakov in https://github.com/zenml-io/zenml/pull/2159
• Endpoint artifacts rename to deployment artifacts by @avishniakov in https://github.com/zenml-io/zenml/pull/2134
• Fix get_pipeline_context().model_version.get_artifact(...) flow by @avishniakov in https://github.com/zenml-io/zenml/pull/2162
• Add CodeRabbit config to repo base by @strickvl in https://github.com/zenml-io/zenml/pull/2165
• Feature: use service connectors to authenticate secrets stores. by @stefannica in https://github.com/zenml-io/zenml/pull/2154
• Add dependabot updates for Github Actions on CI by @strickvl in https://github.com/zenml-io/zenml/pull/2087
• Run DB migration testing using MySQL alongside SQLite by @strickvl in https://github.com/zenml-io/zenml/pull/2113
• Remove precommit by @strickvl in https://github.com/zenml-io/zenml/pull/2164
• Remove support for secrets managers by @stefannica in https://github.com/zenml-io/zenml/pull/2163
• Add MariaDB test harnesses by @christianversloot in https://github.com/zenml-io/zenml/pull/2155
• Feature/update quickstart from template by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2157
• Bump MLFlow to 2.9.2 by @christianversloot in https://github.com/zenml-io/zenml/pull/2156

Full Changelog: https://github.com/zenml-io/zenml/compare/0.52.0...0.53.0

- Python
Published by avishniakov about 2 years ago

https://github.com/zenml-io/zenml - 0.52.0

This release adds the ability to pass in pipeline parameters as YAML configuration and fixes a couple of minor issues affecting the W&B integration and the way expiring credentials are refreshed when service connectors are used.

Breaking Change

The current pipeline YAML configurations are now being validated to ensure that configured parameters match what is available in the code. This means that if you have a pipeline that is configured with a parameter that has a different value that what is provided through code, the pipeline will fail to run. This is a breaking change, but it is a good thing as it will help you catch errors early on.

This is an example of a pipeline configuration that will fail to run:

```yaml parameters: some_param: 24

steps: mystep: parameters: input2: 42 ```

```python

run.py

@step def mystep(input1: int, input_2: int) -> None: pass

@pipeline def mypipeline(someparam: int): # here an error will be raised since input_2 is # 42 in config, but 43 was provided in the code mystep(input1=42, input_2=43)

if name=="main": # here an error will be raised since some_param is # 24 in config, but 23 was provided in the code my_pipeline(23) ```

What's Changed

• Passing pipeline parameters as yaml config by @avishniakov in https://github.com/zenml-io/zenml/pull/2058
• Side-effect free tests by @avishniakov in https://github.com/zenml-io/zenml/pull/2065
• Fix various bugs by @stefannica in https://github.com/zenml-io/zenml/pull/2124

Full Changelog: https://github.com/zenml-io/zenml/compare/0.51.0...0.52.0

- Python
Published by stefannica about 2 years ago

https://github.com/zenml-io/zenml - 0.51.0

This release comes with a breaking change to the model version model, a new use-case example for NLP, and a range of bug fixes and enhancements to the artifact management and pipeline run management features.

Breaking Change

• Artifact Version Table + Artifact Tagging by @fa9r in https://github.com/zenml-io/zenml/pull/2081
• Converting model models to use the new hydration paradigm by @bcdurak in https://github.com/zenml-io/zenml/pull/2101

New Example

• NLP Template Example is a new example that demonstrates how to use ZenML for NLP tasks. by @safoinme in https://github.com/zenml-io/zenml/pull/2070

What's Changed

• Updated to one quickstart again by @htahir1 in https://github.com/zenml-io/zenml/pull/2092
• Fix Nightly Build workflow files by @strickvl in https://github.com/zenml-io/zenml/pull/2090
• Make PyPi release depend on DB migration tests passing by @strickvl in https://github.com/zenml-io/zenml/pull/2088
• Bump mlstacks version in ZenML extra by @strickvl in https://github.com/zenml-io/zenml/pull/2091
• Fix SQL schema imports by @stefannica in https://github.com/zenml-io/zenml/pull/2098
• Fix migration for unowned stacks/components by @schustmi in https://github.com/zenml-io/zenml/pull/2099
• Polymorthic run_metadata by @avishniakov in https://github.com/zenml-io/zenml/pull/2064
• Update ruff formatter (for bugfixes) by @strickvl in https://github.com/zenml-io/zenml/pull/2106
• Lock in airflow version as higher versions will fail by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2108
• Swap contents for HTMLString and MarkdownString in docs by @christianversloot in https://github.com/zenml-io/zenml/pull/2110
• Fix secrets list with cloud secrets stores and RBAC by @stefannica in https://github.com/zenml-io/zenml/pull/2107
• More track events by @htahir1 in https://github.com/zenml-io/zenml/pull/2112
• Fix pipeline run cascade deletion by @fa9r in https://github.com/zenml-io/zenml/pull/2104
• Take integrations tests out of unit tests folder by @safoinme in https://github.com/zenml-io/zenml/pull/2100
• Allow extra values when dehydrating response models by @schustmi in https://github.com/zenml-io/zenml/pull/2114
• Request optimizations by @schustmi in https://github.com/zenml-io/zenml/pull/2103
• Pagination in model versions by @avishniakov in https://github.com/zenml-io/zenml/pull/2115
• Add StepContext.inputs property by @fa9r in https://github.com/zenml-io/zenml/pull/2105

Full Changelog: https://github.com/zenml-io/zenml/compare/0.50.0...0.51.0

- Python
Published by htahir1 about 2 years ago

https://github.com/zenml-io/zenml - 0.50.0

In this release, we introduce key updates aimed at improving user experience and security. The ModelConfig object has been renamed to ModelVersion for a more intuitive interface. Additionally, the release features enhancements such as optimized model hydration for better performance, alongside a range of bug fixes and contributions from both new and returning community members.

Breaking Change

• We have renamed the ModelConfig object to ModelVersion with other related changes to the model control plane, the goal of this is to bring a simplified user-interface experience, so once ModelVersion is configured in pipeline or step it will travel into all other user-facing places: step context, client, etc. by @avishniakov in #2044
• introducing RBAC for server endpoints, ensuring users have appropriate permissions for actions on resources. Additionally, it improves data handling by dehydrating response models to redact inaccessible information, while service accounts retain full permissions due to current database constraints. by @schustmi in #1999
• We have completely reworked our API models. While the request models are mostly the same, now with the new hydration logic, most of our response models feature a body and a metadata field which allow us to control the responses of our API and optimize the flow for anyone using it By @bcdurak #1971
• We also worked on adding a new "Artifacts" tab to our dashboard. With these new changes, it will become much easier to understand and adjust how ZenML versions your data. Moreover, by using "ExternalArtifacts", you will be able to save and load artifacts manually and use them in your pipelines. By @fa9r #1943

Enhancements

• Improve alembic migration safety by @fa9r in #2073
• Model Link Filtering by Artifact / Run Name by @fa9r in #2074

Bug Fixes

• Fix tag<>resource ID generator to fix the issue of manipulating migrated tags properly #2056
• Fixes for k3d deployments via mlstacks using the ZenML CLI wrapper #2059
• Fix some filter options for pipeline runs by @schustmi #2078
• Fix Label Studio image annotation example by @strickvl #2010
• Alembic migration fix for databases with scheduled pipelines with 2+ runs by @bcdurak #2072
• Model version endpoint fixes by @schustmi in #2060

ZenML Helm Chart Changes

• Make helm chart more robust to accidental secret deletions by @stefannica in #2053
• Separate helm hook resources from regular resources by @stefannica in #2055

Other Changes

• Connectors docs small fixes by @strickvl in https://github.com/zenml-io/zenml/pull/2050
• Feature/configurable service account for seldon predictor service by @Johnyz21 in https://github.com/zenml-io/zenml/pull/1725
• Adding NLP Template Example by @safoinme in https://github.com/zenml-io/zenml/pull/2051
• Fix CI by @fa9r in https://github.com/zenml-io/zenml/pull/2069
• Depaginate step runs to allow running pipelines with arbitrary step count by @schustmi in https://github.com/zenml-io/zenml/pull/2068
• Remove user name from orchestrator run name by @schustmi in https://github.com/zenml-io/zenml/pull/2067
• Artifacts Tab by @fa9r in https://github.com/zenml-io/zenml/pull/1943
• Add warnings/updates to Huggingface Spaces deployment docs by @strickvl in https://github.com/zenml-io/zenml/pull/2052
• Nightly builds by @strickvl in https://github.com/zenml-io/zenml/pull/2031
• Allow for custom disk size and type when using VertexAI Step Operator by @strickvl in https://github.com/zenml-io/zenml/pull/2054
• Set nightly builds to run at half-past the hour by @strickvl in https://github.com/zenml-io/zenml/pull/2077
• Set DCP template tag by @avishniakov in https://github.com/zenml-io/zenml/pull/2076
• Add missing dehydration in getserviceconnector endpoint by @schustmi in https://github.com/zenml-io/zenml/pull/2080
• Replace black with ruff format / bump mypy by @strickvl in https://github.com/zenml-io/zenml/pull/2082
• ModelVersion in pipeline context to pass in steps by @avishniakov in https://github.com/zenml-io/zenml/pull/2079
• Pin bcrypt by @strickvl in https://github.com/zenml-io/zenml/pull/2083

New Contributors

• @Johnyz21 made their first contribution in https://github.com/zenml-io/zenml/pull/1725

Full Changelog: https://github.com/zenml-io/zenml/compare/0.47.0...0.50.0

- Python
Published by safoinme about 2 years ago

https://github.com/zenml-io/zenml - 0.47.0

This release fixes a bug that was introduced in 0.46.1 where the default user was made inaccessible and was inadvertently duplicated. This release rescues the original user and renames the duplicate.

What's Changed

• Create tags table by @avishniakov in https://github.com/zenml-io/zenml/pull/2036
• Bring dashboard back to the release by @avishniakov in https://github.com/zenml-io/zenml/pull/2046
• Fix duplicate default user by @stefannica in https://github.com/zenml-io/zenml/pull/2045

Full Changelog: https://github.com/zenml-io/zenml/compare/0.46.1...0.47.0

- Python
Published by stefannica over 2 years ago

https://github.com/zenml-io/zenml - 0.44.4

This patch release backports some important fixes that have been introduced in more recent versions of ZenML to the 0.44.x release line.

What's Changed

• [backport/0.44] Backport important patches by @stefannica in https://github.com/zenml-io/zenml/pull/2042

Full Changelog: https://github.com/zenml-io/zenml/compare/0.44.3...0.44.4

- Python
Published by stefannica over 2 years ago

https://github.com/zenml-io/zenml - 0.46.1 [YANKED]

THIS RELEASE HAS BEEN YANKED

• Due to an unforeseen alembic migration issue and missing dashboard, this release has been yanked
• 0.47.0 is out to fix the bugs that have been uncovered

0.46.1

The 0.46.1 release introduces support for Service Accounts and API Keys that can be used to authenticate with the ZenML server from environments that do not support the web login flow, such as CI/CD environments, for example.

Also included in this release are some documentation updates and bug fixes, notably moving the database migration logic deployed with the Helm chart out of the init containers and into a Kubernetes Job, which makes it possible to scale out the ZenML server deployments without the risk of running into database migration conflicts.

What's Changed

• Small improvements to Hub docs page by @strickvl in https://github.com/zenml-io/zenml/pull/2015
• Pin OpenAI integration to <1.0.0 by @strickvl in https://github.com/zenml-io/zenml/pull/2027
• Make error message nicer for when two artifacts that share a prefix are found by @strickvl in https://github.com/zenml-io/zenml/pull/2023
• Move db-migration to job instead of init-container to allow replicas by @safoinme in https://github.com/zenml-io/zenml/pull/2021
• Fix stuck/broken CI by @strickvl in https://github.com/zenml-io/zenml/pull/2032
• Increase step.source_code Cut-Off Limit by @fa9r in https://github.com/zenml-io/zenml/pull/2025
• Improve artifact linkage logging in MCP by @avishniakov in https://github.com/zenml-io/zenml/pull/2016
• Upgrade feast so apidocs don't fail no mo by @AlexejPenner in https://github.com/zenml-io/zenml/pull/2028
• Remove NumPy Visualizations for 2D Arrays by @fa9r in https://github.com/zenml-io/zenml/pull/2033
• Fix user activation bug by @stefannica in https://github.com/zenml-io/zenml/pull/2037
• Remove create_new_model_version arg of ModelConfig by @avishniakov in https://github.com/zenml-io/zenml/pull/2030
• Extend the wait period in between PyPi package publication and Docker image building for releases by @strickvl in https://github.com/zenml-io/zenml/pull/2029
• Make zenml up prefill username when launching dashboard by @strickvl in https://github.com/zenml-io/zenml/pull/2024
• Add warning when artifact store cannot be loaded by @strickvl in https://github.com/zenml-io/zenml/pull/2011
• Add extra config to Kaniko docs by @safoinme in https://github.com/zenml-io/zenml/pull/2019
• ZenML API Keys and Service Accounts by @stefannica in https://github.com/zenml-io/zenml/pull/1840

Full Changelog: https://github.com/zenml-io/zenml/compare/0.46.0...0.46.1

- Python
Published by stefannica over 2 years ago

https://github.com/zenml-io/zenml - 0.46.0

This release brings some upgrades, documentation updates and bug fixes. Notably, our langchain integration now supports more modern versions and has been upgraded to a new version at the lower edge of supported packages on account of a security vulnerability.

Other fixes related to the Model Control Plane which was updated to support the deletion of model versions via the CLI, for example.

Breaking Change

We removed the llama_index integration in this release. This related to unsolvable dependency clashes that relate to sqlmodel and our database. We expect these clashes to be resolved in the future and then we will add our integration back in. If you were using the llama_index materializer that was part of the integration, you will have to use a custom materializer in the meanwhile. We apologize for the inconvenience.

What's Changed

• MCP-driven E2E template by @avishniakov in https://github.com/zenml-io/zenml/pull/2004
• Model scoped endpoints by @avishniakov in https://github.com/zenml-io/zenml/pull/2003
• Delete model version in cli by @avishniakov in https://github.com/zenml-io/zenml/pull/2006
• Add latest version to model list response by @avishniakov in https://github.com/zenml-io/zenml/pull/2007
• Fix gcs bucket docs error message by @safoinme in https://github.com/zenml-io/zenml/pull/2018
• Fix Skypilot docs configuration by @safoinme in https://github.com/zenml-io/zenml/pull/2017
• Bump langchain, disable llama_index, and fix Vector Store materializer by @strickvl in https://github.com/zenml-io/zenml/pull/2013
• Fix Build Options of GCPImageBuilder by @fa9r in https://github.com/zenml-io/zenml/pull/1992
• Fix the stack component describe CLI output by @stefannica in https://github.com/zenml-io/zenml/pull/2001

Full Changelog: https://github.com/zenml-io/zenml/compare/0.45.6...0.46.0

- Python
Published by strickvl over 2 years ago

https://github.com/zenml-io/zenml - 0.45.6

This release brings an array of enhancements and refinements. Notable improvements include allowing for disconnecting service connectors from stack components, adding connector support to the sagemaker step operator, turning synchronous mode on by default for all orchestrators, and enabling server-side component config validation.

What's Changed

• Updating README.md and update images by @znegrin in https://github.com/zenml-io/zenml/pull/1986
• Always set the active workspace to be the default workspace server side by @stefannica in https://github.com/zenml-io/zenml/pull/1989
• Update outdated CLI docs by @strickvl in https://github.com/zenml-io/zenml/pull/1990
• Turn synchronous mode on by default for all orchestrators by @stefannica in https://github.com/zenml-io/zenml/pull/1991
• Use docker credentials in the skypilot orchestrator by @stefannica in https://github.com/zenml-io/zenml/pull/1983
• Add missing space to @step warning message by @strickvl in https://github.com/zenml-io/zenml/pull/1994
• Fix sagemaker orchestrator and step operator env vars and other minor bugs by @stefannica in https://github.com/zenml-io/zenml/pull/1993
• fix: BasePyTorchMaterliazer -> Materializer by @cameronraysmith in https://github.com/zenml-io/zenml/pull/1969
• allow calling old base pytorch materilizzer by @safoinme in https://github.com/zenml-io/zenml/pull/1997
• Add connector support to sagemaker step operator. by @stefannica in https://github.com/zenml-io/zenml/pull/1996
• Server-Side Component Config Validation by @fa9r in https://github.com/zenml-io/zenml/pull/1988
• Allow disconnecting service-connector from stack component by @safoinme in https://github.com/zenml-io/zenml/pull/1864

New Contributors

• @znegrin made their first contribution in https://github.com/zenml-io/zenml/pull/1986

Full Changelog: https://github.com/zenml-io/zenml/compare/0.45.5...0.45.6

- Python
Published by stefannica over 2 years ago

https://github.com/zenml-io/zenml - 0.45.5

This minor release contains bugfixes and documentation improvements. Notably, our sqlmodel dependency has been pinned to 0.0.8 which fixes installation errors following the release of 0.0.9.

What's Changed

• Add a 'how do I...' section into docs by @strickvl in https://github.com/zenml-io/zenml/pull/1953
• Bump mypy, ruff and black by @strickvl in https://github.com/zenml-io/zenml/pull/1963
• Fix double slashes in weblogin by @schustmi in https://github.com/zenml-io/zenml/pull/1972
• SQLModel docs backport fixes by @strickvl in https://github.com/zenml-io/zenml/pull/1975
• Updated quickstart command in cloud quickstart by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1977
• Make sure vertex job id is only lower case letter, number or dash by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1978
• Fix DB initialization when using external authentication by @schustmi in https://github.com/zenml-io/zenml/pull/1965
• Pin SQLModel dependency to 0.0.8 by @strickvl in https://github.com/zenml-io/zenml/pull/1973
• Source code repository type not included in resolve source by @safoinme in https://github.com/zenml-io/zenml/pull/1982
• Prepare release 0.45.5 by @strickvl in https://github.com/zenml-io/zenml/pull/1981

Full Changelog: https://github.com/zenml-io/zenml/compare/0.45.4...0.45.5

- Python
Published by strickvl over 2 years ago

https://github.com/zenml-io/zenml - 0.45.4

This minor update fixes a database migration bug that you could potentially encounter while upgrading your ZenML version and relates to use of the ExternalArtifact object. If you are upgrading from <0.45.x version, this is the recommended release.

PROBLEMS?: If you upgraded to ZenML v0.45.2 or v0.45.3 and are experiencing issues with your database, please consider upgrading to v0.45.4 instead.

What's Changed

• Increase reuse of ModelConfig by @avishniakov in https://github.com/zenml-io/zenml/pull/1954
• resolve alembic branches by @avishniakov in https://github.com/zenml-io/zenml/pull/1964
• Fix corrupted migration for old dbs by @avishniakov in https://github.com/zenml-io/zenml/pull/1966

Full Changelog: https://github.com/zenml-io/zenml/compare/0.45.3...0.45.4

- Python
Published by avishniakov over 2 years ago

https://github.com/zenml-io/zenml - 0.45.3

This minor update fixes a database migration bug that you could potentially encounter while upgrading your ZenML version and relates to use of the ExternalArtifact object.

PROBLEMS?: If you upgraded to ZenML v0.45.2 and are experiencing issues with your database, please either reach out to us on Slack directly or feel free to use this migration script that will manually fix the issue. (Please do backup your database before using the migration script so as to prevent any data loss!)

This release also includes a bugfix from @cameronraysmith relating to the resolution of our Helm chart OCI location. Thank you!

What's Changed

• fix: match chart name in docs to publish workflow by @cameronraysmith in https://github.com/zenml-io/zenml/pull/1942
• Evaluate YAML based config early + OSS-2511 by @avishniakov in https://github.com/zenml-io/zenml/pull/1876
• Fixing nullable parameter to avoid extra migrations by @bcdurak in https://github.com/zenml-io/zenml/pull/1955
• Pin Helm version to avoid 400 Bad Request error by @wjayesh in https://github.com/zenml-io/zenml/pull/1958
• external_input_artifact backward compatibility with alembic by @avishniakov in https://github.com/zenml-io/zenml/pull/1957
• Fix nlp template link by @htahir1 in https://github.com/zenml-io/zenml/pull/1960
• handle NULLs in migration by @avishniakov in https://github.com/zenml-io/zenml/pull/1961
• Prepare release 0.45.3 by @strickvl in https://github.com/zenml-io/zenml/pull/1959

New Contributors

• @cameronraysmith made their first contribution in https://github.com/zenml-io/zenml/pull/1942

Full Changelog: https://github.com/zenml-io/zenml/compare/0.45.2...0.45.3

- Python
Published by strickvl over 2 years ago

https://github.com/zenml-io/zenml - 0.45.2

This release replaces 0.45.0 and 0.45.1, and fixes the major migration bugs that were in that yanked release. Please upgrade directly to 0.45.2 and avoid upgrading to 0.45.0 to avoid unexpected migration issues.

Note that 0.45.0 and 0.45.1 were removed from PyPI due to an issue with the alembic versions + migration which could affect the database state. This release fixes that issue.

If you have already upgraded to 0.45.0 please let us know in Slack and we'll happy to assist in rollback and recovery.

This release introduces a major upgrade to ZenML, featuring a new authentication mechanism, performance improvements, the introduction of the model control plane, and internal enhancements.

New Authentication Mechanism (#4303)

Our improved authentication mechanism offers a more secure way of connecting to the ZenML server. It initiates a device flow that prompts you to log in via the browser dashboard:

zenml connect --url <YOUR_SERVER_URL>

This eliminates the need for explicit credential input. The previous method (zenml connect --url <URL> --username <USERNAME> --password <PASSWORD>) remains operational but is less recommended due to security concerns.

Critical This change disrupts existing pipeline schedules. After upgrading, manually cancel and reschedule pipelines using the updated version of ZenML.

For more information, read about the device flow in our documentation.

Performance enhancements (#3207)

Internal API adjustments have reduced the footprint of ZenML API objects by up to 35%. This will particularly benefit users with large step and pipeline configurations. Further reductions will be implemented in our next release.

Model Control Plane debut (#5648)

ZenML now includes a preliminary version of the model control plane, a feature for registering models and their metadata on a single ZenML dashboard view. Future releases will provide more details. To test this early version, follow this example.

Breaking Changes

• Environment variables ZENML_AUTH_TYPE and ZENML_JWT_SECRET_KEY have been renamed to ZENML_SERVER_AUTH_SCHEME and ZENML_SERVER_JWT_SECRET_KEY, respectively.
• All ZenML server-issued JWT tokens now include an issuer and an audience. After the server update, current scheduled pipelines become invalidated. Reset your schedules and reconnect all clients to the server to obtain new tokens.
• UnmaterializedArtifact has been relocated to zenml.artifacts. Change your import statement from from zenml.materializers import UnmaterializedArtifact to from zenml.artifacts.unmaterialized_artifact import UnmaterializedArtifact.

Deprecations

• zenml.steps.external_artifact.ExternalArtifact has moved to zenml.artifacts.external_artifact.ExternalArtifact.

And the rest:

• Discord alerter integration by @bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1818. Huge shoutout to you priyadutt - we're sending some swag your way!
• Update Neptune dependency: neptune-client > neptune by @fa9r in https://github.com/zenml-io/zenml/pull/1837
• Disable codeql on pushes to develop by @strickvl in https://github.com/zenml-io/zenml/pull/1842
• Template not updating due to git diff misuse by @avishniakov in https://github.com/zenml-io/zenml/pull/1844
• Bump feast version to fix api docs generation by @fa9r in https://github.com/zenml-io/zenml/pull/1845
• CI Fixes / Improvements by @fa9r in https://github.com/zenml-io/zenml/pull/1848
• Fix MLflow registry methods with empty metadata by @fa9r in https://github.com/zenml-io/zenml/pull/1843
• Use configured template REF in CI by @avishniakov in https://github.com/zenml-io/zenml/pull/1851
• Fix template REF in CI by @avishniakov in https://github.com/zenml-io/zenml/pull/1852
• Fix AWS service connector installation requirements by @stefannica in https://github.com/zenml-io/zenml/pull/1850
• [Docs] Improvements to custom flavor and custom orchestrator pages by @htahir1 in https://github.com/zenml-io/zenml/pull/1747
• Optimizing the performance through database changes by @bcdurak in https://github.com/zenml-io/zenml/pull/1835
• Add README for examples folder by @strickvl in https://github.com/zenml-io/zenml/pull/1860
• Free up disk space in CI by @strickvl in https://github.com/zenml-io/zenml/pull/1863
• Make Terraform Optional Again by @fa9r in https://github.com/zenml-io/zenml/pull/1855
• Model watchtower becomes Model control plane by @strickvl in https://github.com/zenml-io/zenml/pull/1868
• Update documentation by @VishalKumar-S in https://github.com/zenml-io/zenml/pull/1872
• Fix CI by freeing up space on runner by @strickvl in https://github.com/zenml-io/zenml/pull/1866
• Allow for user param to be specified (successfully) in DockerSettings by @strickvl in https://github.com/zenml-io/zenml/pull/1857
• Add get_pipeline_context by @avishniakov in https://github.com/zenml-io/zenml/pull/1870
• [Helm] Use GCP creds directly instead of a file. by @wjayesh in https://github.com/zenml-io/zenml/pull/1874
• External authenticator support, authorized devices and web login by @stefannica in https://github.com/zenml-io/zenml/pull/1814
• Connect to Service-connector at component registration by @safoinme in https://github.com/zenml-io/zenml/pull/1858
• Fixing the upgrade migration script after the database changes by @bcdurak in https://github.com/zenml-io/zenml/pull/1877
• [Model Control Plane] v0.1 mega-branch by @avishniakov in https://github.com/zenml-io/zenml/pull/1816
• Update to templates by @htahir1 in https://github.com/zenml-io/zenml/pull/1878
• Docs for orgs, rbac and sso by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1875
• Convert network_config dict to NetworkConfig object in SageMaker orchestrator by @christianversloot in https://github.com/zenml-io/zenml/pull/1873
• Add missing Docker build options for GCP image builder by @strickvl in https://github.com/zenml-io/zenml/pull/1856
• Solve alembic branching issue by @avishniakov in https://github.com/zenml-io/zenml/pull/1879
• Fix typo for 0.45 release by @strickvl in https://github.com/zenml-io/zenml/pull/1881
• Only import ipinfo when necessary by @schustmi in https://github.com/zenml-io/zenml/pull/1888
• [Model Control Plane] Suppress excessive logging in model control plane by @avishniakov in https://github.com/zenml-io/zenml/pull/1885
• Add warning generation scripts for Gitbook docs by @strickvl in https://github.com/zenml-io/zenml/pull/1929
• Fix calling click decorator in model CLI command by @safoinme in https://github.com/zenml-io/zenml/pull/1932
• Lightweight template CI by @avishniakov in https://github.com/zenml-io/zenml/pull/1930
• Update Skypilot orchestrator setting docs section by @safoinme in https://github.com/zenml-io/zenml/pull/1931

New Contributors

• @VishalKumar-S made their first contribution in https://github.com/zenml-io/zenml/pull/1872

Full Changelog: https://github.com/zenml-io/zenml/compare/0.44.3...0.45.0

- Python
Published by strickvl over 2 years ago

https://github.com/zenml-io/zenml - 0.45.1

0.45.1

Update: It's one of those days! This release is also yanked due to alembic migration issues. We are working hard to solve it. In the meanwhile please use 0.44.3

This release replaces 0.45.0, and fixes the major migration bugs that were in that yanked release. Please upgrade directly to 0.45.1 and avoid upgrading to 0.45.0 to avoid unexpected migration issues.

What's changed

This release introduces a major upgrade to ZenML, featuring a new authentication mechanism, performance improvements, the introduction of the model control plane, and internal enhancements.

New Authentication Mechanism (#4303)

Our improved authentication mechanism offers a more secure way of connecting to the ZenML server. It initiates a device flow that prompts you to log in via the browser dashboard:

zenml connect --url <YOUR_SERVER_URL>

This eliminates the need for explicit credential input. The previous method (zenml connect --url <URL> --username <USERNAME> --password <PASSWORD>) remains operational but is less recommended due to security concerns.

Critical This change disrupts existing pipeline schedules. After upgrading, manually cancel and reschedule pipelines using the updated version of ZenML.

For more information, read about the device flow in our documentation.

Performance enhancements (#3207)

Internal API adjustments have reduced the footprint of ZenML API objects by up to 35%. This will particularly benefit users with large step and pipeline configurations. Further reductions will be implemented in our next release.

Model Control Plane debut (#5648)

ZenML now includes a preliminary version of the model control plane, a feature for registering models and their metadata on a single ZenML dashboard view. Future releases will provide more details. To test this early version, follow this example.

Breaking Changes

• Environment variables ZENML_AUTH_TYPE and ZENML_JWT_SECRET_KEY have been renamed to ZENML_SERVER_AUTH_SCHEME and ZENML_SERVER_JWT_SECRET_KEY, respectively.
• All ZenML server-issued JWT tokens now include an issuer and an audience. After the server update, current scheduled pipelines become invalidated. Reset your schedules and reconnect all clients to the server to obtain new tokens.
• UnmaterializedArtifact has been relocated to zenml.artifacts. Change your import statement from from zenml.materializers import UnmaterializedArtifact to from zenml.artifacts.unmaterialized_artifact import UnmaterializedArtifact.

Deprecations

• zenml.steps.external_artifact.ExternalArtifact has moved to zenml.artifacts.external_artifact.ExternalArtifact.

And the rest:

• Discord alerter integration by @bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1818. Huge shoutout to you priyadutt - we're sending some swag your way!
• Update Neptune dependency: neptune-client > neptune by @fa9r in https://github.com/zenml-io/zenml/pull/1837
• Disable codeql on pushes to develop by @strickvl in https://github.com/zenml-io/zenml/pull/1842
• Template not updating due to git diff misuse by @avishniakov in https://github.com/zenml-io/zenml/pull/1844
• Bump feast version to fix api docs generation by @fa9r in https://github.com/zenml-io/zenml/pull/1845
• CI Fixes / Improvements by @fa9r in https://github.com/zenml-io/zenml/pull/1848
• Fix MLflow registry methods with empty metadata by @fa9r in https://github.com/zenml-io/zenml/pull/1843
• Use configured template REF in CI by @avishniakov in https://github.com/zenml-io/zenml/pull/1851
• Fix template REF in CI by @avishniakov in https://github.com/zenml-io/zenml/pull/1852
• Fix AWS service connector installation requirements by @stefannica in https://github.com/zenml-io/zenml/pull/1850
• [Docs] Improvements to custom flavor and custom orchestrator pages by @htahir1 in https://github.com/zenml-io/zenml/pull/1747
• Optimizing the performance through database changes by @bcdurak in https://github.com/zenml-io/zenml/pull/1835
• Add README for examples folder by @strickvl in https://github.com/zenml-io/zenml/pull/1860
• Free up disk space in CI by @strickvl in https://github.com/zenml-io/zenml/pull/1863
• Make Terraform Optional Again by @fa9r in https://github.com/zenml-io/zenml/pull/1855
• Model watchtower becomes Model control plane by @strickvl in https://github.com/zenml-io/zenml/pull/1868
• Update documentation by @VishalKumar-S in https://github.com/zenml-io/zenml/pull/1872
• Fix CI by freeing up space on runner by @strickvl in https://github.com/zenml-io/zenml/pull/1866
• Allow for user param to be specified (successfully) in DockerSettings by @strickvl in https://github.com/zenml-io/zenml/pull/1857
• Add get_pipeline_context by @avishniakov in https://github.com/zenml-io/zenml/pull/1870
• [Helm] Use GCP creds directly instead of a file. by @wjayesh in https://github.com/zenml-io/zenml/pull/1874
• External authenticator support, authorized devices and web login by @stefannica in https://github.com/zenml-io/zenml/pull/1814
• Connect to Service-connector at component registration by @safoinme in https://github.com/zenml-io/zenml/pull/1858
• Fixing the upgrade migration script after the database changes by @bcdurak in https://github.com/zenml-io/zenml/pull/1877
• [Model Control Plane] v0.1 mega-branch by @avishniakov in https://github.com/zenml-io/zenml/pull/1816
• Update to templates by @htahir1 in https://github.com/zenml-io/zenml/pull/1878
• Docs for orgs, rbac and sso by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1875
• Convert network_config dict to NetworkConfig object in SageMaker orchestrator by @christianversloot in https://github.com/zenml-io/zenml/pull/1873
• Add missing Docker build options for GCP image builder by @strickvl in https://github.com/zenml-io/zenml/pull/1856
• Solve alembic branching issue by @avishniakov in https://github.com/zenml-io/zenml/pull/1879
• Fix typo for 0.45 release by @strickvl in https://github.com/zenml-io/zenml/pull/1881
• Only import ipinfo when necessary by @schustmi in https://github.com/zenml-io/zenml/pull/1888
• [Model Control Plane] Suppress excessive logging in model control plane by @avishniakov in https://github.com/zenml-io/zenml/pull/1885
• Add warning generation scripts for Gitbook docs by @strickvl in https://github.com/zenml-io/zenml/pull/1929
• Fix calling click decorator in model CLI command by @safoinme in https://github.com/zenml-io/zenml/pull/1932
• Lightweight template CI by @avishniakov in https://github.com/zenml-io/zenml/pull/1930
• Update Skypilot orchestrator setting docs section by @safoinme in https://github.com/zenml-io/zenml/pull/1931
• Add missing space when creating new model version by @strickvl in https://github.com/zenml-io/zenml/pull/1935
• Fix alembic inconsistencies by @fa9r in https://github.com/zenml-io/zenml/pull/1944
• [Model Control Plane] link cached artifacts by @avishniakov in https://github.com/zenml-io/zenml/pull/1946
• Fixing the migration issues by @bcdurak in https://github.com/zenml-io/zenml/pull/1945
• Release prepped by @htahir1 in https://github.com/zenml-io/zenml/pull/1948

New Contributors

• @VishalKumar-S made their first contribution in https://github.com/zenml-io/zenml/pull/1872

Full Changelog: https://github.com/zenml-io/zenml/compare/0.44.3...0.45.0

- Python
Published by htahir1 over 2 years ago

https://github.com/zenml-io/zenml - 0.45.0

What change's

This release introduces a major upgrade to ZenML, featuring a new authentication mechanism, performance improvements, the introduction of the model control plane, and internal enhancements.

New Authentication Mechanism (#4303)

Our improved authentication mechanism offers a more secure way of connecting to the ZenML server. It initiates a device flow that prompts you to log in via the browser dashboard:

zenml connect --url <YOUR_SERVER_URL>

This eliminates the need for explicit credential input. The previous method (zenml connect --url <URL> --username <USERNAME> --password <PASSWORD>) remains operational but is less recommended due to security concerns.

Critical This change disrupts existing pipeline schedules. After upgrading, manually cancel and reschedule pipelines using the updated version of ZenML.

For more information, read about the device flow in our documentation.

Performance enhancements (#3207)

Internal API adjustments have reduced the footprint of ZenML API objects by up to 35%. This will particularly benefit users with large step and pipeline configurations. Further reductions will be implemented in our next release.

Model Control Plane debut (#5648)

ZenML now includes a preliminary version of the model control plane, a feature for registering models and their metadata on a single ZenML dashboard view. Future releases will provide more details. To test this early version, follow this example.

Breaking Changes

• Environment variables ZENML_AUTH_TYPE and ZENML_JWT_SECRET_KEY have been renamed to ZENML_SERVER_AUTH_SCHEME and ZENML_SERVER_JWT_SECRET_KEY, respectively.
• All ZenML server-issued JWT tokens now include an issuer and an audience. After the server update, current scheduled pipelines become invalidated. Reset your schedules and reconnect all clients to the server to obtain new tokens.
• UnmaterializedArtifact has been relocated to zenml.artifacts. Change your import statement from from zenml.materializers import UnmaterializedArtifact to from zenml.artifacts.unmaterialized_artifact import UnmaterializedArtifact.

Deprecations

• zenml.steps.external_artifact.ExternalArtifact has moved to zenml.artifacts.external_artifact.ExternalArtifact.

And the rest:

• Discord alerter integration by @bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1818. Huge shoutout to you priyadutt - we're sending some swag your way!
• Update Neptune dependency: neptune-client > neptune by @fa9r in https://github.com/zenml-io/zenml/pull/1837
• Disable codeql on pushes to develop by @strickvl in https://github.com/zenml-io/zenml/pull/1842
• Template not updating due to git diff misuse by @avishniakov in https://github.com/zenml-io/zenml/pull/1844
• Bump feast version to fix api docs generation by @fa9r in https://github.com/zenml-io/zenml/pull/1845
• CI Fixes / Improvements by @fa9r in https://github.com/zenml-io/zenml/pull/1848
• Fix MLflow registry methods with empty metadata by @fa9r in https://github.com/zenml-io/zenml/pull/1843
• Use configured template REF in CI by @avishniakov in https://github.com/zenml-io/zenml/pull/1851
• Fix template REF in CI by @avishniakov in https://github.com/zenml-io/zenml/pull/1852
• Fix AWS service connector installation requirements by @stefannica in https://github.com/zenml-io/zenml/pull/1850
• [Docs] Improvements to custom flavor and custom orchestrator pages by @htahir1 in https://github.com/zenml-io/zenml/pull/1747
• Optimizing the performance through database changes by @bcdurak in https://github.com/zenml-io/zenml/pull/1835
• Add README for examples folder by @strickvl in https://github.com/zenml-io/zenml/pull/1860
• Free up disk space in CI by @strickvl in https://github.com/zenml-io/zenml/pull/1863
• Make Terraform Optional Again by @fa9r in https://github.com/zenml-io/zenml/pull/1855
• Model watchtower becomes Model control plane by @strickvl in https://github.com/zenml-io/zenml/pull/1868
• Update documentation by @VishalKumar-S in https://github.com/zenml-io/zenml/pull/1872
• Fix CI by freeing up space on runner by @strickvl in https://github.com/zenml-io/zenml/pull/1866
• Allow for user param to be specified (successfully) in DockerSettings by @strickvl in https://github.com/zenml-io/zenml/pull/1857
• Add get_pipeline_context by @avishniakov in https://github.com/zenml-io/zenml/pull/1870
• [Helm] Use GCP creds directly instead of a file. by @wjayesh in https://github.com/zenml-io/zenml/pull/1874
• External authenticator support, authorized devices and web login by @stefannica in https://github.com/zenml-io/zenml/pull/1814
• Connect to Service-connector at component registration by @safoinme in https://github.com/zenml-io/zenml/pull/1858
• Fixing the upgrade migration script after the database changes by @bcdurak in https://github.com/zenml-io/zenml/pull/1877
• [Model Control Plane] v0.1 mega-branch by @avishniakov in https://github.com/zenml-io/zenml/pull/1816
• Update to templates by @htahir1 in https://github.com/zenml-io/zenml/pull/1878
• Docs for orgs, rbac and sso by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1875
• Convert network_config dict to NetworkConfig object in SageMaker orchestrator by @christianversloot in https://github.com/zenml-io/zenml/pull/1873
• Add missing Docker build options for GCP image builder by @strickvl in https://github.com/zenml-io/zenml/pull/1856
• Solve alembic branching issue by @avishniakov in https://github.com/zenml-io/zenml/pull/1879
• Fix typo for 0.45 release by @strickvl in https://github.com/zenml-io/zenml/pull/1881
• Only import ipinfo when necessary by @schustmi in https://github.com/zenml-io/zenml/pull/1888
• [Model Control Plane] Suppress excessive logging in model control plane by @avishniakov in https://github.com/zenml-io/zenml/pull/1885
• Add warning generation scripts for Gitbook docs by @strickvl in https://github.com/zenml-io/zenml/pull/1929
• Fix calling click decorator in model CLI command by @safoinme in https://github.com/zenml-io/zenml/pull/1932
• Lightweight template CI by @avishniakov in https://github.com/zenml-io/zenml/pull/1930
• Update Skypilot orchestrator setting docs section by @safoinme in https://github.com/zenml-io/zenml/pull/1931

New Contributors

• @VishalKumar-S made their first contribution in https://github.com/zenml-io/zenml/pull/1872

Full Changelog: https://github.com/zenml-io/zenml/compare/0.44.3...0.45.0

- Python
Published by safoinme over 2 years ago

https://github.com/zenml-io/zenml -

This release, introduces SkyPilot, a new VM orchestrator for ZenML that lets users run pipelines on their choice of cloud provider VMs, offering a GPU option without Kubernetes or serverless orchestrators. This release also brings bug fixes and improvements, including a streamlined 'connect' command, interactive configuration for 'zenml stack deploy,' and enhanced documentation covering SageMaker, GCP, and service connectors with MFA.

New Orchestrator: SkyPilot (#1765)

This release introduces a new orchestrator called SkyPilot. SkyPilot is a VM orchestrator that can be used to run ZenML pipelines on a VM of choice in one of the three supported cloud providers. It is an excellent choice for users who want to run ZenML pipelines on a GPU instance, but don't want to use Kubernetes or serverless orchestrators like SageMaker.

Fixes and Improvements

This release fixes several bugs and improves the user experience of the CLI and the documentation. The most notable changes are: * The new connect command that allows connecting all stack components within a stack to a service connector with a single command. * Adding an interactive flow to the zenml stack deploy command that allows users to configure their stack in a guided manner. * Add documentation on how to debug the SageMaker orchestrator, how to get started with a quick cloud stack on GCP, and documentation on using service connectors with enabled MFA.

What's Changed

• Add support for empty API token in Kubernetes service connector. by @stefannica in https://github.com/zenml-io/zenml/pull/1808
• Use the container registry credentials to build images with the local image builder by @stefannica in https://github.com/zenml-io/zenml/pull/1804
• Fix CI by @fa9r in https://github.com/zenml-io/zenml/pull/1809
• Add documentation on how to debug the SageMaker orchestrator by @fa9r in https://github.com/zenml-io/zenml/pull/1810
• Bump rich and uvicorn by @jlopezpena in https://github.com/zenml-io/zenml/pull/1750
• SageMaker: Enable configuring authentication credentials explicitly by @fa9r in https://github.com/zenml-io/zenml/pull/1805
• Fix: ZenML DB migrations don't run if zenml is installed in path with spaces by @stefannica in https://github.com/zenml-io/zenml/pull/1815
• Fix mlflow 'run_name' variable overwriting by @iraadit in https://github.com/zenml-io/zenml/pull/1821
• Add SECURITY.md file for vulnerability disclosures. by @strickvl in https://github.com/zenml-io/zenml/pull/1824
• Add MFA limitation to service-connectors docs by @safoinme in https://github.com/zenml-io/zenml/pull/1827
• Improve zenml stack describe to show mlstacks outputs by @strickvl in https://github.com/zenml-io/zenml/pull/1826
• Documentation to get started with a quick cloud stack on GCP by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1807
• Fix missing text in git repo docs by @strickvl in https://github.com/zenml-io/zenml/pull/1831
• Handle irregular plural of code_repository for error message by @strickvl in https://github.com/zenml-io/zenml/pull/1832
• Connect stack to a service account by @safoinme in https://github.com/zenml-io/zenml/pull/1828
• SkyPilot Integration with VM Orchestrators by @htahir1 in https://github.com/zenml-io/zenml/pull/1765
• Add interactive CLI flow for zenml stack deploy by @strickvl in https://github.com/zenml-io/zenml/pull/1829
• Add README file for helm chart by @strickvl in https://github.com/zenml-io/zenml/pull/1830
• Fix slack environment variable in in generative_chat example README by @bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1836

New Contributors

• @iraadit made their first contribution in https://github.com/zenml-io/zenml/pull/1821

Full Changelog: https://github.com/zenml-io/zenml/compare/0.44.2...tes

- Python
Published by safoinme over 2 years ago

https://github.com/zenml-io/zenml - 0.44.2

This release contains updates for some of the most popular integrations, as well as several bug fixes and documentation improvements.

Minor Default Behavior Changes

• The default page size for zenml list commands was reduced to 20 (from 50) to speed up the runtime of such commands.
• Simultaneous connection to local and remote ZenML servers is no longer possible since this caused several unexpected behaviors in the past.

Integration Updates

• The mlflow integration now supports the newest MLflow version 2.6.0.
• The evidently integration now supports the latest Evidently version 0.4.4.
• The SageMaker orchestrator of the aws integration now supports authentication via service connectors.

What's Changed

• Add bandit to CI for security linting by @strickvl in https://github.com/zenml-io/zenml/pull/1775
• Add mlstacks compatibility check to CI by @strickvl in https://github.com/zenml-io/zenml/pull/1767
• extend StepContext visibility to materializers by @avishniakov in https://github.com/zenml-io/zenml/pull/1769
• Revert GH changes to fix colima bug in macos gh by @safoinme in https://github.com/zenml-io/zenml/pull/1779
• Reduce CI runner count by @strickvl in https://github.com/zenml-io/zenml/pull/1777
• Add E2E template as example by @avishniakov in https://github.com/zenml-io/zenml/pull/1766
• Fix CI step names by @avishniakov in https://github.com/zenml-io/zenml/pull/1784
• Add vulnerability scanner by @strickvl in https://github.com/zenml-io/zenml/pull/1776
• Stop CI from running on push to develop by @strickvl in https://github.com/zenml-io/zenml/pull/1788
• Skip update templates outside PR by @avishniakov in https://github.com/zenml-io/zenml/pull/1786
• Fix azure service connector docs by @stefannica in https://github.com/zenml-io/zenml/pull/1778
• fix: use k8s V1CronJob instead of V1beta1CronJob (#1781) by @francoisserra in https://github.com/zenml-io/zenml/pull/1787
• Page limit adjustment by @bcdurak in https://github.com/zenml-io/zenml/pull/1791
• Prevent simultaneous connection to local and remote servers by @fa9r in https://github.com/zenml-io/zenml/pull/1792
• Update MLflow version to allow support for 2.6.0 by @safoinme in https://github.com/zenml-io/zenml/pull/1782
• Improve ConnectionError error message by @fa9r in https://github.com/zenml-io/zenml/pull/1783
• Stop old MLflow services when deploying new ones by @fa9r in https://github.com/zenml-io/zenml/pull/1793
• Prevent adding private components into shared stacks by @fa9r in https://github.com/zenml-io/zenml/pull/1794
• Publish server helm chart as part of CI by @wjayesh in https://github.com/zenml-io/zenml/pull/1740
• Docs on the use of ZenML-specific environment variables by @strickvl in https://github.com/zenml-io/zenml/pull/1796
• Add support for newer Evidently versions by @fa9r in https://github.com/zenml-io/zenml/pull/1780
• Link E2E example to docs by @avishniakov in https://github.com/zenml-io/zenml/pull/1790
• Copy step instance before applying configuration by @schustmi in https://github.com/zenml-io/zenml/pull/1798
• Fix AWS container registry image pushing with service connectors by @fa9r in https://github.com/zenml-io/zenml/pull/1797
• Make Sagemaker orchestrator work with connectors by @fa9r in https://github.com/zenml-io/zenml/pull/1799
• Add rebase Pre-requisite to PRs template by @safoinme in https://github.com/zenml-io/zenml/pull/1801

Full Changelog: https://github.com/zenml-io/zenml/compare/0.44.1...0.44.2

- Python
Published by fa9r over 2 years ago

https://github.com/zenml-io/zenml - 0.44.1

This release brings various improvements over the previous version, mainly focusing on using the newly refactored mlstacks package, ZenML's logging module and the changes in our analytics.

IMPORTANT NOTICE

Note: 0.44.0 was removed from Pypi due to an issue with the alembic versions which could affect the database state. A branch occurred in the versions: 0.42.1 -> [0.43.0, e1d66d91a099] -> 0.44.0. This release fixes the issue.
The primary issue arises when deploying version 0.44.0 using a MySQL backend. Although the alembic migration executes all tasks up to 0.44.0, the alembic version represented in the database remains at 0.43.0. This issue persists irrespective of the measures taken, including trying various versions after 0.43.0.
This imbalance leads to failure when running a second replica migration because the database's state is at 0.44.0 while the alembic version remains at 0.43.0. Similarly, attempts to run a second replica or restart the pod fail as the alembic tries to migrate from 0.43.0 to 0.44.0, which is not possible because these changes already exist in the database.
Please note: If you encounter this problem, we recommend you roll back to previous versions and upgrade to 0.43.0. If you still experience difficulties, please join our Slack community at https://zenml.io/slack. We're ready to help you work through this issue.

What's Changed

• Remove e2e example and point to templates by @avishniakov in https://github.com/zenml-io/zenml/pull/1752
• Add cloud architecture docs by @htahir1 in https://github.com/zenml-io/zenml/pull/1751
• Update docs/docstrings following mlstacks repo name change by @strickvl in https://github.com/zenml-io/zenml/pull/1754
• Update Cloud deployment scenarios by @stefannica in https://github.com/zenml-io/zenml/pull/1757
• Fixing the logging message regarding caching by @bcdurak in https://github.com/zenml-io/zenml/pull/1748
• Improvements to the step logs storage functionality by @bcdurak in https://github.com/zenml-io/zenml/pull/1733
• Fix qemu/colima Github Actions bug by @safoinme in https://github.com/zenml-io/zenml/pull/1760
• Bump ruff and mypy by @strickvl in https://github.com/zenml-io/zenml/pull/1762
• Add Template Testing in Core by @avishniakov in https://github.com/zenml-io/zenml/pull/1745
• Removing analytics v1 and optimizing v2 by @bcdurak in https://github.com/zenml-io/zenml/pull/1753
• Update publish script to take a token by @strickvl in https://github.com/zenml-io/zenml/pull/1758
• Update variable name for release publication token by @strickvl in https://github.com/zenml-io/zenml/pull/1764
• Lock the MYSQL Database during DB migrations by @safoinme in https://github.com/zenml-io/zenml/pull/1763
• mlstacks integration (and deprecation of old deployment logic) by @strickvl in https://github.com/zenml-io/zenml/pull/1721
• Upgrade typing extensions within api docs build workflow by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1741
• Fix branching alembic history by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1772
• Remove pinned zenml version specified in TOC for SDK docs by @strickvl in https://github.com/zenml-io/zenml/pull/1770
• Modified the track metadata for the opt-in event by @bcdurak in https://github.com/zenml-io/zenml/pull/1774
• Check alembic branch divergence in CI by @strickvl in https://github.com/zenml-io/zenml/pull/1773
• Remove the DB lock by @safoinme in https://github.com/zenml-io/zenml/pull/1771

Full Changelog: https://github.com/zenml-io/zenml/compare/0.43.0...0.44.1

- Python
Published by safoinme over 2 years ago

https://github.com/zenml-io/zenml - 0.44.0 (YANKED [SEE RELEASE NOTES])

IMPORTANT NOTICE

This release has been removed from pypi due to an issue with the alembic versions which could affect the database state. A branch occurred in the versions: 0.42.1 -> [0.43.0, e1d66d91a099] -> 0.44.0.

The primary issue arises when deploying version 0.44.0 using a MySQL backend. Although the alembic migration executes all tasks up to 0.44.0, the alembic version represented in the database remains at 0.43.0. This issue persists irrespective of the measures taken, including trying various versions after 0.43.0.

This imbalance leads to failure when running a second replica migration because the database's state is at 0.44.0 while the alembic version remains at 0.43.0. Similarly, attempts to run a second replica or restart the pod fail as the alembic tries to migrate from 0.43.0 to 0.44.0, which is not possible because these changes already exist in the database.

Please note: If you encounter this problem, we recommend that you rollback to previous versions and then upgrade to 0.43.0. If you still experience difficulties, please join our Slack community at https://zenml.io/slack. We're ready to help you work through this issue.

---

Older Notes

This release brings various improvements over the previous version, mainly focusing on the usage of newly refactored mlstacks package, ZenML's logging module and the changes in our analytics.

What's Changed

• Remove e2e example and point to templates by @avishniakov in https://github.com/zenml-io/zenml/pull/1752
• Add cloud architecture docs by @htahir1 in https://github.com/zenml-io/zenml/pull/1751
• Update docs/docstrings following mlstacks repo name change by @strickvl in https://github.com/zenml-io/zenml/pull/1754
• Update Cloud deployment scenarios by @stefannica in https://github.com/zenml-io/zenml/pull/1757
• Fixing the logging message regarding caching by @bcdurak in https://github.com/zenml-io/zenml/pull/1748
• Improvements to the step logs storage functionality by @bcdurak in https://github.com/zenml-io/zenml/pull/1733
• Fix qemu/colima Github Actions bug by @safoinme in https://github.com/zenml-io/zenml/pull/1760
• Bump ruff and mypy by @strickvl in https://github.com/zenml-io/zenml/pull/1762
• Add Template Testing in Core by @avishniakov in https://github.com/zenml-io/zenml/pull/1745
• Removing analytics v1 and optimizing v2 by @bcdurak in https://github.com/zenml-io/zenml/pull/1753
• Update publish script to take a token by @strickvl in https://github.com/zenml-io/zenml/pull/1758
• Update variable name for release publication token by @strickvl in https://github.com/zenml-io/zenml/pull/1764
• Lock MYSQL Database during DB migrations by @safoinme in https://github.com/zenml-io/zenml/pull/1763
• mlstacks integration (and deprecation of old deployment logic) by @strickvl in https://github.com/zenml-io/zenml/pull/1721

Full Changelog: https://github.com/zenml-io/zenml/compare/0.43.0...0.44.0

- Python
Published by bcdurak over 2 years ago

https://github.com/zenml-io/zenml - 0.43.0

This release brings limited support for Python 3.11, improves quickstart experience with the fully reworked flow, enhances the user experience while dealing with ZenML docs, offers new extended templates for projects and fixes GCP connector creation issue.

Limited support for Python 3.11

This release adds limited support for Python 3.11. The following integrations are currently not tested and may behave unexpectedly with Python 3.11: - gcp - kubeflow - tekton

This is because: - GCP packages that support Python 3.11 are not compatible with KFP 1 - Upgrade to KFP 2 is blocked by the fact that Tekton doesn't have any release compatible with KFP 2 yet (https://github.com/zenml-io/zenml/pull/1697)

Breaking Changes

A minor breaking change in CLI for zenml init: - previously supported flag --starter - new flag --template-with-defaults - behavior remains the same - flag is responsible for usage of default settings in the template

What's Changed

• Disable implicit auth methods for service connectors by default by @stefannica in https://github.com/zenml-io/zenml/pull/1704
• New quickstart by @strickvl in https://github.com/zenml-io/zenml/pull/1692
• Set MLflow configuration as environment variables before deployment subprocess by @safoinme in https://github.com/zenml-io/zenml/pull/1705
• Fix Migration Guide Links by @fa9r in https://github.com/zenml-io/zenml/pull/1706
• Improve Input Validation Error Message by @fa9r in https://github.com/zenml-io/zenml/pull/1712
• Update link in cloudpickle_materializer.py by @duarteocarmo in https://github.com/zenml-io/zenml/pull/1713
• catch exceptions in list_model_versions by @avishniakov in https://github.com/zenml-io/zenml/pull/1703
• Rename transition_model_stage to transition_model_version_stage by @avishniakov in https://github.com/zenml-io/zenml/pull/1707
• pandas input to predict by @avishniakov in https://github.com/zenml-io/zenml/pull/1715
• Small fixes to global config docs page by @schustmi in https://github.com/zenml-io/zenml/pull/1714
• Allow specifying extra hosts for LocalDockerOrchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/1709
• Flexible use of ignore_cols in evidently_report_step by @avishniakov in https://github.com/zenml-io/zenml/pull/1711
• Add external artifacts and direct links to run DAG by @fa9r in https://github.com/zenml-io/zenml/pull/1718
• E2E flow example for templates by @avishniakov in https://github.com/zenml-io/zenml/pull/1710
• Fix bug in service connector, Closes #1720 by @soubenz in https://github.com/zenml-io/zenml/pull/1726
• Document the namespace and service account k8s orchestrator settings by @stefannica in https://github.com/zenml-io/zenml/pull/1722
• Refactoring done and reduced some functions complexity and work-time by @thanseefpp in https://github.com/zenml-io/zenml/pull/1719
• Update custom orchestrator guide by @schustmi in https://github.com/zenml-io/zenml/pull/1728
• Improve error message when passing non-json serializable parameter by @schustmi in https://github.com/zenml-io/zenml/pull/1729
• Bump ruff to 0.0.282 by @strickvl in https://github.com/zenml-io/zenml/pull/1730
• Docs and README update for ZenML Cloud by @bcdurak in https://github.com/zenml-io/zenml/pull/1723
• bump MLflow to 2.5.0 by @safoinme in https://github.com/zenml-io/zenml/pull/1708
• Move Examples to Tests by @fa9r in https://github.com/zenml-io/zenml/pull/1673
• Add Error Handling for Empty Pipelines by @fa9r in https://github.com/zenml-io/zenml/pull/1734
• Revert "Add Error Handling for Empty Pipelines" by @fa9r in https://github.com/zenml-io/zenml/pull/1735
• Changing the links to the public roadmap by @bcdurak in https://github.com/zenml-io/zenml/pull/1737
• Add Error Handling for Empty Pipelines by @fa9r in https://github.com/zenml-io/zenml/pull/1736
• Revisit init --template CLI for new templates by @avishniakov in https://github.com/zenml-io/zenml/pull/1731
• Add Python 3.11 Support by @fa9r in https://github.com/zenml-io/zenml/pull/1702
• fix error on scheduled pipelines with KubernetesOrchestrator by @francoisserra in https://github.com/zenml-io/zenml/pull/1738
• Bugfix for identify calls with empty email strings by @bcdurak in https://github.com/zenml-io/zenml/pull/1739

New Contributors

• @duarteocarmo made their first contribution in https://github.com/zenml-io/zenml/pull/1713
• @thanseefpp made their first contribution in https://github.com/zenml-io/zenml/pull/1719

Full Changelog: https://github.com/zenml-io/zenml/compare/0.42.0...0.43.0

- Python
Published by avishniakov over 2 years ago

https://github.com/zenml-io/zenml - 0.42.1

0.42.1

This is a minor release that fixes a couple of minor issues and improves the quickstart example.

Breaking Changes

Disable Implicit Auth Methods for Service Connectors by Default

The implicit authentication methods supported by cloud Service Connectors method may constitute a security risk, because they can give users access to the same cloud resources and services that the ZenML Server itself is allowed to access.

For this reason, the default behavior of ZenML Service Connectors has been changed to disable implicit authentication methods by default. If you try to configure any of the AWS, GCP or Azure Service Connectors using the implicit authentication method, you will now receive an error message.

To enable implicit authentication methods, you have to set the ZENML_ENABLE_IMPLICIT_AUTH_METHODS environment variable or the ZenML helm chart enableImplicitAuthMethods configuration option to true.

What's Changed

• Disable implicit auth methods for service connectors by default by @stefannica in https://github.com/zenml-io/zenml/pull/1704
• New quickstart by @strickvl in https://github.com/zenml-io/zenml/pull/1692
• Set MLflow configuration as environment variables before deployment subprocess by @safoinme in https://github.com/zenml-io/zenml/pull/1705
• Fix Migration Guide Links by @fa9r in https://github.com/zenml-io/zenml/pull/1706

Full Changelog: https://github.com/zenml-io/zenml/compare/0.42.0...0.42.1

- Python
Published by stefannica over 2 years ago

https://github.com/zenml-io/zenml - 0.42.0

This release brings major user experience improvements to how ZenML logs are managed and displayed, removes Python 3.7 support, and fixes the Python 3.10 PyYAML issues caused by the Cython 3.0 release.

Improved Logging UX

The log messages written by ZenML when running pipelines or executing ZenML CLI commands are now more concise and easier to digest and the log message colors were adjusted to be more intuitive. Additionally, all log messages, including custom prints to stdout, now show up as step logs in the dashboard.

Breaking Changes

Python 3.7 Support Dropped

Python 3.7 reached its end of life on on June 27th, 2023. Since then, several MLOps tools have stopped supporting Python 3.7. To prevent dependency issues with our integrations and other open-source packages, ZenML will also no longer support Python 3.7 starting from this release.

Dependency and Integration Version Updates

ZenML now requires PyYAML 6 since older versions are broken under Python 3.10. Subsequently, the following integrations now require a higher package version: - Kubeflow now requires kfp==1.8.22 - Tekton now requires kfk-tekton==1.7.1 - Evidently now requires evidently==0.2.7 or evidently==0.2.8

What's Changed

• Add missing quote in docs by @schustmi in https://github.com/zenml-io/zenml/pull/1674
• Update Local Docker orchestrator docs by @strickvl in https://github.com/zenml-io/zenml/pull/1676
• Relax fastapi dependency version by @fa9r in https://github.com/zenml-io/zenml/pull/1675
• Improve flavor registration error message by @schustmi in https://github.com/zenml-io/zenml/pull/1671
• Simplified Page Iteration by @fa9r in https://github.com/zenml-io/zenml/pull/1679
• Document how to deploy ZenML with custom Docker image by @fa9r in https://github.com/zenml-io/zenml/pull/1672
• Document the ZenML Client and Models by @fa9r in https://github.com/zenml-io/zenml/pull/1678
• Add Label Studio text classification integration and example by @adamwawrzynski in https://github.com/zenml-io/zenml/pull/1658
• Improve yaml config docs page by @schustmi in https://github.com/zenml-io/zenml/pull/1680
• Catch correct exception when trying to access step context by @schustmi in https://github.com/zenml-io/zenml/pull/1681
• Add option to only export requirements for installed integrations by @schustmi in https://github.com/zenml-io/zenml/pull/1682
• Fix copy-paste error (Seldon / KServe docstring) by @strickvl in https://github.com/zenml-io/zenml/pull/1687
• Add avishniakov to teams.yaml by @avishniakov in https://github.com/zenml-io/zenml/pull/1688
• [NEW PR] Set contains_code to 1 instead of True by @kobiche in https://github.com/zenml-io/zenml/pull/1685
• Misc slack fixes by @schustmi in https://github.com/zenml-io/zenml/pull/1686
• Docs: Migration Guide by @fa9r in https://github.com/zenml-io/zenml/pull/1691
• fix: :cardfilebox: Extend pipeline spec storage length by @francoisserra in https://github.com/zenml-io/zenml/pull/1694
• Make the workspace statistics endpoint more performant by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1689
• Deprecate examples CLI by @avishniakov in https://github.com/zenml-io/zenml/pull/1693
• Add cloud server deployment type by @schustmi in https://github.com/zenml-io/zenml/pull/1699
• Fix Python 3.10 PyYAML Installation Issues by @fa9r in https://github.com/zenml-io/zenml/pull/1695
• Remove Python 3.7 Support by @fa9r in https://github.com/zenml-io/zenml/pull/1652
• Improved logs for pipeline execution and CLI usage by @bcdurak in https://github.com/zenml-io/zenml/pull/1664
• Docs: Restructure Advanced Guide by @fa9r in https://github.com/zenml-io/zenml/pull/1698

New Contributors

• @adamwawrzynski made their first contribution in https://github.com/zenml-io/zenml/pull/1658
• @avishniakov made their first contribution in https://github.com/zenml-io/zenml/pull/1688
• @kobiche made their first contribution in https://github.com/zenml-io/zenml/pull/1685

Full Changelog: https://github.com/zenml-io/zenml/compare/0.41.0...0.42.0

- Python
Published by fa9r over 2 years ago

https://github.com/zenml-io/zenml - 0.41.0

ZenML release 0.41.0 comes with a second round of updates to the pipeline and step interface with major changes in how step outputs are defined, how information about previous runs can be fetched programmatically, and how information about the current run can be obtained.

See this docs page for an overview of all pipeline interface changes introduced since release 0.40.0 and for more information on how to migrate your existing ZenML pipelines to the latest syntax.

Fetching Runs Programmatically (#1635)

The entire syntax of fetching previous runs programmatically was majorly redesigned. While the overall user flow is still almost identical, the new approach does not contain pipeline-versioning-related inconsistencies, has a more intuitive syntax, and is also easier for users to learn since the new syntax uses the ZenML Client and response models natively instead of requiring the zenml.post_execution util functions and corresponding ...View wrapper classes.

Accessing Current Run Information (#1648)

How to fetch information about the current pipeline run from within the run has been majorly redesigned: - Instead of being an argument of the step function, the StepContext is now a singleton that can be accessed via the new zenml.get_step_context() function. - The StepContext is now decoupled from the StepEnvironment and the StepEnvironment is deprecated. - The StepContext now contains the full PipelineRunResponseModel and StepRunResponseModel so all information about the run is accessible, not just the name / id / params.

Defining Step Outputs (#1653)

Instead of using the zenml.steps.Output class to annotate steps with multiple outputs, ZenML can now handle Tuple annotations natively and output names can now be assigned to any step output using typing_extensions.Annotated.

What's Changed

• Remove remaining BaseParameters references by @schustmi in https://github.com/zenml-io/zenml/pull/1625
• Fix the s3 integration dependencies by @stefannica in https://github.com/zenml-io/zenml/pull/1641
• Don't run whylogs example on windows by @stefannica in https://github.com/zenml-io/zenml/pull/1644
• Adding the missing pages to our docs by @bcdurak in https://github.com/zenml-io/zenml/pull/1640
• Connectors startup guide and stack component references by @stefannica in https://github.com/zenml-io/zenml/pull/1632
• Fixing the listing functionality of several objects in our CLI by @bcdurak in https://github.com/zenml-io/zenml/pull/1616
• Revamp Post Execution by @fa9r in https://github.com/zenml-io/zenml/pull/1635
• Fix run configuration parameter merging by @schustmi in https://github.com/zenml-io/zenml/pull/1638
• Simplify email opt-in telemetry by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1637
• Fix Step Logs on Windows by @fa9r in https://github.com/zenml-io/zenml/pull/1645
• Improve config section of containerization docs page by @schustmi in https://github.com/zenml-io/zenml/pull/1649
• Validating slack alerter by @bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1609
• Added some error handling in gcp cloud function scheduling by @htahir1 in https://github.com/zenml-io/zenml/pull/1634
• CI: Disable Python 3.7 Mac Runners by @fa9r in https://github.com/zenml-io/zenml/pull/1650
• Redesign StepContext by @fa9r in https://github.com/zenml-io/zenml/pull/1648
• Fix output of dashboard url on pipeline run by @strickvl in https://github.com/zenml-io/zenml/pull/1629
• fix: use k8s orchestrator service account in step pod's manifest by @francoisserra in https://github.com/zenml-io/zenml/pull/1654
• Fix Image Builder Warning Message by @fa9r in https://github.com/zenml-io/zenml/pull/1659
• New step output annotations by @schustmi in https://github.com/zenml-io/zenml/pull/1653
• Add Python 3.10 to listed versions supported via PyPi by @strickvl in https://github.com/zenml-io/zenml/pull/1662
• Add DatabricksShell on list of notebooks allowed to show dashboard by @lucasbissaro in https://github.com/zenml-io/zenml/pull/1643
• Fixing broken links in our examples folder by @bcdurak in https://github.com/zenml-io/zenml/pull/1661
• Feature/frw 2013 docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1639
• Update Pipeline Migration Page by @fa9r in https://github.com/zenml-io/zenml/pull/1667
• Fix/set env variables before installing packages by @lopezco in https://github.com/zenml-io/zenml/pull/1665
• Fix the zenml deploy story by @wjayesh in https://github.com/zenml-io/zenml/pull/1651
• Always keep link to API docs pointed at the version of the release branch by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1636
• Fix BentoML deployer by @safoinme in https://github.com/zenml-io/zenml/pull/1647
• Corrected all mentions in docs from API docs to SDK docs. by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1669
• Update outdated docs by @schustmi in https://github.com/zenml-io/zenml/pull/1668

New Contributors

• @lucasbissaro made their first contribution in https://github.com/zenml-io/zenml/pull/1643
• @lopezco made their first contribution in https://github.com/zenml-io/zenml/pull/1665

Full Changelog: https://github.com/zenml-io/zenml/compare/0.40.3...0.41.0

- Python
Published by fa9r over 2 years ago

https://github.com/zenml-io/zenml - 0.40.3

This is a minor ZenML release that introduces a few new features around Service Connectors and Dashboard improvements:

• Service Connectors can now also be managed through the ZenML Dashboard

  

• the Azure Service Connector is now available in addition to the AWS and GCP ones. It can be used to connect ZenML and Stack Components to Azure cloud infrastructure resources like Azure Blob Storage, Azure Container Registry and Azure Kubernetes Service.

  

• added ability to view pipeline run logs in the ZenML Dashboard

• adds zenml secret export CLI command to export secrets from the ZenML Secret Store to a local file

• adds the ability to create/update ZenML secrets from JSON/YAML files or command line arguments (courtesy of @bhatt-priyadutt)

In addition to that, this release also contains a couple of bug fixes and improvements, including:

• better documentation and fixes for the ZenML Vertex AI Orchestrator and Vertex AI Step Operator
• adjust Seldon and BentoML Steps and Examples to new pipeline interface

What's Changed

• Add option to list all resources when verifying service connector config by @stefannica in https://github.com/zenml-io/zenml/pull/1573
• Fix sandbox time limit by @schustmi in https://github.com/zenml-io/zenml/pull/1602
• Secrets input structure change method by @bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1547
• Implement Azure service connector by @stefannica in https://github.com/zenml-io/zenml/pull/1589
• Adding the ability to tag the source of an event for the analytics by @bcdurak in https://github.com/zenml-io/zenml/pull/1599
• Move all the logic into the script to make it as easy as possible to … by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1605
• Only set mysql session variables when necessary by @schustmi in https://github.com/zenml-io/zenml/pull/1568
• Bug in creating upstream_steps by @sidsaurb in https://github.com/zenml-io/zenml/pull/1601
• Added logs endpoint to display on the dashboard by @htahir1 in https://github.com/zenml-io/zenml/pull/1526
• Fix CI by @fa9r in https://github.com/zenml-io/zenml/pull/1612
• Fix Azure Integration Imports and Improve Flavor Registration Error Handling by @fa9r in https://github.com/zenml-io/zenml/pull/1615
• Deprecation Cleanup by @fa9r in https://github.com/zenml-io/zenml/pull/1608
• Cleanup Local Logging Temp Files by @fa9r in https://github.com/zenml-io/zenml/pull/1621
• Add cloud orchestrator warning message by @strickvl in https://github.com/zenml-io/zenml/pull/1418
• Update custom code run in sandbox docs by @safoinme in https://github.com/zenml-io/zenml/pull/1610
• Remove the GH Actions review reminder bot by @strickvl in https://github.com/zenml-io/zenml/pull/1624
• Automatically optimize image sizes on PR creation by @strickvl in https://github.com/zenml-io/zenml/pull/1626
• Deprecation Warning Improvements by @fa9r in https://github.com/zenml-io/zenml/pull/1620
• Fix ZenML Installation when FastAPI is not Installed by @fa9r in https://github.com/zenml-io/zenml/pull/1627
• Fix unnecessary / extra deprecation warnings by @strickvl in https://github.com/zenml-io/zenml/pull/1630
• Add zenml secret export CLI command by @fa9r in https://github.com/zenml-io/zenml/pull/1607
• Missing pipeline features docs by @schustmi in https://github.com/zenml-io/zenml/pull/1619
• Fix for valid secret name by @bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1617
• Fix and document Vertex AI orchestrator and step operator by @stefannica in https://github.com/zenml-io/zenml/pull/1606
• Deprecate KServe Integration by @fa9r in https://github.com/zenml-io/zenml/pull/1631
• Adjust Seldon Steps and Examples to New Pipeline Interface by @fa9r in https://github.com/zenml-io/zenml/pull/1560
• Adjust BentoML Steps and Example to New Pipeline Interface by @fa9r in https://github.com/zenml-io/zenml/pull/1614
• Moved kubernetes imports to inner function to avoid module not found error by @htahir1 in https://github.com/zenml-io/zenml/pull/1622

New Contributors

• @sidsaurb made their first contribution in https://github.com/zenml-io/zenml/pull/1601

Full Changelog: https://github.com/zenml-io/zenml/compare/0.40.3...0.40.2

- Python
Published by stefannica over 2 years ago

https://github.com/zenml-io/zenml - 0.40.2

Minor release with a few docs changes and preparation of the MLOps Platform Sandbox

What's Changed

• Update Example for sandbox by @safoinme in https://github.com/zenml-io/zenml/pull/1576
• Document zenml show by @fa9r in https://github.com/zenml-io/zenml/pull/1570
• Clean up for the new docs by @bcdurak in https://github.com/zenml-io/zenml/pull/1575
• Add orchestrator outputs for sandbox examples by @strickvl in https://github.com/zenml-io/zenml/pull/1579
• Docs: Added some adjustments to the code repository page. by @bcdurak in https://github.com/zenml-io/zenml/pull/1582
• Sandbox documentation (and other docs updates) by @strickvl in https://github.com/zenml-io/zenml/pull/1574
• Minor README update regarding the sandbox. by @bcdurak in https://github.com/zenml-io/zenml/pull/1586
• Fix failing mlflow_tracking example test by @strickvl in https://github.com/zenml-io/zenml/pull/1581
• Bump ruff and mypy by @strickvl in https://github.com/zenml-io/zenml/pull/1590
• Remove config.yaml references in example docs by @strickvl in https://github.com/zenml-io/zenml/pull/1585
• update mlflow tracking example and reduce number of epochs by @safoinme in https://github.com/zenml-io/zenml/pull/1598
• Improve error message when requirements file does not exist by @schustmi in https://github.com/zenml-io/zenml/pull/1596
• Fix build reuse for integrations with apt packages by @schustmi in https://github.com/zenml-io/zenml/pull/1594
• make the Github repo token optional by @safoinme in https://github.com/zenml-io/zenml/pull/1593

Full Changelog: https://github.com/zenml-io/zenml/compare/0.40.1...0.40.2

- Python
Published by htahir1 over 2 years ago

https://github.com/zenml-io/zenml - 0.40.1

0.40.1

Small bug and docs fixes following the 0.40.0 release.

What's Changed

• Convert dict to tuple in ArtifactConfiguration validator by @schustmi in https://github.com/zenml-io/zenml/pull/1571
• Docs cleanup by @schustmi in https://github.com/zenml-io/zenml/pull/1569
• Fix boto3<=1.24.59 by @safoinme in https://github.com/zenml-io/zenml/pull/1572

Full Changelog: https://github.com/zenml-io/zenml/compare/0.40.0...0.40.1

- Python
Published by strickvl over 2 years ago

https://github.com/zenml-io/zenml - 0.40.0

ZenML release 0.40.0 introduces two big updates: a fresh and more flexible pipeline interface and a new way to connect and authenticate with external services in ZenML Connectors. See below for full details on these two major new sets of functionality.

The release also contains many bug fixes and quality-of-life improvements. Specifically, we reworked our documentation from the ground up with particular attention to the structure to help you find what you need. Our Label Studio integration example is now working again and allows you to use more recent versions of the label-studio package that backs it.

A Fresh Pipeline Interface

This release introduces a completely reworked interface for developing your ZenML steps and pipelines:

No More BaseParameters Class

In the previous version, you had to define a separate class for step parameters using BaseParameters. This is no longer necessary, although it is still supported for backward compatibility. You can now pass parameters directly in the step function:

python @step def trainer(data: pd.Dataframe, lr: float = 0.1, gamma: Optional[float] = 0.02) -> ...: print(lr) print(gamma)

Simplified Pipeline Execution

With the new changes, you no longer need to create a pipeline instance and then run it separately. You can now pass parameters directly at pipeline instance creation and execute the pipeline in a single step:

python my_pipeline(lr=0.000001)

Multiple step invocations

You can now call steps multiple times inside a pipeline, allowing you to create more complex workflows and reuse steps with different parameters:

python @pipeline def my_pipeline(step_count: int) -> None: data = load_data_step() after = [] for i in range(step_count): train_step(data, learning_rate=i * 0.0001, id=f"train_step_{i}") after.append(f"train_step_{i}") model = select_model_step(..., after=after)

Pipeline inputs and outputs

Pipelines can now define inputs and outputs, providing a clearer interface for working with data and dependencies between pipelines:

python @pipeline(enable_cache=False) def subpipeline(pipeline_param: int): out = step_1(k=None) step_2(a=3, b=pipeline_param) return 17

Call pipelines from within pipelines (alpha)

You can now call pipelines within other pipelines. This currently does not execute the inner pipeline but instead adds its steps to the parent pipeline, allowing you to create modular and reusable workflows:

python @pipeline(enable_cache=False) def my_pipeline(a: int = 1): p1_output = subpipeline(pipeline_param=22) step_2(a=a, b=p1_output)

Flexible types on steps

Increased flexibility when defining steps: Steps can now have Optional, Union, and Any type annotations for their inputs and outputs. Additionally, default values are allowed for step inputs.

python @step def trainer(data: pd.Dataframe, start_model: Union[svm.SVC, svm.SVR], coef0: Optional[int] = None) -> Any: pass

Easy to debug steps

You can now easily run a step outside of a pipeline, making it easier to test and debug your code:

python trainer(data=pd.Dataframe(...), start_model=svc.SVC(...))

External Artifacts

External artifacts can be used to pass values to steps that are not produced by an upstream step. This provides more flexibility when working with external data or models:

```python from zenml.steps import ExternalArtifact

@pipeline def mypipeline(lr: float): data = processdata() trainer(data=data, start_model=ExternalArtifact(svc.SVC(...))) ```

Get Started with the new interface and features!

To get started, simply import the new @step and @pipeline decorator and check out our new starter guide for more information.

```python from zenml import step, pipeline

@step def my_step(...): ...

@pipeline def my_pipeline(...): ... ```

The old pipeline and step interface is still working using the imports from previous ZenML releases but is deprecated and will be removed in the future.

'Connectors' for authentication

In this update, we're pleased to present a new feature to ZenML: Service Connectors. The intention behind these connectors is to offer a reliable and more user-friendly method for integrating ZenML with external resources and services. We aim to simplify processes such as validating, storing, and generating security-sensitive data, along with the authentication and authorization of access to external services. We believe ZenML Service Connectors will be a useful tool to alleviate some of the common challenges in managing pipeline across various Stack Components.

Regardless of your background in infrastructure management - whether you're a beginner looking for quick cloud stack integration, or an experienced engineer focused on maintaining robust infrastructure security practices - our Service Connectors are designed to assist your work while maintaining high security standards.

Here are just a few ways you could use ZenML Service Connectors:

• Easy utilization of cloud resources: With ZenML's Service Connectors, you can use resources from AWS, GCP, and Azure without the need for extensive knowledge of cloud infrastructure or environment configuration. All you'll need is a ZenML Service Connector and a few Python libraries.
• Assisted setup with security in mind: Our Service Connectors come with features for configuration validation and verification, the generation of temporary, low-privilege credentials, and pre-authenticated and pre-configured clients for Python libraries.
• Easy local configuration transfer: ZenML's Service Connectors aim to resolve the reproducibility issue in ML pipelines. They do this by automatically transferring authentication configurations and credentials from your local machine, storing them securely, and allowing for effortless sharing across different environments.

Visit our documentation pages to learn more about ZenML Connectors and how you can use them in a way that supports your ML workflows.

What's Changed

• Cleanup remaining references of zenml.artifacts by @fa9r in https://github.com/zenml-io/zenml/pull/1534
• Upgrading the black version by @bcdurak in https://github.com/zenml-io/zenml/pull/1535
• Remove dev breakpoints by @strickvl in https://github.com/zenml-io/zenml/pull/1540
• Removing old option command from contribution doc by @bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1544
• Improve CLI help text for zenml integration install -i ... by @strickvl in https://github.com/zenml-io/zenml/pull/1545
• Fix RestZenStore error handling for list responses by @fa9r in https://github.com/zenml-io/zenml/pull/1539
• Simplify Dashboard UX via zenml.show() by @fa9r in https://github.com/zenml-io/zenml/pull/1511
• Removed hardcoded variable by @bhatt-priyadutt in https://github.com/zenml-io/zenml/pull/1543
• Revert Quickstart Changes by @fa9r in https://github.com/zenml-io/zenml/pull/1546
• Deprecate some long overdue functions by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1541
• ZenML Connectors by @stefannica in https://github.com/zenml-io/zenml/pull/1514
• Fix automatic dashboard opening after zenml up by @fa9r in https://github.com/zenml-io/zenml/pull/1551
• Update Neptune README by @strickvl in https://github.com/zenml-io/zenml/pull/1554
• Update example READMEs following deployment PR by @strickvl in https://github.com/zenml-io/zenml/pull/1555
• Fix and update Label Studio example by @strickvl in https://github.com/zenml-io/zenml/pull/1542
• Fix linter errors by @stefannica in https://github.com/zenml-io/zenml/pull/1557
• Add Vertex as orchestrator and step operator to deploy CLI by @wjayesh in https://github.com/zenml-io/zenml/pull/1559
• Fix dashboard secret references by @stefannica in https://github.com/zenml-io/zenml/pull/1561
• New pipeline and step interface by @schustmi in https://github.com/zenml-io/zenml/pull/1466
• Major Documentation Rehaul by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1562
• Easy CSV Visualization by @fa9r in https://github.com/zenml-io/zenml/pull/1556

New Contributors

• @bhatt-priyadutt made their first contribution in https://github.com/zenml-io/zenml/pull/1544

Full Changelog: https://github.com/zenml-io/zenml/compare/0.39.1...0.40

- Python
Published by strickvl over 2 years ago

https://github.com/zenml-io/zenml - 0.39.1

Minor hotfix release for running ZenML in Google Colab environments.

What's Changed

• Fix Source Resolving in Colab by @fa9r in https://github.com/zenml-io/zenml/pull/1530

Full Changelog: https://github.com/zenml-io/zenml/compare/0.39.0...0.39.1

- Python
Published by fa9r almost 3 years ago

https://github.com/zenml-io/zenml - 0.39.0

ZenML release 0.39.0 introduces several big new features: - The zenml stack recipe CLI commands now support fine-grained handling of individual stack components. - Artifacts are now automatically visualized in the dashboard. - Materializers received an overhaul: a new cloudpickle default materializer was added that works for arbitrary objects, and a pycaret materializer that can handle various modeling frameworks in a unified format.

The release also contains many bug fixes and quality-of-life improvements, such as new settings options for the SageMaker and Kubernetes orchestrators.

Individual Stack Component Deployment

In this release, we've enhanced the ZenML stack recipe CLI to support conditional deployment, destruction, and configuration of individual stack components. Users can now quickly deploy and destroy components with options for each flavor, and pass a config file for custom variables. The new output CLI command allows users to retrieve outputs from their recipes. Overall, this update streamlines deploying and managing stack components by providing a more efficient and user-friendly experience.

Artifact Visualization

Artifact visualizations are now automatically extracted by ZenML and embedded in the ZenML dashboard. Visualizations can now be defined by overriding the save_visualizations method of the materializer that handles an artifact. These visualizations are then automatically shown in the dashboard and can also be displayed in Jupyter notebooks using the new visualize post-execution method.

Default Cloudpickle Materializer

ZenML now uses cloudpickle under the hood to save/load artifacts that other materializers cannot handle. This makes it even easier to get started with ZenML since you no longer need to define custom materializers if you just want to experiment with some new data types.

What's Changed

• Docs/zenml hub documentation by @bcdurak in https://github.com/zenml-io/zenml/pull/1490
• Sort integration list before display by @strickvl in https://github.com/zenml-io/zenml/pull/1494
• Update docs to surface CLI filtering syntax by @strickvl in https://github.com/zenml-io/zenml/pull/1496
• ZenML Hub Tests & CLI Improvements by @fa9r in https://github.com/zenml-io/zenml/pull/1495
• Delete Legacy Docs by @fa9r in https://github.com/zenml-io/zenml/pull/1497
• Improve the REST API error handling by @stefannica in https://github.com/zenml-io/zenml/pull/1451
• Fix circular import of PipelineRunConfiguration by @schustmi in https://github.com/zenml-io/zenml/pull/1501
• Delete Deprecated Artifacts and Materializer Code by @fa9r in https://github.com/zenml-io/zenml/pull/1498
• Allow filtering runs by code repo id by @schustmi in https://github.com/zenml-io/zenml/pull/1499
• Add example to docs for passing stack component specific settings by @christianversloot in https://github.com/zenml-io/zenml/pull/1506
• Increase step run field lengths by @schustmi in https://github.com/zenml-io/zenml/pull/1503
• Fix Sagemaker orchestrator pipeline name bug by @strickvl in https://github.com/zenml-io/zenml/pull/1508
• Generate unique SageMaker training job name based on pipeline and ste… by @christianversloot in https://github.com/zenml-io/zenml/pull/1505
• [CI Fix] Pin Llama Index Version by @fa9r in https://github.com/zenml-io/zenml/pull/1516
• Basic PyCaret integration and materializer by @christianversloot in https://github.com/zenml-io/zenml/pull/1512
• Specify line endings for different operating systems by @strickvl in https://github.com/zenml-io/zenml/pull/1513
• Extend SageMakerOrchestratorSettings with processor_args enabling step level configuration by @christianversloot in https://github.com/zenml-io/zenml/pull/1509
• Fix post execution get_pipeline() and pipeline.get_runs() by @fa9r in https://github.com/zenml-io/zenml/pull/1510
• Default cloudpickle Materializer & Materializer Inheritance by @fa9r in https://github.com/zenml-io/zenml/pull/1507
• Artifact Visualization by @fa9r in https://github.com/zenml-io/zenml/pull/1472
• Add Kubernetes Orchestrator Settings by @fa9r in https://github.com/zenml-io/zenml/pull/1518
• Bump ruff to 0.0.265 by @strickvl in https://github.com/zenml-io/zenml/pull/1520
• feat: Set cloud function service account to the one defined in Vertex… by @francoisserra in https://github.com/zenml-io/zenml/pull/1519
• Fix Kubernetes Orchestrator Config Loading by @fa9r in https://github.com/zenml-io/zenml/pull/1523
• Resolve path during module resolving by @schustmi in https://github.com/zenml-io/zenml/pull/1521
• Fix SO_REUSEPORT issue by @fa9r in https://github.com/zenml-io/zenml/pull/1524
• Add individual stack component deployment through recipes by @wjayesh in https://github.com/zenml-io/zenml/pull/1328
• Raise 501 for Unauthenticated Artifact Stores by @fa9r in https://github.com/zenml-io/zenml/pull/1522
• Fix Duplicate Step Error by @fa9r in https://github.com/zenml-io/zenml/pull/1527
• Fix pulling of stack recipes on zenml init by @wjayesh in https://github.com/zenml-io/zenml/pull/1528
• Store dockerfile and requirements for builds by @schustmi in https://github.com/zenml-io/zenml/pull/1525

Full Changelog: https://github.com/zenml-io/zenml/compare/0.38.0...0.39.0

- Python
Published by fa9r almost 3 years ago

https://github.com/zenml-io/zenml - 0.38.0

Get ready to supercharge your ML pipelines with the revolutionary ZenML Hub, a central platform that enables our users to search, share and discover community-contributed code, now available with the ZenML 0.38.0 release! Our cutting-edge plugin system lets you seamlessly contribute and consume a variety of code pieces, including stack component flavors, pipelines, steps, materializers, and more. With the ZenML Hub, you can easily integrate these components into your workflow and take your ML game to the next level.

Say goodbye to tedious integration processes and hello to exciting, effortless experimentation!

The ZenML Hub makes its debut already equipped with a set of plugins that include ZenML-verified steps (with a special focus on data loader steps), setting the standard for seamless integration. And that's just the beginning! Our team is already hard at work on expanding the Hub's capabilities, with plans to introduce exciting new entities like materializers and flavors. Stay tuned for more groundbreaking updates as we continue to revolutionize the ML landscape!

But why wait ? With the ZenML Hub, anyone can contribute their own plugins and take part in accelerating innovation in the ML field. Be a part of the cutting-edge community that is shaping the future of machine learning and driving impactful solutions forward. Becoming a contributor starts with logging in to the ZenML Hub through the ZenML zenml hub login CLI command or through the ZenML dashboard. More information on this process is available in our documentation.

If you're interested in learning more about our motivation for implementing the ZenML Hub and our plans for its future, we invite you to read our new blog post. In addition to our technical documentation, the blog post provides a comprehensive overview of the ZenML Hub's goals and objectives, as well as the features that we plan to introduce in the future.

Aside from this major new feature, the release also includes a number of small improvements and bug fixes, the most notable of which is the ability to manage secrets from the ZenML dashboard, as well as to reference them in the configuration of stack components.

What's Changed

• Fix broken ENV variable by @strickvl in https://github.com/zenml-io/zenml/pull/1458
• fix screenshot size in code repo by @safoinme in https://github.com/zenml-io/zenml/pull/1467
• Fix CI (Deepchecks integration tests) by @fa9r in https://github.com/zenml-io/zenml/pull/1470
• chore: update teams.yml by @Cahllagerfeld in https://github.com/zenml-io/zenml/pull/1459
• Fix BuiltInContainerMaterializer for subtypes and non-built-in types by @fa9r in https://github.com/zenml-io/zenml/pull/1464
• Kubernetes Orchestrator Improvements by @fa9r in https://github.com/zenml-io/zenml/pull/1460
• Fix flaky CLI tests by @schustmi in https://github.com/zenml-io/zenml/pull/1465
• Fix circular import during type checking by @schustmi in https://github.com/zenml-io/zenml/pull/1463
• Allow secret values replacement in REST API PUT by @stefannica in https://github.com/zenml-io/zenml/pull/1471
• Fix two steps race condition by @safoinme in https://github.com/zenml-io/zenml/pull/1473
• Downgrading ZenML Version in global config by @safoinme in https://github.com/zenml-io/zenml/pull/1474
• Revert "Downgrading ZenML Version in global config" by @safoinme in https://github.com/zenml-io/zenml/pull/1476
• Add metadata to stack components by @wjayesh in https://github.com/zenml-io/zenml/pull/1416
• remove modules from the list output for stack recipes by @wjayesh in https://github.com/zenml-io/zenml/pull/1480
• Pin openai integration to >0.27.0 by @strickvl in https://github.com/zenml-io/zenml/pull/1461
• Apply formatting fixes to /scripts by @strickvl in https://github.com/zenml-io/zenml/pull/1462
• Move import outside of type checking by @schustmi in https://github.com/zenml-io/zenml/pull/1482
• Delete extra word from bentoml docs by @strickvl in https://github.com/zenml-io/zenml/pull/1484
• Remove top-level config from recommended repo structure by @schustmi in https://github.com/zenml-io/zenml/pull/1485
• Bump mypy and ruff by @strickvl in https://github.com/zenml-io/zenml/pull/1481
• ZenML Version Downgrade - Silence Warnning by @safoinme in https://github.com/zenml-io/zenml/pull/1477
• Update ZenServer recipes to include secret stores by @wjayesh in https://github.com/zenml-io/zenml/pull/1483
• Fix alembic order by @schustmi in https://github.com/zenml-io/zenml/pull/1487
• Fix source resolving for classes in notebooks by @schustmi in https://github.com/zenml-io/zenml/pull/1486
• fix: use poolpreping to discard invalid SQL connections when borrow… by @francoisserra in https://github.com/zenml-io/zenml/pull/1489

New Contributors

• @Cahllagerfeld made their first contribution in https://github.com/zenml-io/zenml/pull/1459
• @francoisserra made their first contribution in https://github.com/zenml-io/zenml/pull/1489

Full Changelog: https://github.com/zenml-io/zenml/compare/0.37.0...0.38.0

- Python
Published by stefannica almost 3 years ago

https://github.com/zenml-io/zenml - 0.37.0

In this ZenML release, we are pleased to introduce a compelling new feature: ZenML Code Repositories. This innovative addition formalizes the principles of code versioning and tracking while consolidating their pivotal role in executing pipelines and caching pipeline steps. With Code Repositories, ZenML is equipped to maintain an accurate record of the code version employed in your pipeline runs. Furthermore, executing a pipeline that is monitored by a registered code repository can significantly accelerate the Docker image building process for containerized stack components.

As is the case with everything ZenML, we designed the ZenML Code Repository concept as a highly extensible abstraction. The update defines the basic Code Repository interface an includes two implementations integrating ZenML with two popular code repository flavors: GitHub and GitLab.

Other Enhancements

We've updated the pytorch-lightning integration to support the 2.0 version. We also updated the mlflow integration to support the 2.2.2 version.

IMPORTANT: it is not recommended to continue using MLflow older than 2.2.1 as a model registry with ZenML, as it is vulnerable to a security issue.

Last but not least, two stellar additions from our community members:

• zenml stack delete now supports a --recursive flag to delete all components in a stack. Many thanks to @KenmogneThimotee for the contribution!
• the ZenML Sagemaker step operator has been expanded to support S3 input data and additional input arguments. Many thanks to @christianversloot for the contribution!

Breaking Changes

The ZenML GitHub Orchestrator and GitHub Secrets Manager have been removed in this release. Given that their concerns overlapped with the new ZenML GitHub Code Repository and they didn't provide sufficient value on their own, we decided to discontinue them. If you were using these components, you can continue to use GitHub Actions to run your pipelines, in combination with the ZenML GitHub Code Repository.

What's Changed

• Test integration for seldon example by @safoinme in https://github.com/zenml-io/zenml/pull/1285
• Update pytorch-lightning to support 2.0 by @safoinme in https://github.com/zenml-io/zenml/pull/1425
• Code repository by @schustmi in https://github.com/zenml-io/zenml/pull/1344
• Bump ruff to 0.259 by @strickvl in https://github.com/zenml-io/zenml/pull/1439
• Change pipeline_run_id to run_name by @safoinme in https://github.com/zenml-io/zenml/pull/1390
• Update mypy>=1.1.1 and fix new errors by @safoinme in https://github.com/zenml-io/zenml/pull/1432
• Add --upgrade option to ZenML integration install by @safoinme in https://github.com/zenml-io/zenml/pull/1435
• Bump MLflow to 2.2.2 by @safoinme in https://github.com/zenml-io/zenml/pull/1441
• HuggingFace Spaces server deployment option by @strickvl in https://github.com/zenml-io/zenml/pull/1427
• Bugfix for server import by @bcdurak in https://github.com/zenml-io/zenml/pull/1442
• Fix HF Spaces URL by @strickvl in https://github.com/zenml-io/zenml/pull/1444
• Remove all zenml.cli imports outside of zenml.cli by @fa9r in https://github.com/zenml-io/zenml/pull/1447
• Add recursive deletion of components for zenml stack delete by @KenmogneThimotee in https://github.com/zenml-io/zenml/pull/1437
• Temporarily disable primary key requirement for newer mysql versions by @schustmi in https://github.com/zenml-io/zenml/pull/1450
• Add step name suffix for sagemaker job name by @schustmi in https://github.com/zenml-io/zenml/pull/1452
• Code repo docs by @schustmi in https://github.com/zenml-io/zenml/pull/1448
• Allow resource settings for airflow kubernetes pod operators by @schustmi in https://github.com/zenml-io/zenml/pull/1378
• SageMaker step operator: expand input arguments and add support for S3 input data by @christianversloot in https://github.com/zenml-io/zenml/pull/1381
• Add Screenshots to Code Repo Token by @safoinme in https://github.com/zenml-io/zenml/pull/1454

New Contributors

• @KenmogneThimotee made their first contribution in https://github.com/zenml-io/zenml/pull/1437
• @christianversloot made their first contribution in https://github.com/zenml-io/zenml/pull/1381

Full Changelog: https://github.com/zenml-io/zenml/compare/0.36.1...0.37.0

- Python
Published by stefannica almost 3 years ago

https://github.com/zenml-io/zenml - 0.36.1

This minor release contains some small fixes and improvements.

• We fixed a bug with the way hooks were being parsed, which was causing pipelines to fail.
• We brought various parts of the documentation up to date with features that had previously been added, notably the new image building functionality.
• We added a failure hook that connects to OpenAI's ChatGPT API to allow you to receive a message when a pipeline fails that includes suggestions on how to fix the failing step.
• We added a new integration with langchain and llama_hub to allow you to build on top of those libraries as part of a more robust MLOps workflow.
• We made the first some bigger changes to our analytics system to make it more robust and secure. This release begins that migration. Users should expect no changes in behavior and all telemetry-related preferences will be preserved.

What's Changed

• Fix hook parser by @strickvl in https://github.com/zenml-io/zenml/pull/1428
• Fix some pipeline bugs by @schustmi in https://github.com/zenml-io/zenml/pull/1426
• Add image builders to Examples by @safoinme in https://github.com/zenml-io/zenml/pull/1434
• ZenML Failure Hook for OpenAI ChatGPT fixes by @strickvl in https://github.com/zenml-io/zenml/pull/1430
• Integrations with langchain and llama_hub by @fa9r in https://github.com/zenml-io/zenml/pull/1404
• Add basic tests for the server and recipes CLI by @wjayesh in https://github.com/zenml-io/zenml/pull/1306
• Add to our alembic migration guide by @strickvl in https://github.com/zenml-io/zenml/pull/1423
• Analytics 2.0 by @bcdurak in https://github.com/zenml-io/zenml/pull/1411
• Improve Slack Alerter by adding message blocks by @soubenz in https://github.com/zenml-io/zenml/pull/1402
• Add HF deployment type by @strickvl in https://github.com/zenml-io/zenml/pull/1438

New Contributors

• @soubenz made their first contribution in https://github.com/zenml-io/zenml/pull/1402

Full Changelog: https://github.com/zenml-io/zenml/compare/0.36.0...0.36.1

- Python
Published by strickvl almost 3 years ago

https://github.com/zenml-io/zenml - 0.36.0

Our latest release adds hooks to ZenML pipelines to handle custom logic that occurs on pipeline failure or success. This is a powerful feature that allows you to easily receive custom alerts, for example, when a pipeline fails or succeeds. (Check out our video showcasing the feature here.)

The release is also packed with bug fixes and documentation updates. Some smaller improvements include an increase of the step_configurations column size in the database to accommodate really large configurations and the ability to click through to orchestrator logs for the Sagemaker orchestrator directly from the ZenML dashboard.

Breaking Changes

Secrets are now handled internally by ZenML. This changes some behaviors that you may have become used to with the (now-deprecated) Secrets Manager stack component. The default behavior for the KServe and Seldon Core Model Deployer if explicit credentials are not configured through the secret stack component attribute has changed. Now, the model deployer will attempt to reuse credentials configured for the Artifact Store in the same stack and may, in some cases, fail if it cannot use them. In most cases, if credentials are not configured for the active Artifact Store, the model deployer will assume some form of implicit in-cloud authentication is configured for the Kubernetes cluster where KServe / Seldon Core is installed and default to using that.

What's Changed

• Add CLI utils tests by @strickvl in https://github.com/zenml-io/zenml/pull/1383
• Don't use docker client when building images remotely by @schustmi in https://github.com/zenml-io/zenml/pull/1394
• Fix zenml-quickstart-model typo by @safoinme in https://github.com/zenml-io/zenml/pull/1397
• Ignore starting quotes from Artifact store path by @safoinme in https://github.com/zenml-io/zenml/pull/1388
• CI speed improvements by @stefannica in https://github.com/zenml-io/zenml/pull/1384
• Fix stack recipe link by @strickvl in https://github.com/zenml-io/zenml/pull/1393
• Switch FastAPI response class to orjson so NaN values don't break the server by @fa9r in https://github.com/zenml-io/zenml/pull/1395
• Numpy materializer metadata for arrays with strings by @safoinme in https://github.com/zenml-io/zenml/pull/1392
• Fix last remaining runs index by @stefannica in https://github.com/zenml-io/zenml/pull/1399
• Add failure (and success hooks) by @htahir1 in https://github.com/zenml-io/zenml/pull/1361
• Replace pyspelling with typos by @strickvl in https://github.com/zenml-io/zenml/pull/1400
• Fix the download nltk param for report step by @wjayesh in https://github.com/zenml-io/zenml/pull/1409
• Add build_timeout attribute to GCPImageBuilderConfig by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/1408
• Bump ruff to v0.255 by @strickvl in https://github.com/zenml-io/zenml/pull/1403
• Update title of deployment docs page by @strickvl in https://github.com/zenml-io/zenml/pull/1412
• Changed to debug log by @htahir1 in https://github.com/zenml-io/zenml/pull/1406
• Fix incorrect --sort_by help text by @strickvl in https://github.com/zenml-io/zenml/pull/1413
• Document CLI filtering query language by @strickvl in https://github.com/zenml-io/zenml/pull/1414
• Fix GitHub pip download cache key by @stefannica in https://github.com/zenml-io/zenml/pull/1405
• Add orchestrator logs link for Sagemaker by @strickvl in https://github.com/zenml-io/zenml/pull/1375
• Phase out secrets managers from other stack components. by @stefannica in https://github.com/zenml-io/zenml/pull/1401
• Add MLflow UI message to quickstart example and fix autolog spillage by @stefannica in https://github.com/zenml-io/zenml/pull/1421
• Add tests for the model registry by @safoinme in https://github.com/zenml-io/zenml/pull/1415
• Remove Aspell installation by @strickvl in https://github.com/zenml-io/zenml/pull/1419
• Increase step_configurations column size to 2^24 by @strickvl in https://github.com/zenml-io/zenml/pull/1422
• Add help text for enable_service option in recipe sub-command by @safoinme in https://github.com/zenml-io/zenml/pull/1424

Full Changelog: https://github.com/zenml-io/zenml/compare/0.35.1...test

- Python
Published by strickvl almost 3 years ago

https://github.com/zenml-io/zenml - 0.35.1

Note: This release replaces the previous 0.35.0 release that was yanked from PyPI due to a bug. If you already installed 0.35.0 and are experiencing issues, we recommend you downgrade to 0.34.0 before installing and upgrading to 0.35.1.

This release is packed with big features as well as documentation updates and some bug fixes.

The 0.35.1 release puts models front and center in ZenML with the addition of the Model Registry abstraction and Stack Component. You can now register, version and manage models as first class citizens in ZenML. This is a major milestone for ZenML and we are excited to see what you build with it! The introduction of Model Registries greatly simplifies the journey that the model takes from training to deployment and extends the ZenML ecosystem to include model registry tools and libraries. The first Model Registry integration included in this release is MLFlow, with many more to come in the future.

This release also continues the deprecation of Secrets Managers and the introduction of Secret Stores. You now have the option of configuring the ZenML server to use AWS, GCP, Azure or Hashicorp Vault directly as a centralized secrets store back-end. This is meant to replace all Secrets Manager flavors which were previously used to store secrets using the same cloud services.

Please be reminded that all Secrets Managers are now deprecated and will be removed in the near future. We recommend that you migrate all your secrets from the Secrets Manager stack components to the centralized secrets store by means of the included zenml secrets-manager secret migrate CLI command.

Last but not least, this release includes an updated Evidently integration that is compatible with the latest and greatest features from Evidently: reports and test suites. Check out the updated example to get a feel for the new features.

Breaking Changes

This release introduces a few breaking changes. Please update your code to reflect the changes below:

• the order of pipelines and runs in the post-execution results has been reversed. This means that the most recent pipeline and pipeline run can be accessed using the first index of the respective lists instead of the last index. This change was made to make the post-execution results more intuitive and to allow returning multi-page results in the future. This is a code snippet outlining the changes that you need to make in your post-execution code:

  ```python from zenml.postexecution import getpipelines, getunlistedruns

  pipelines = get_pipelines()

  instead of calling this to get the pipeline last created

  latest_pipeline = pipelines[-1]

  you now have to call this

  latest_pipeline = pipelines[0]

  and instead of calling this to get the latest run of a pipeine

  latestpipelinerun = latestpipeline.getruns()[-1]

  or

  latestpipelinerun = latest_pipeline.runs[-1]

  you now have to call this

  latestpipelinerun = latestpipeline.getruns()[0]

  or

  latestpipelinerun = latest_pipeline.runs[0]

  the same applies to the unlisted runs; instead of

  lastunlistedrun = getunlistedruns()[-1]

  you now have to call this

  lastunlistedrun = getunlistedruns()[0] ```

• if you were using the StepEnvironment to fetch the name of the active step in your step implementation, this name no longer reflects the name of the step function. Instead, it now reflects the name of the step used in the pipeline DAG, similar to what you would see in the ZenML dashboard when visualizing the pipeline. This is also implicitly reflected in the output of zenml model-deployer model CLI commands.

What's Changed

• Upgrade dev dependencies by @strickvl in https://github.com/zenml-io/zenml/pull/1334
• Add warning when attempting server connection without user permissions by @strickvl in https://github.com/zenml-io/zenml/pull/1314
• Keep CLI help text for zenml pipeline to a single line by @strickvl in https://github.com/zenml-io/zenml/pull/1338
• Rename page attributes by @schustmi in https://github.com/zenml-io/zenml/pull/1266
• Add missing docs for pipeline build by @schustmi in https://github.com/zenml-io/zenml/pull/1341
• Sagemaker orchestrator docstring and example update by @strickvl in https://github.com/zenml-io/zenml/pull/1350
• Fix secret create docs error for secret store by @strickvl in https://github.com/zenml-io/zenml/pull/1355
• Update README for test environment provisioning by @strickvl in https://github.com/zenml-io/zenml/pull/1336
• Disable name prefix matching when updating/deleting entities by @schustmi in https://github.com/zenml-io/zenml/pull/1345
• Add Kubeflow Pipeline UI Port to deprecated config by @safoinme in https://github.com/zenml-io/zenml/pull/1358
• Small clarifications for slack alerter by @htahir1 in https://github.com/zenml-io/zenml/pull/1365
• update Neptune integration for v1.0 compatibility by @AleksanderWWW in https://github.com/zenml-io/zenml/pull/1335
• Integrations conditional requirements by @safoinme in https://github.com/zenml-io/zenml/pull/1255
• Fix fetching versioned pipelines in post execution by @schustmi in https://github.com/zenml-io/zenml/pull/1363
• Load artifact store before loading artifact to register filesystem by @schustmi in https://github.com/zenml-io/zenml/pull/1367
• Remove poetry from CI by @schustmi in https://github.com/zenml-io/zenml/pull/1346
• Fix Sagemaker example readme by @strickvl in https://github.com/zenml-io/zenml/pull/1370
• Update evidently to include reports and tests by @wjayesh in https://github.com/zenml-io/zenml/pull/1283
• Fix neptune linting error on develop (and bump ruff) by @strickvl in https://github.com/zenml-io/zenml/pull/1372
• Add pydantic materializer by @htahir1 in https://github.com/zenml-io/zenml/pull/1371
• Registering GIFs added by @htahir1 in https://github.com/zenml-io/zenml/pull/1368
• Refresh CLI cheat sheet by @strickvl in https://github.com/zenml-io/zenml/pull/1347
• Add dependency resolution docs by @strickvl in https://github.com/zenml-io/zenml/pull/1337
• [BUGFIX] Fix error while using an existing SQL server with GCP ZenServer by @wjayesh in https://github.com/zenml-io/zenml/pull/1353
• Update step name assignment with the parameter name by @strickvl in https://github.com/zenml-io/zenml/pull/1310
• Copy huggingface data directory to local before loading in materializers by @TimovNiedek in https://github.com/zenml-io/zenml/pull/1351
• Update huggingface token classification example by @strickvl in https://github.com/zenml-io/zenml/pull/1369
• Use the most specialized materializer based on MRO by @schustmi in https://github.com/zenml-io/zenml/pull/1376
• Update Kserve to support 0.10.0 by @safoinme in https://github.com/zenml-io/zenml/pull/1373
• Add more examples to integration tests by @schustmi in https://github.com/zenml-io/zenml/pull/1245
• Fix order of runs and order of pipelines in post-execution by @stefannica in https://github.com/zenml-io/zenml/pull/1380
• Add Cloud Secrets Store back-ends by @stefannica in https://github.com/zenml-io/zenml/pull/1348
• Model Registry Stack Component + MLFlow integration by @safoinme in https://github.com/zenml-io/zenml/pull/1309
• Fix broken docs URLs and add SDK docs url by @strickvl in https://github.com/zenml-io/zenml/pull/1349
• Fix label studio dataset delete command by @strickvl in https://github.com/zenml-io/zenml/pull/1377
• Add missing links to Quickstart by @strickvl in https://github.com/zenml-io/zenml/pull/1379
• Fix PyPI readme logo display by @strickvl in https://github.com/zenml-io/zenml/pull/1382
• Fixed broken migration for flavors by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1386
• Add debug mode flag for zenml info by @strickvl in https://github.com/zenml-io/zenml/pull/1374
• Update issue creation for bugs by @strickvl in https://github.com/zenml-io/zenml/pull/1387
• Integration sdk docs generated correctly now by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1389

Full Changelog: https://github.com/zenml-io/zenml/compare/0.34.0...0.35.1

- Python
Published by strickvl almost 3 years ago

https://github.com/zenml-io/zenml - 0.35.0 [YANKED, SEE 0.35.1]

This release is packed with big features as well as documentation updates and some bug fixes.

The 0.35.0 release puts models front and center in ZenML with the addition of the Model Registry abstraction and Stack Component. You can now register, version and manage models as first class citizens in ZenML. This is a major milestone for ZenML and we are excited to see what you build with it! The introduction of Model Registries greatly simplifies the journey that the model takes from training to deployment and extends the ZenML ecosystem to include model registry tools and libraries. The first Model Registry integration included in this release is MLFlow, with many more to come in the future.

This release also continues the deprecation of Secrets Managers and the introduction of Secret Stores. You now have the option of configuring the ZenML server to use AWS, GCP, Azure or Hashicorp Vault directly as a centralized secrets store back-end. This is meant to replace all Secrets Manager flavors which were previously used to store secrets using the same cloud services.

Note - Please be reminded that all Secrets Managers are now deprecated and will be removed in the near future. We recommend that you migrate all your secrets from the Secrets Manager stack components to the centralized secrets store by means of the included zenml secrets-manager secret migrate CLI command.

Last but not least, this release includes an updated Evidently integration that is compatible with the latest and greatest features from Evidently: reports and test suites. Check out the updated example to get a feel for the new features.

Breaking Changes

This release introduces a few breaking changes. Please update your code to reflect the changes below:

• the order of pipelines and runs in the post-execution results has been reversed. This means that the most recent pipeline and pipeline run can be accessed using the first index of the respective lists instead of the last index. This change was made to make the post-execution results more intuitive and to allow returning multi-page results in the future. This is a code snippet outlining the changes that you need to make in your post-execution code:

  ```python from zenml.postexecution import getpipelines, getunlistedruns

  pipelines = get_pipelines()

  instead of calling this to get the pipeline last created

  latest_pipeline = pipelines[-1]

  you now have to call this

  latest_pipeline = pipelines[0]

  and instead of calling this to get the latest run of a pipeine

  latestpipelinerun = latestpipeline.getruns()[-1]

  or

  latestpipelinerun = latest_pipeline.runs[-1]

  you now have to call this

  latestpipelinerun = latestpipeline.getruns()[0]

  or

  latestpipelinerun = latest_pipeline.runs[0]

  the same applies to the unlisted runs; instead of

  lastunlistedrun = getunlistedruns()[-1]

  you now have to call this

  lastunlistedrun = getunlistedruns()[0] ```

• if you were using the StepEnvironment to fetch the name of the active step in your step implementation, this name no longer reflects the name of the step function. Instead, it now reflects the name of the step used in the pipeline DAG, similar to what you would see in the ZenML dashboard when visualizing the pipeline. This is also implicitly reflected in the output of zenml model-deployer model CLI commands.

What's Changed

• Upgrade dev dependencies by @strickvl in https://github.com/zenml-io/zenml/pull/1334
• Add warning when attempting server connection without user permissions by @strickvl in https://github.com/zenml-io/zenml/pull/1314
• Keep CLI help text for zenml pipeline to a single line by @strickvl in https://github.com/zenml-io/zenml/pull/1338
• Rename page attributes by @schustmi in https://github.com/zenml-io/zenml/pull/1266
• Add missing docs for pipeline build by @schustmi in https://github.com/zenml-io/zenml/pull/1341
• Sagemaker orchestrator docstring and example update by @strickvl in https://github.com/zenml-io/zenml/pull/1350
• Fix secret create docs error for secret store by @strickvl in https://github.com/zenml-io/zenml/pull/1355
• Update README for test environment provisioning by @strickvl in https://github.com/zenml-io/zenml/pull/1336
• Disable name prefix matching when updating/deleting entities by @schustmi in https://github.com/zenml-io/zenml/pull/1345
• Add Kubeflow Pipeline UI Port to deprecated config by @safoinme in https://github.com/zenml-io/zenml/pull/1358
• Small clarifications for slack alerter by @htahir1 in https://github.com/zenml-io/zenml/pull/1365
• update Neptune integration for v1.0 compatibility by @AleksanderWWW in https://github.com/zenml-io/zenml/pull/1335
• Integrations conditional requirements by @safoinme in https://github.com/zenml-io/zenml/pull/1255
• Fix fetching versioned pipelines in post execution by @schustmi in https://github.com/zenml-io/zenml/pull/1363
• Load artifact store before loading artifact to register filesystem by @schustmi in https://github.com/zenml-io/zenml/pull/1367
• Remove poetry from CI by @schustmi in https://github.com/zenml-io/zenml/pull/1346
• Fix Sagemaker example readme by @strickvl in https://github.com/zenml-io/zenml/pull/1370
• Update evidently to include reports and tests by @wjayesh in https://github.com/zenml-io/zenml/pull/1283
• Fix neptune linting error on develop (and bump ruff) by @strickvl in https://github.com/zenml-io/zenml/pull/1372
• Add pydantic materializer by @htahir1 in https://github.com/zenml-io/zenml/pull/1371
• Registering GIFs added by @htahir1 in https://github.com/zenml-io/zenml/pull/1368
• Refresh CLI cheat sheet by @strickvl in https://github.com/zenml-io/zenml/pull/1347
• Add dependency resolution docs by @strickvl in https://github.com/zenml-io/zenml/pull/1337
• [BUGFIX] Fix error while using an existing SQL server with GCP ZenServer by @wjayesh in https://github.com/zenml-io/zenml/pull/1353
• Update step name assignment with the parameter name by @strickvl in https://github.com/zenml-io/zenml/pull/1310
• Copy huggingface data directory to local before loading in materializers by @TimovNiedek in https://github.com/zenml-io/zenml/pull/1351
• Update huggingface token classification example by @strickvl in https://github.com/zenml-io/zenml/pull/1369
• Use the most specialized materializer based on MRO by @schustmi in https://github.com/zenml-io/zenml/pull/1376
• Update Kserve to support 0.10.0 by @safoinme in https://github.com/zenml-io/zenml/pull/1373
• Add more examples to integration tests by @schustmi in https://github.com/zenml-io/zenml/pull/1245
• Fix order of runs and order of pipelines in post-execution by @stefannica in https://github.com/zenml-io/zenml/pull/1380
• Add Cloud Secrets Store back-ends by @stefannica in https://github.com/zenml-io/zenml/pull/1348
• Model Registry Stack Component + MLFlow integration by @safoinme in https://github.com/zenml-io/zenml/pull/1309
• Fix broken docs URLs and add SDK docs url by @strickvl in https://github.com/zenml-io/zenml/pull/1349
• Fix label studio dataset delete command by @strickvl in https://github.com/zenml-io/zenml/pull/1377
• Add missing links to Quickstart by @strickvl in https://github.com/zenml-io/zenml/pull/1379

Full Changelog: https://github.com/zenml-io/zenml/compare/0.34.0...0.35.0

- Python
Published by stefannica almost 3 years ago

https://github.com/zenml-io/zenml - 0.34.0

This release comes with major upgrades to the python library as well as the dashboard: - You can now store you secrets in a centralized way instead of having them tied to a secrets manager stack component. The secrets manager component is deprecated but will still work while we continue migrating all secrets manager flavors to be available as a backend to store centralized secrets. Check out the docs for more information. - Pipelines are now versioned: ZenML detects changes to your steps and structure of your pipelines and automatically creates new pipeline versions for you. - You can now build the required Docker images for your pipeline without actually running it with the zenml pipeline build command. This build can later be used to run the pipeline using the zenml pipeline run command or by passing it to pipeline.run() in python. - DockerSettings can now be specified separately for each step of your pipeline. This allows you to build specialized Docker images for each step in your pipeline if needed. - Metadata for runs and artifacts is now displayed in the dashboard: When viewing a pipeline run in the dashboard, click on a step or artifact to get useful metadata like the endpoint where your model is deployed or statistics about your training data.

What's Changed

• Move inbuilt Flavors into the Database by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1187
• Bump ruff version to 241 by @strickvl in https://github.com/zenml-io/zenml/pull/1289
• Add docs for run name templates by @schustmi in https://github.com/zenml-io/zenml/pull/1290
• Remove excess help text for zenml connect command by @strickvl in https://github.com/zenml-io/zenml/pull/1291
• Increase default service timeout to 60 by @safoinme in https://github.com/zenml-io/zenml/pull/1294
• increase timeout on quickstart example by @safoinme in https://github.com/zenml-io/zenml/pull/1296
• Add warning about MacOS not being supported by @strickvl in https://github.com/zenml-io/zenml/pull/1303
• Always include .zen in docker builds by @schustmi in https://github.com/zenml-io/zenml/pull/1292
• Add warning and docs update for label_studio installation issue by @strickvl in https://github.com/zenml-io/zenml/pull/1299
• Loosen version requirements for Great Expectations integration by @strickvl in https://github.com/zenml-io/zenml/pull/1302
• Change zenml init --template to optionally prompt and track email by @stefannica in https://github.com/zenml-io/zenml/pull/1298
• Update docs for Neptune experiment tracker integration by @strickvl in https://github.com/zenml-io/zenml/pull/1307
• Fix the destroy function on the stack recipe CLI by @wjayesh in https://github.com/zenml-io/zenml/pull/1301
• Add missing flavor migrations, make workspace ID optional by @schustmi in https://github.com/zenml-io/zenml/pull/1315
• Bump ruff 246 by @strickvl in https://github.com/zenml-io/zenml/pull/1316
• Remove tag from image name in gcp image builder by @schustmi in https://github.com/zenml-io/zenml/pull/1317
• Fix docs typo by @strickvl in https://github.com/zenml-io/zenml/pull/1318
• Fix step parameter merging by @schustmi in https://github.com/zenml-io/zenml/pull/1320
• Increase timeout for mlflow deployment example by @strickvl in https://github.com/zenml-io/zenml/pull/1308
• Workspace/projects fix for dashboard URL output when running pipeline by @strickvl in https://github.com/zenml-io/zenml/pull/1322
• Component Metadata Tracking Docs by @fa9r in https://github.com/zenml-io/zenml/pull/1319
• Add user environment zenml info command to CLI for debugging by @strickvl in https://github.com/zenml-io/zenml/pull/1312
• Added caching to quickstart by @htahir1 in https://github.com/zenml-io/zenml/pull/1321
• Renovation of the zenstore tests by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1275
• Fixes GCP docs typo by @luckri13 in https://github.com/zenml-io/zenml/pull/1327
• Remove deprecated CLI options by @strickvl in https://github.com/zenml-io/zenml/pull/1325
• GCP Image Builder network by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/1323
• improved flavor docs by @htahir1 in https://github.com/zenml-io/zenml/pull/1324
• Commands to register, build and run pipelines from the CLI by @schustmi in https://github.com/zenml-io/zenml/pull/1293
• Validate kserve model name by @strickvl in https://github.com/zenml-io/zenml/pull/1304
• Fix post-execution run sorting by @schustmi in https://github.com/zenml-io/zenml/pull/1332
• Secrets store with SQL back-end by @stefannica in https://github.com/zenml-io/zenml/pull/1313

New Contributors

• @luckri13 made their first contribution in https://github.com/zenml-io/zenml/pull/1327

Full Changelog: https://github.com/zenml-io/zenml/compare/0.33.0...0.34.0

- Python
Published by schustmi about 3 years ago

https://github.com/zenml-io/zenml - 0.33.0

This release introduces several big new features: - Docker images can now be built in GCP using the new Google Cloud Image Builder integration. Special shoutout to @gabrielmbmb for this amazing contribution! - Getting started with ZenML has been made even easier. You can now use one of the new ZenML Project Templates to initialize your ZenML repository with a basic project structure including a functional pipeline and basic scaffolding for materializers, parameters, and other classes you might want to extend. - Orchestrating runs on local Kubernetes has been made easier: The KubeFlow, Kubernetes, and Tekton orchestrators have been redesigned to be compatible with the K3D modular stack recipe that lets you spin up a local K3D Kubernetes cluster with a single line of code! - The MLflow integration has been updated and can now be used with the new MLflow 2.x! - You can now specify parameters and resources for your Seldon model deployers thanks to @d-lowl!

Furthermore, the internal project concept has been renamed to workspace to avoid confusion with the zenml-projects repository. This should only be relevant to you if you have custom applications that are interacting with the REST API of the ZenML server directly since all models sent from/to the server need to contain a workspace instead of a project now.

What's Changed

• Renaming Project to Workspace by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1254
• Integration tests for post execution functions by @fa9r in https://github.com/zenml-io/zenml/pull/1264
• Introduce post_execution.BaseView by @fa9r in https://github.com/zenml-io/zenml/pull/1238
• Make /cloud point to enterprise page by @strickvl in https://github.com/zenml-io/zenml/pull/1268
• update mlflow to verion greater than 2.0 by @safoinme in https://github.com/zenml-io/zenml/pull/1249
• Store run start time by @schustmi in https://github.com/zenml-io/zenml/pull/1271
• Relax pydantic dependency by @jlopezpena in https://github.com/zenml-io/zenml/pull/1262
• Fix failing filter on stacks by component id by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1276
• Track server version by @schustmi in https://github.com/zenml-io/zenml/pull/1265
• Bump ruff, drop autoflake, add darglint back by @strickvl in https://github.com/zenml-io/zenml/pull/1279
• Fixed startswith and endswith by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1278
• Fix workspace scoping on list_workspace_... endpoints again by @fa9r in https://github.com/zenml-io/zenml/pull/1284
• Custom Metadata Tracking by @fa9r in https://github.com/zenml-io/zenml/pull/1151
• Bug: local ZenML server ignores ip-address CLI argument by @stefannica in https://github.com/zenml-io/zenml/pull/1282
• Configure the zenml-server docker image and helm chart to run as non-privileged user by @stefannica in https://github.com/zenml-io/zenml/pull/1273
• GCP Image Builder by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/1270
• Disentangle K3D code from ZenML by @safoinme in https://github.com/zenml-io/zenml/pull/1185
• Rework params / artifact docs by @strickvl in https://github.com/zenml-io/zenml/pull/1277
• Always add active user to analytics by @stefannica in https://github.com/zenml-io/zenml/pull/1286
• Fix step and pipeline run metadata in LineageGraph by @fa9r in https://github.com/zenml-io/zenml/pull/1288
• add validator to endpoint url to replace hostname with k3d or docker … by @safoinme in https://github.com/zenml-io/zenml/pull/1189
• Add option to use project templates to initialize a repository by @stefannica in https://github.com/zenml-io/zenml/pull/1287
• Add example for Hyperparameter Tuning with ZenML by @nitay93 in https://github.com/zenml-io/zenml/pull/1206
• Add seldon deployment predictor parameters and resource requirements by @d-lowl in https://github.com/zenml-io/zenml/pull/1280

New Contributors

• @jlopezpena made their first contribution in https://github.com/zenml-io/zenml/pull/1262
• @nitay93 made their first contribution in https://github.com/zenml-io/zenml/pull/1206
• @d-lowl made their first contribution in https://github.com/zenml-io/zenml/pull/1280

Full Changelog: https://github.com/zenml-io/zenml/compare/0.32.1...0.33.0

- Python
Published by fa9r about 3 years ago

https://github.com/zenml-io/zenml - 0.32.1

This release resolves several minor bugs and inconveniences introduced during the filtering and pagination overhaul in the last release. Additionally, the release includes new integration tests to improve future stability.

What's Changed

• Update and improve docker and helm deployment docs by @stefannica in https://github.com/zenml-io/zenml/pull/1246
• Fixed broken link returned form pipeline runs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1257
• Fix project scoping on list_project_... endpoints by @fa9r in https://github.com/zenml-io/zenml/pull/1256
• Orchestrator tests by @schustmi in https://github.com/zenml-io/zenml/pull/1258
• Add integration tests for lineage graph creation by @fa9r in https://github.com/zenml-io/zenml/pull/1253
• Always instantiate a zen_store before startup. by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1261
• Fix post execution run fetching by @schustmi in https://github.com/zenml-io/zenml/pull/1263
• Implemented the option to choose between ascending and descending on list calls by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1260
• Fix logger warning message by @strickvl in https://github.com/zenml-io/zenml/pull/1267

Full Changelog: https://github.com/zenml-io/zenml/compare/0.32.0...0.32.1

- Python
Published by fa9r about 3 years ago

https://github.com/zenml-io/zenml - 0.32.0

Release 0.32.0 introduces two big new features: * A new stack component, the "image builder", with a corresponding new Kaniko integration. * Logic for filtering and pagination of list requests.

Image Builder Abstraction and Kaniko Integration

ZenML stacks can now contain an image builder as additional optional stack component. The image builder defines how the Docker images are built that are required by many of the other stack components such as Airflow or Kubeflow. Previously, all image building was handled implicitly by ZenML using local Docker, which has now been refactored into the "local" image builder flavor. As an alternative, you can now install the new "kaniko" integration to build your images in Kubernetes using Kaniko.

Filtering and Pagination

All list commands in ZenML are now capable of advanced filtering such as zenml stack list --created="gt:22-12-04 17:00:00" --name contains:def.

Additionally, list commands now return pages of results, which significantly improves performance for power ZenML users that have already created many runs or other entities.

What's Changed

• UserResponseModel contains roles, block recursion properly on more Models, reduce amount of Runs on a PipelineResponseModel by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1180
• Bump ruff version by @strickvl in https://github.com/zenml-io/zenml/pull/1232
• Zenfile becomes project by @strickvl in https://github.com/zenml-io/zenml/pull/1235
• Fix class resolution in notebooks under Python>=3.10 by @fa9r in https://github.com/zenml-io/zenml/pull/1234
• Fix Sagemaker README images & pipeline addition by @strickvl in https://github.com/zenml-io/zenml/pull/1239
• Step/Pipeline configuration tests by @schustmi in https://github.com/zenml-io/zenml/pull/1233
• Removed gRPC from diagrams by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1242
• Fix MLflow tracking example bug for Macs by @strickvl in https://github.com/zenml-io/zenml/pull/1237
• Fix copy function to copyfile in registered filesystem by @safoinme in https://github.com/zenml-io/zenml/pull/1243
• Image builder abstraction by @schustmi in https://github.com/zenml-io/zenml/pull/1198
• Add support for modular recipes to the recipe CLI by @wjayesh in https://github.com/zenml-io/zenml/pull/1247
• Add docs on upgrading and troubleshooting zenml server by @wjayesh in https://github.com/zenml-io/zenml/pull/1244
• Improve Seldon and Kserve Docs by @wjayesh in https://github.com/zenml-io/zenml/pull/1236
• Add Pagination to all List commands by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1113

Full Changelog: https://github.com/zenml-io/zenml/compare/0.31.1...0.32.0

- Python
Published by schustmi about 3 years ago

https://github.com/zenml-io/zenml - 0.31.1

This release includes several bug fixes and new additions under the hood such as testing for various internal utility functions. This should help keep ZenML more stable over time. Additionally, we added the ability to customize default materializers for custom artifact stores, and the ability to track system info and the Python version of pipeline runs (both where pipelines are initially executed as well as wherever they eventually run). We added better support for pipeline scheduling (particularly from within the CLI) and tracking of the source code of steps. The release also includes the addition of information about whether the pipeline is running on a stack created by the active user, and the ability to specify Kubernetes container resource requests and limits. Finally, we addressed issues with caching such that caching is enabled for steps that have explicit enable_cache=True specified (even when pipelines have it turned off).

What's Changed

• Test for enum_utils by @strickvl in https://github.com/zenml-io/zenml/pull/1209
• Add missing space in Azure docs by @strickvl in https://github.com/zenml-io/zenml/pull/1218
• Test for dashboard_utils by @strickvl in https://github.com/zenml-io/zenml/pull/1202
• Cloud version gets love by @htahir1 in https://github.com/zenml-io/zenml/pull/1219
• ZenFiles to ZenML Projects by @htahir1 in https://github.com/zenml-io/zenml/pull/1220
• Track System Info and Python Version of Pipeline Runs by @fa9r in https://github.com/zenml-io/zenml/pull/1215
• Tests for pydantic_utils by @strickvl in https://github.com/zenml-io/zenml/pull/1207
• Customizing Default Materializers for Custom Artifact Stores by @safoinme in https://github.com/zenml-io/zenml/pull/1224
• Test typed_model utilities by @strickvl in https://github.com/zenml-io/zenml/pull/1208
• Enable Airflow<2.4 by @schustmi in https://github.com/zenml-io/zenml/pull/1222
• Fix alembic_start migration if tables exist by @fa9r in https://github.com/zenml-io/zenml/pull/1214
• Tests for network_utils by @strickvl in https://github.com/zenml-io/zenml/pull/1201
• Tests for io_utils and removal of duplicate code by @strickvl in https://github.com/zenml-io/zenml/pull/1199
• Use ruff to replace our linting suite by @strickvl in https://github.com/zenml-io/zenml/pull/1211
• Test materializer utilities by @safoinme in https://github.com/zenml-io/zenml/pull/1221
• Add information whether pipeline is running on a stack created by the active user by @schustmi in https://github.com/zenml-io/zenml/pull/1229
• Test daemon util functions by @strickvl in https://github.com/zenml-io/zenml/pull/1210
• Test filesync_model utils by @strickvl in https://github.com/zenml-io/zenml/pull/1230
• Track Source Code of Steps by @fa9r in https://github.com/zenml-io/zenml/pull/1216
• Track Pipeline Run Schedules by @fa9r in https://github.com/zenml-io/zenml/pull/1227
• Tests for analytics by @bcdurak in https://github.com/zenml-io/zenml/pull/1228
• Allow specifying Kubernetes container resource requests and limits by @schustmi in https://github.com/zenml-io/zenml/pull/1223
• Enable cache for all steps that have explicit enable_cache=True by @fa9r in https://github.com/zenml-io/zenml/pull/1217
• Make shared stacks visible again by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1225

Full Changelog: https://github.com/zenml-io/zenml/compare/0.31.0...0.31.1

- Python
Published by fa9r about 3 years ago

https://github.com/zenml-io/zenml - 0.31.0

The highlights of this release are:

• our Materializers have been redesigned to be more flexible and easier to use
• we have added a new integration test framework
• the SageMaker orchestrator has been added to our list of supported orchestrators
• pipeline runs and artifacts can now be deleted from the ZenML database via the CLI or the Client API
• some integrations have been updated to a more recent version: Kubeflow, Seldon Core and Tekton

This release also includes a few bug fixes and other minor improvements to existing features.

What's Changed

• Fix installation instructions in readme and docs by @schustmi in https://github.com/zenml-io/zenml/pull/1167
• Fix broken TOC for scheduling docs by @strickvl in https://github.com/zenml-io/zenml/pull/1169
• Ensure model string fields have a max length by @strickvl in https://github.com/zenml-io/zenml/pull/1136
• Integration test framework by @stefannica in https://github.com/zenml-io/zenml/pull/1099
• Check if all ZenML server dependencies are installed for local zenml deployment using zenml up by @dnth in https://github.com/zenml-io/zenml/pull/1144
• Persist the server ID in the database by @stefannica in https://github.com/zenml-io/zenml/pull/1173
• Tiny docs improvements by @strickvl in https://github.com/zenml-io/zenml/pull/1179
• Changing some interactions with analytics fields by @bcdurak in https://github.com/zenml-io/zenml/pull/1174
• Fix PyTorchDataLoaderMaterializer for older torch versions by @fa9r in https://github.com/zenml-io/zenml/pull/1178
• Redesign Materializers by @fa9r in https://github.com/zenml-io/zenml/pull/1154
• Fixing the error messages when fetching entities by @bcdurak in https://github.com/zenml-io/zenml/pull/1171
• Moved the activeuser property onto the client, implemented getmyself as zenstore method by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1161
• Bugfix/bump evidently version by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1183
• Alembic migration to update size of flavor config schema by @fa9r in https://github.com/zenml-io/zenml/pull/1181
• Deleting pipeline runs and artifacts by @fa9r in https://github.com/zenml-io/zenml/pull/1164
• Signer email checked before setting in google cloud scheduler by @htahir1 in https://github.com/zenml-io/zenml/pull/1184
• Fix zenml helm chart to not leak analytics events by @stefannica in https://github.com/zenml-io/zenml/pull/1190
• Tests for dict_utils by @strickvl in https://github.com/zenml-io/zenml/pull/1196
• Adding exception tracking to zeml init by @bcdurak in https://github.com/zenml-io/zenml/pull/1192
• Prevent crashes during Airflow server forking on MacOS by @schustmi in https://github.com/zenml-io/zenml/pull/1186
• add alpha as server deployment type by @wjayesh in https://github.com/zenml-io/zenml/pull/1197
• Bugfix for custom flavor registration by @bcdurak in https://github.com/zenml-io/zenml/pull/1195
• Tests for uuid_utils by @strickvl in https://github.com/zenml-io/zenml/pull/1200
• Sagemaker orchestrator integration by @strickvl in https://github.com/zenml-io/zenml/pull/1177
• Fix Pandas Materializer Index by @safoinme in https://github.com/zenml-io/zenml/pull/1193
• Add support for deploying custom stack recipes using the ZenML CLI by @wjayesh in https://github.com/zenml-io/zenml/pull/1188
• Add cloud CI environments by @stefannica in https://github.com/zenml-io/zenml/pull/1176
• Fix project scoping for artifact list through ZenServer by @fa9r in https://github.com/zenml-io/zenml/pull/1203

- Python
Published by stefannica about 3 years ago

https://github.com/zenml-io/zenml - 0.30.0

In this release, ZenML finally adds Mac M1 support, Python 3.10 support and much greater flexibility and configurability under the hood by deprecating some large dependencies like ml-pipelines-sdk.

Scheduling

Based on some community feedback around scheduling, this release comes with improved docs concerning scheduling in general. Additionally, the Vertex AI orchestrator now also supports scheduling.

Slimmer Dependencies

By removing dependencies on some of the packages that ZenML was built on, this version of ZenML is slimmer, faster and more configurable than ever. This also finally makes ZenML run natively on Macs with M1 processors without the need for Rosetta. This also finally enables ZenML to run on Python 3.10.

Breaking Changes

• The removal of ml-pipelines-sdk and tfx leads to some larger changes in the database that is tracking your pipeline runs and artifacts. Note: There is an automatic migration to upgrade this automatically, However, please note that downgrading back down to 0.23.0 is not supported.
• The CLI commands to export and import pipeline runs have been deprecated. Namely: zenml pipeline runs export and zenml pipeline runs import These commands were meant for migrating from zenml<0.20.0 to 0.20.0<=zenml<0.30.0.
• The azure-ml integration dependency on azureml-core has been upgraded from 1.42 to 1.48

What's Changed

• Remove stack extra from installation, enable re-running the quickstart by @schustmi in https://github.com/zenml-io/zenml/pull/1133
• Secrets manager support to experiment trackers docs by @safoinme in https://github.com/zenml-io/zenml/pull/1137
• Updating the README files of our examples by @bcdurak in https://github.com/zenml-io/zenml/pull/1128
• Prevent running with local ZenStore and remote code execution by @schustmi in https://github.com/zenml-io/zenml/pull/1134
• Remove ml-pipelines-sdk dependency by @schustmi in https://github.com/zenml-io/zenml/pull/1103
• Fix Huggingface dataset materializer by @safoinme in https://github.com/zenml-io/zenml/pull/1142
• Disallow alembic downgrades for 0.30.0 release by @fa9r in https://github.com/zenml-io/zenml/pull/1140
• Fix Client flavor-related methods by @schustmi in https://github.com/zenml-io/zenml/pull/1153
• Replace User Password with Token in docker images by @safoinme in https://github.com/zenml-io/zenml/pull/1147
• Remove zenml pipeline runs export / import CLI commands by @fa9r in https://github.com/zenml-io/zenml/pull/1150
• Context manager to track events by @bcdurak in https://github.com/zenml-io/zenml/pull/1149
• Made explicit is not None calls to allow for empty pwd again by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1159
• Add Neptune exp tracker into flavors table by @dnth in https://github.com/zenml-io/zenml/pull/1156
• Fix step operators by @schustmi in https://github.com/zenml-io/zenml/pull/1155
• Display correct name when updating a stack component by @schustmi in https://github.com/zenml-io/zenml/pull/1160
• Update mysql database creation by @schustmi in https://github.com/zenml-io/zenml/pull/1152
• Adding component conditions to experiment tracker examples and adding to the environmental variable docs by @bcdurak in https://github.com/zenml-io/zenml/pull/1162
• Increase dependency range for protobuf by @schustmi in https://github.com/zenml-io/zenml/pull/1163
• Scheduling documentation by @strickvl in https://github.com/zenml-io/zenml/pull/1158
• Adding scheduling for Vertex Pipelines by @htahir1 in https://github.com/zenml-io/zenml/pull/1148
• Fix alembic migration for sqlite<3.25 by @fa9r in https://github.com/zenml-io/zenml/pull/1165
• Fix pandas Series materializer by @jordandelbar in https://github.com/zenml-io/zenml/pull/1146

New Contributors

• @jordandelbar made their first contribution in https://github.com/zenml-io/zenml/pull/1146

- Python
Published by AlexejPenner about 3 years ago

https://github.com/zenml-io/zenml - 0.30.0rc3

This pre-release builds on top of 0.30.0rc2.

Major changes: - Improved database migration from previous versions - Fixes for step operators

Note: Installing and using this will migrate your ZenML metadata database tables without any way of downgrading to previous releases. Use at your own risk.

- Python
Published by schustmi about 3 years ago

https://github.com/zenml-io/zenml - 0.30.0rc2

This pre-release builds on top of 0.30.0rc1.

Major changes: - Docker images are now built for python 3.10 - The database tables for pipeline and step runs are cleaned up.

Note: Installing and using this will migrate your ZenML metadata database tables without any way of downgrading to previous releases. Use at your own risk.

- Python
Published by schustmi about 3 years ago

https://github.com/zenml-io/zenml - 0.30.0rc1

This pre-release builds on top of 0.30.0rc0, specifically with a few bug fixes including fixing the quickstart

Note: Installing and using this will migrate your ZenML metadata database tables without any way of downgrading to previous releases. Use at your own risk.

- Python
Published by schustmi about 3 years ago

https://github.com/zenml-io/zenml - 0.30.0rc0

This pre-release removes the ml-pipeline-sdk dependency and adds support for Python 3.10 as well as M1 macs.

Note: Installing and using this will migrate your ZenML metadata database tables without any way of downgrading to previous releases. Use at your own risk.

- Python
Published by schustmi about 3 years ago

https://github.com/zenml-io/zenml - 0.23.0

This release comes with a brand-new Neptune integration to track your ML experiments

as well as lots of performance improvements!

Neptune integration

The new Neptune integration includes a Neptune experiment tracker component that allows you to track your machine learning experiments using Neptune.

Performance Optimization

The 0.20.0 release introduced our new server but brought with it a few performance and scalability issues. Since then we've made many improvements to it, and this release is the final and biggest boost in performance. We reduced the amount of server calls needed for almost all CLI commands and greatly improved the speed of the dashboard as well.

PyArrow dependency removal

We've removed PyArrow as a dependency of the zenml python package. As a consequence of that, our NumPy and Pandas materializer no longer read and write their artifacts using PyArrow but instead use native formats instead. If you still want to use PyArrow to serialize your NumPy arrays and Pandas dataframes, you'll need to install it manually like this: pip install pyarrow

In future releases we'll get rid of other unnecessary dependencies to further slim down the zenml package.

Breaking Changes

The following changes introduces with this release mey require some manual intervention to update your current installations:

• If your code calls some methods of our Client class, it might need to be updated to the new model classes introduced by the performance optimization changes explained above
• The CLI command to remove an attribute from a stack component now takes no more dashes in front of the attribute names: zenml stack-component remove-attribute <COMPONENT_NAME> <ATTRIBUTE_NAME>
• If you're using a custom stack component and have overridden the cleanup_step_run method, you'll need to update the method signature to include a step_failed parameter.

What's Changed

• Docs regarding roles and permissions by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1081
• Add global config dir to zenml status by @schustmi in https://github.com/zenml-io/zenml/pull/1084
• Remove source pins and ignore source pins during step spec comparisons by @schustmi in https://github.com/zenml-io/zenml/pull/1083
• Docs/links for roles permissions by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1091
• Bugfix/eng 1485 fix api docs build by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1089
• fix bento builder step parameters to match bentoml by @safoinme in https://github.com/zenml-io/zenml/pull/1096
• Add bentoctl to BentoML docs and example by @safoinme in https://github.com/zenml-io/zenml/pull/1094
• Fix BaseParameters sample code in docs by @jcarlosgarcia in https://github.com/zenml-io/zenml/pull/1098
• zenml logs defaults to active stack without nameorid by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1101
• Fixed evidently docs by @htahir1 in https://github.com/zenml-io/zenml/pull/1111
• Update sagemaker default instance type by @schustmi in https://github.com/zenml-io/zenml/pull/1112
• The ultimate optimization for performance by @bcdurak in https://github.com/zenml-io/zenml/pull/1077
• Update stack exporting and importing by @schustmi in https://github.com/zenml-io/zenml/pull/1114
• Fix readme by @schustmi in https://github.com/zenml-io/zenml/pull/1116
• Remove Pyarrow dependency by @safoinme in https://github.com/zenml-io/zenml/pull/1109
• Bugfix for listing the runs filtered by a name by @bcdurak in https://github.com/zenml-io/zenml/pull/1118
• Neptune.ai integration by @AleksanderWWW in https://github.com/zenml-io/zenml/pull/1082
• Add YouTube video explaining Stack Components Settings vs Config by @dnth in https://github.com/zenml-io/zenml/pull/1120
• Add failed Status to component when step fails by @safoinme in https://github.com/zenml-io/zenml/pull/1115
• Add architecture diagrams to docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1119
• Remove local orchestrator restriction from step operator docs by @schustmi in https://github.com/zenml-io/zenml/pull/1122
• Validate Stack Before Provision by @safoinme in https://github.com/zenml-io/zenml/pull/1110
• Bugfix/fix endpoints for dashboard development by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1125
• Skip kubeflow UI daemon provisioning if a hostname is configured by @schustmi in https://github.com/zenml-io/zenml/pull/1126
• Update Neptune Example by @safoinme in https://github.com/zenml-io/zenml/pull/1124
• Add debugging guide to docs by @dnth in https://github.com/zenml-io/zenml/pull/1097
• Fix stack component attribute removal CLI command by @schustmi in https://github.com/zenml-io/zenml/pull/1127
• Improving error messages when fetching entitites by @bcdurak in https://github.com/zenml-io/zenml/pull/1117
• Introduce username and password to kubeflow for more native multi-tenant support by @htahir1 in https://github.com/zenml-io/zenml/pull/1123
• Add support for Label Studio OCR config generation by @shivalikasingh95 in https://github.com/zenml-io/zenml/pull/1062
• Misc doc updates by @schustmi in https://github.com/zenml-io/zenml/pull/1131
• Fix Neptune run cleanup by @safoinme in https://github.com/zenml-io/zenml/pull/1130

New Contributors

• @jcarlosgarcia made their first contribution in https://github.com/zenml-io/zenml/pull/1098
• @AleksanderWWW made their first contribution in https://github.com/zenml-io/zenml/pull/1082
• @shivalikasingh95 made their first contribution in https://github.com/zenml-io/zenml/pull/1062

Full Changelog: https://github.com/zenml-io/zenml/compare/0.22.0...0.23.0

- Python
Published by schustmi about 3 years ago

https://github.com/zenml-io/zenml - 0.22.0

This release comes with a new BentoML integration as well as a reworked Airflow orchestrator. Additionally, it greatly improves the server performance as well as other small fixes and updates to our docs!

BentoML integration

The new BentoML integration includes a BentoML model deployer component that allows you to deploy your models from any of the major machine learning frameworks on your local machine.

See example here.

Airflow orchestrator v2

The previous Airflow orchestrator was limited to running locally and had many additional unpleasant constraints that made it hard to work with. This release includes a completely rewritten, new version of the Airflow orchestrator that now relies on Docker images to run your pipelines and works both locally and with remote Airflow deployments.

See what changed in this video and check out the brand-new example here.

Notable bugfixes

• Further improvements to the synchronization that transfers pipeline run information from the MLMD database to the ZenML Server.
• The ZenML Label Studio integration can now be used with non-local (i.e. deployed) instances. For more information see the Label Studiodocs.
• The Spark example is fixed and now works again end-to-end.

Breaking Changes

The following changes introduces with this release mey require some manual intervention to update your current installations:

• the Airflow orchestrator now requires a newer version of Airflow (run zenml integration install airflow to upgrade) and Docker installed to work.

What's Changed

• Fix bug when running non-local annotator instance. by @sheikhomar in https://github.com/zenml-io/zenml/pull/1045
• Introduce Permissions, Link Permissions to Roles, Restrict Access to endpoints based on Permission by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1007
• Fix copy-pasted log message for annotator by @strickvl in https://github.com/zenml-io/zenml/pull/1049
• Add warning message for client server version mismatch by @schustmi in https://github.com/zenml-io/zenml/pull/1047
• Fix path to ingress values in ZenServer recipes by @wjayesh in https://github.com/zenml-io/zenml/pull/1053
• Prevent deletion/update of default entities by @stefannica in https://github.com/zenml-io/zenml/pull/1046
• Fix Publish API docs workflow by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1054
• Fix multiple alembic heads warning by @fa9r in https://github.com/zenml-io/zenml/pull/1051
• Fix Null Step Configuration/Parameters Error by @fa9r in https://github.com/zenml-io/zenml/pull/1050
• Fix role permission migration by @schustmi in https://github.com/zenml-io/zenml/pull/1056
• Made role assignment/revokation possible through zen_server by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1059
• Bugfix/make role assignment work with enum by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1063
• Manually set scoped for each endpoint by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1064
• Add run args to local docker orchestrator settings by @schustmi in https://github.com/zenml-io/zenml/pull/1060
• Docker ZenML deployment improvements and docs by @stefannica in https://github.com/zenml-io/zenml/pull/1061
• Bugfix Mlflow service cleanup configuration by @safoinme in https://github.com/zenml-io/zenml/pull/1067
• Rename DB Tables and Fix Foreign Keys by @fa9r in https://github.com/zenml-io/zenml/pull/1058
• Paginate secrets in AWSSecretsManager by @chiragjn in https://github.com/zenml-io/zenml/pull/1057
• Add explicit dashboard docs by @strickvl in https://github.com/zenml-io/zenml/pull/1052
• Added GA and Gitlab to envs by @htahir1 in https://github.com/zenml-io/zenml/pull/1068
• Add Inference Server Predictor to KServe and Seldon Docs by @safoinme in https://github.com/zenml-io/zenml/pull/1048
• Rename project table to workspace by @fa9r in https://github.com/zenml-io/zenml/pull/1073
• Airflow orchestrator v2 by @schustmi in https://github.com/zenml-io/zenml/pull/1042
• Add getorcreate_run() ZenStore method by @fa9r in https://github.com/zenml-io/zenml/pull/1070
• Fix the flaky fileio tests by @schustmi in https://github.com/zenml-io/zenml/pull/1072
• BentoML Deployer Integration by @safoinme in https://github.com/zenml-io/zenml/pull/1044
• Sync Speedup by @fa9r in https://github.com/zenml-io/zenml/pull/1055
• Fixed broken links in docs and examples. by @dnth in https://github.com/zenml-io/zenml/pull/1076
• Make additional stack component config options available as a setting by @schustmi in https://github.com/zenml-io/zenml/pull/1069
• Rename step_run_artifact table to step_run_input_artifact by @fa9r in https://github.com/zenml-io/zenml/pull/1075
• Update Spark Example to ZenML post 0.20.0 by @safoinme in https://github.com/zenml-io/zenml/pull/1071
• Always set caching to false for all Kubeflow based orchestrators by @schustmi in https://github.com/zenml-io/zenml/pull/1079
• Feature/eng 1402 consolidate stack sharing by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1036

New Contributors

• @sheikhomar made their first contribution in https://github.com/zenml-io/zenml/pull/1045
• @chiragjn made their first contribution in https://github.com/zenml-io/zenml/pull/1057

Full Changelog: https://github.com/zenml-io/zenml/compare/0.21.1...0.22.0

- Python
Published by schustmi over 3 years ago

https://github.com/zenml-io/zenml - 0.21.1

This is an ad-hoc release to fix some bugs introduced the 0.21.0 release that made the local ZenML dashboard unusable.

What's Changed

• Include latest (not oldest) three runs in HydratedPipelineModel by @schustmi in https://github.com/zenml-io/zenml/pull/1039
• Update docs to use pip install [server] by @strickvl in https://github.com/zenml-io/zenml/pull/1037
• Docs fix for Deepchecks by @strickvl in https://github.com/zenml-io/zenml/pull/1040
• Fix the pipeline run sync on sqlite and the --blocking zenml server deployment by @stefannica in https://github.com/zenml-io/zenml/pull/1041

Full Changelog: https://github.com/zenml-io/zenml/compare/0.21.0...0.21.1

- Python
Published by stefannica over 3 years ago

https://github.com/zenml-io/zenml - 0.21.0

This release primarily fixes a number of bugs that were introduced as part of the 0.20.0 ZenServer release. These significantly improve the stability when using ZenML with the ZenML Server.

Notable fixes include:

• Improved the synchronization that transfers pipeline run information from the MLMD database to the ZenML Server. This helps fix a number of issues with missing steps in the post-execution workflow, model deployment steps and other issues.
• The Label Studio example is fixed and now works again end-to-end.
• The ZenML Label Studio integration can now be used with non-local (i.e. deployed) instances. For more information see the Label Studiodocs.

New features and other improvements:

• ZenML now uses alembic for automated database migrations. The migrations happen automatically after every ZenML update.
• New zenml pipeline runs export / import / migrate CLI commands are now available to export, import and migrate pipeline runs from older, pre-0.20.0 versions of ZenML. The ZenML server now also automatically picks up older pipeline runs that have been logged in the metadata store by ZenML prior to 0.20.0.
• An MLMD gRPC service can now be deployed with the ZenML Helm chart to act as a proxy between clients, orchestrators and the MySQL database. This significantly reduces the time it takes to run pipelines locally.
• You can now specify affinity and tolerations and node selectors to all Kubernetes based orchestrators with the new Kubernetes Pod settings feature.

Breaking Changes

The following changes introduces with this release mey require some manual intervention to update your current installations:

• the zenml server helm chart values.yaml file has been restructured to make it easier to configure and to clearly distinguish between the zenml server component and the newly introduced gRPC service component. Please update your values.yaml copies accordingly.
• the Azure integration dependency versions have been updated. Please run zenml integration install azure to update your current installation, if you're using Azure.

What's Changed

• Implement automatic alembic migration by @AlexejPenner in https://github.com/zenml-io/zenml/pull/990
• Fix GCP Artifact Store listdir empty path by @safoinme in https://github.com/zenml-io/zenml/pull/998
• Add flavors mini-video to docs by @strickvl in https://github.com/zenml-io/zenml/pull/999
• Remove the Client() warning when used inside a step by @stefannica in https://github.com/zenml-io/zenml/pull/1000
• Fix broken links caused by updated by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1002
• Fix FileNotFoundError with remote path in HuggingFace Dataset materializer by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/995
• Add zenml pipeline runs export / import / migrate CLI commands by @fa9r in https://github.com/zenml-io/zenml/pull/977
• Log message when activating a stack as part of registration by @schustmi in https://github.com/zenml-io/zenml/pull/1005
• Minor fixes in Migration to 0.20.0 documentation by @alvarobartt in https://github.com/zenml-io/zenml/pull/1009
• Doc updates by @htahir1 in https://github.com/zenml-io/zenml/pull/1006
• Fixing broken links in docs by @dnth in https://github.com/zenml-io/zenml/pull/1018
• Label Studio example fix by @strickvl in https://github.com/zenml-io/zenml/pull/1021
• Docs for using CUDA-enabled docker images by @strickvl in https://github.com/zenml-io/zenml/pull/1010
• Add social media heading on docs page by @dnth in https://github.com/zenml-io/zenml/pull/1020
• Add executing custom command for getting requirements by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/1012
• Delay user instruction in dockerfile generation by @schustmi in https://github.com/zenml-io/zenml/pull/1004
• Update link checker configs for faster, more accurate checks by @dnth in https://github.com/zenml-io/zenml/pull/1022
• Add pip install zenml[server] to relevant examples by @dnth in https://github.com/zenml-io/zenml/pull/1027
• Add Tolerations and NodeAffinity to Kubernetes executor by @wefner in https://github.com/zenml-io/zenml/pull/994
• Support pydantic subclasses in BaseParameter attributes by @schustmi in https://github.com/zenml-io/zenml/pull/1023
• Unify run names across orchestrators by @schustmi in https://github.com/zenml-io/zenml/pull/1025
• Add gRPC metadata service to the ZenML helm chart by @stefannica in https://github.com/zenml-io/zenml/pull/1026
• Make the MLMD pipeline run information transfer synchronous by @stefannica in https://github.com/zenml-io/zenml/pull/1032
• Add console spinner back by @strickvl in https://github.com/zenml-io/zenml/pull/1034
• Fix Azure CLI auth problem by @wjayesh in https://github.com/zenml-io/zenml/pull/1035
• Allow non-local Label Studio instances for annotation by @strickvl in https://github.com/zenml-io/zenml/pull/1033
• Before deleting the global zen_server files, spin it down by @AlexejPenner in https://github.com/zenml-io/zenml/pull/1029
• Adding zenserver integration to stack recipe CLI by @wjayesh in https://github.com/zenml-io/zenml/pull/1017
• Add support for Azure ZenServer by @wjayesh in https://github.com/zenml-io/zenml/pull/1024
• Kubernetes Pod settings by @schustmi in https://github.com/zenml-io/zenml/pull/1008

New Contributors

• @alvarobartt made their first contribution in https://github.com/zenml-io/zenml/pull/1009
• @wefner made their first contribution in https://github.com/zenml-io/zenml/pull/994

Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.5...0.21.0

- Python
Published by stefannica over 3 years ago

https://github.com/zenml-io/zenml - 0.20.5

ZenML 0.20.5 fixes another series of minor bugs, significantly improves the performance of the CLI, and adds an option to specify APT packages in Docker images.

What's Changed

• Fix accessing local zen store and artifact store in containers by @stefannica in https://github.com/zenml-io/zenml/pull/976
• K3d local registry pod spec updated by @wjayesh in https://github.com/zenml-io/zenml/pull/972
• Update readme page by @dnth in https://github.com/zenml-io/zenml/pull/985
• Remove beam dependency by @schustmi in https://github.com/zenml-io/zenml/pull/986
• Fix error message when registering secret without secrets manager by @schustmi in https://github.com/zenml-io/zenml/pull/981
• Update cheat sheet up to zenml==0.20.4 by @dnth in https://github.com/zenml-io/zenml/pull/987
• Example fixes (part 2) by @strickvl in https://github.com/zenml-io/zenml/pull/971
• Allow duplicate step classes inside a pipeline by @schustmi in https://github.com/zenml-io/zenml/pull/989
• Include deployment in azureml docker build by @schustmi in https://github.com/zenml-io/zenml/pull/984
• Automatically open browser upon zenml up command by @dnth in https://github.com/zenml-io/zenml/pull/978
• Add a just_mine flag for zenml stack list by @strickvl in https://github.com/zenml-io/zenml/pull/979
• Add option to specify apt packages by @schustmi in https://github.com/zenml-io/zenml/pull/982
• Replace old flavor references, fix the windows local ZenML server and other fixes by @stefannica in https://github.com/zenml-io/zenml/pull/988
• Improve docker and k8s detection by @schustmi in https://github.com/zenml-io/zenml/pull/991
• Update GH actions example by @schustmi in https://github.com/zenml-io/zenml/pull/993
• Update MissingStepParameterError exception message by @gabrielmbmb in https://github.com/zenml-io/zenml/pull/996
• Seprated code docs into core and integration docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/983
• Add docs/mkdocstrings_helper.py to format script sources by @fa9r in https://github.com/zenml-io/zenml/pull/997
• Further CLI optimization by @bcdurak in https://github.com/zenml-io/zenml/pull/992

Full Changelog: https://github.com/zenml-io/zenml/compare/0.20.4...0.20.5

- Python
Published by fa9r over 3 years ago