theodolite - v0.9.0

This is the release of Theodolite 0.9.0. The most important new features are: * SLO thresholds can now be defined relatively to tested load intensities and provisioned resources. Even arbitrary complex mathematical expressions with load and resources as variables are possible. * A new generic patcher allows to patch arbitrary Kubernetes resources (including CRDs) by specifying the JSON path ob the field to be patched. * The configuration and implementations of the Theodolite stream processing benchmarks are further aligned to allow for a better comparison of different frameworks. * The Theodolite benchmark definitions for the stream processing benchmarks are extended to include options for vertical scaling and provide a dropped records SLO. They also define topics now using Strimzi. * Configurability of the Strimzi Kafka deployment is improved.

- Java
Published by SoerenHenning over 2 years ago

theodolite - v0.8.6

This is the release of Theodolite 0.8.6. It introduces the following changes: * Theodolite's Hazelcast Jet benchmarks can be now installed via Helm * A bug is fixed regarding the restriction search * Consumer group filters are added to all benchmarks (required if used with Strimzi) * Benchmarks are assigned Kubernetes labels * Theodolite's CPU and memory resources are now configurable via Helm

- Java
Published by SoerenHenning almost 3 years ago

theodolite - v0.8.5

This is the release of Theodolite 0.8.5. It introduces support for setting nodeSelectors or nodeAffinities for Strimzi's Kafka Exporter and the ZooEntrance pods via Helm.

- Java
Published by SoerenHenning about 3 years ago

theodolite - v0.8.4

This is the release of Theodolite 0.8.4. It fixes an issue introduced with the release of 0.8.3.

- Java
Published by SoerenHenning about 3 years ago

theodolite - v0.8.3

This is the release of Theodolite 0.8.3. It provides a bug fix for setting Strimzi's ZooKeeper Pod templates via Helm.

- Java
Published by SoerenHenning about 3 years ago

theodolite - v0.8.2

This is the release of Theodolite 0.8.2. It upgrades the kube-prometheus-stack to v41.7.* in our Helm chart. This comes with the following under-the-hood changes:

• We now use Grafana from kube-prometheus-stack instead of own subchart.
  • This also upgrades Grafana to the latest version.
  • As Grafana is only used for passive observation, nothing should have changed concerning the benchmark execution. However, to recover the old installation, set kube-prometheus-stack.grafana.enabled to false and set grafana to the default configuration from v0.8.1 in your values.yaml.
• The optional full-stack monitoring is now enhanced.
  • Note that this should not change anything in the default configuration but only if you use the kube-prometheus-stack to collect cluster metrics, node-level metrics, etc.
  • For full-stack monitoring, the default Prometheus from kube-prometheus-stack can now be used.
  • We provide a Helm values file for enabling full-stack monitoring.
  • Using this file now enables additional Kubernetes metrics exporters and Grafana dashboards.

- Java
Published by SoerenHenning over 3 years ago

theodolite - v0.8.1

This is the release of Theodolite 0.8.1. It upgrades Strimzi to v0.29.0 to support Kubernetes >=1.25. The default Kafka version does not change.

- Java
Published by SoerenHenning over 3 years ago

theodolite - v0.8.0

This is the release of Theodolite 0.8.0. The most important new features are: * Theodolite now comes with four benchmark implementations for Hazelcast Jet. * SLOs are now defined per benchmark and not per execution. This is more natural as typically the benchmark designer knows about relevant metrics and SLOs. However, benchmarkers can still override SLO configurations such as thresholds. * Theodolite adds full supports for the load capacity metric. Benchmarkers can choose whether they would like to use the already existing resource demand metric or the new load capacity metric. * Delete Actions are a second type of before/after actions, which, as their name suggests, delete Kubernetes resources when being executed. * The execution CRD definition for configuring the search strategy has changed. * Installation of the Confluent Helm chart is now disabled per default and will be completely removed in a future release.

Please note that the Theodolite's CRDs have changed. When upgrading from a previous version, a simple helm upgrade will not be sufficient. Instead you should uninstall and re-install Theodolite. As our CRDs full name has also changed, you might need to additionally run kubectl delete crd executions.theodolite.com; kubectl delete crd benchmarks.theodolite.com, before re-installing Theodolite.

- Java
Published by SoerenHenning over 3 years ago

theodolite - v0.7.0

This is the release of Theodolite 0.7.0. The most important new features are: * Theodolite now supports deploying arbitrary resources. This includes CRDs, managed by other operators. * Strimzi is used instead of the Confluent Helm charts for managing Kafka topics. * Before and After Actions can be used to define specific behavior to be run directly before the benchmark is started or before it is finished. * Theodolite creates a result CSV file directly after running a benchmark. Using the analysis notebooks is now optional, e.g., to evaluate different settings or plot results. * The initial guess strategy for running SLO experiments provides some advantages over the linear search. * Benchmark implementations for Apache Beam were added. * UC1 provides a generic interface for potentially different sinks (e.g., databases). * The UC1 implementation for Apache Beam supports Google Cloud Firestore as one of these sinks. * The load generator now exports metrics in Prometheus format (using Micrometer under the hood). * The load generator can now generate messages via HTTP and for Google Cloud Pub/Sub. * A HTTP Bridge was added, which receives HTTP messages from Theodolite's load generator and forwards them as Kafka messages.

- Java
Published by SoerenHenning almost 4 years ago

theodolite - v0.6.4

This is the release of Theodolite 0.6.4. It fixes an unresolveable Helm dependency (https://github.com/seglo/kafka-lag-exporter/issues/316).

- Java
Published by SoerenHenning about 4 years ago

theodolite - v0.5.2

This is the release of Theodolite 0.5.2. It fixes an unresolveable Helm dependency (https://github.com/seglo/kafka-lag-exporter/issues/316).

- Java
Published by SoerenHenning about 4 years ago

theodolite - v0.6.3

This is the release of Theodolite 0.6.3. It provides the following bug fix: * Theodolite's Prometheus now watches all ServiceMonitors in the current namespace. This fixes the missing documentation of requiring specific labels to be set for ServiceMonitors that should be used by Theodolite.

- Java
Published by SoerenHenning about 4 years ago

theodolite - v0.6.2

This is the release of Theodolite 0.6.2. With this release, we provides bug fixes as well as a preview of features coming with Theodolite v0.7. Specifically, these are: * A benchmark no longer has to define a Kafka object. If no Kafka object is defined, Theodolite does not create any topics. * Kubernetes files ending with .yml are now supported to be used by Theodolite. Previously, only .yaml files were supported. * Logging of exceptions is improved. * A bug in our Helm chart not allowing to enable the generic SLO checker while disabling the dropped records SLO is resolved. * Immense resource usage of the SLO checker caused by a bug in uvicorn-gunicorn-fastapi-docker was fixed by creating custom images. * Resource constraints for the Theodolite containers are set.

- Java
Published by SoerenHenning about 4 years ago

theodolite - v0.6.1

This is the release of Theodolite 0.6.1. With this release, we provide a preview for the following features coming with Theodolite v0.7:

• Add the generic SLO, which can be declaratively configured with a PromQL query, aggregation functions, a threshold and a comparison operator.
• Create a expXYZ_demand.csv file directly from Theodolite such that the analysis notebooks have not necessarily be used to calculate the demand metric.

- Java
Published by SoerenHenning about 4 years ago

theodolite - v0.6.0

This is the release of Theodolite 0.6.0. The most important new features are:

• Kubernetes resources describing the SUT or the workload generator don't longer need to be present inside the Theodolite container. Instead they can (and should) be put into ConfigMaps, from where Theodolite reads them
• In addition to SUT and load generator resources, benchmarks can now also define infrastructure components with run for an entire benchmark execution.
• Theodolite now generates Kubernetes events indicating the current state of an execution.
• Benchmarks now have a status indicating whether they are ready to be executed (all referring resources are accessible) or not.
• Theodolite's benchmarks are now automatically registered with Kubernetes when installing Theodolite. This can be disabled.
• A new SLO checker is available for Kafka Streams, which checks whether the amount of dropped records exceeds a certain threshold.
• Kubernetes volumes for benchmark results can be created from Helm.
• The Helm allows to define a nodeSelector for Theodolite.
• We updated some dependencies of our Helm chart.
• With this release our website and documentation is available via https://www.theodolite.rocks

Please note that the Theodolite CRDs have changed. When upgrading from a previous version, a simple helm upgrade will not be sufficient. Instead you should uninstall and re-install Theodolite.

- Java
Published by SoerenHenning about 4 years ago

theodolite - v0.5.1

This release provides two fixes: - minimal Helm installation does not require a PersiententVolumeClaim to be created before - Support for Kubernetes v1.22

- Java
Published by SoerenHenning over 4 years ago

theodolite - v0.5.0

This is the release of Theodolite 0.5.0. With this release we entirely revised Theodolite's architecture and re-implemented it from scratch. This release is a huge step toward the first stable version of Theodolite. Features of this release include:

• Entire rewrite of Theodolite in Kotlin and Quarkus for better interaction with the Kubernetes API
• Optional execution as Kubernetes Operator for better usability (recommended)
• A data model for defining benchmarks (as CRD in operator mode) to support arbitrary benchmarks and SUTs apart from the Theodolite benchmarks. This includes:
  • Description of SUTs and LoadGenerators as sets of Kubernetes resources
  • Support for arbitrary load and resources types, according to our scalability metric
  • Flexible configuration of Kafka topics
  • Benchmark CRs for the Theodolite Kafka Streams and Flink benchmarks are provided
• A data model for describing benchmark executions (again as CRD in operator mode). This includes:
  • Support for repeating SLO experiments, from which the median lag trend value is then chosen
  • Extension mechanism for custom SLO checkers
  • Support for arbitrary configuration changes of the SUT or load generator to benchmark different SUT configurations etc.
• A Theodolite Helm chart (recommended way for installation)
• Optional usage of a random scheduler to mitigate scheduling effects on the benchmark results
• Full support for implementations of the Theodolite benchmarks with Flink (partly already releases with v.0.4.0)

We thank @benediktwetzel, @lorenzboguhn, @sehrenstein, @bvonheid and @NicoBiernat for their massive support in designing, implementing, and testing Theodolite v0.5.0.

- Java
Published by SoerenHenning over 4 years ago

theodolite - v0.4.0

This is the release of Theodolite 0.4.0. The most important features of this release are:

• Changed enumeration of benchmarks to sort them by ascending complexity and match our Theodolite publication DOI: 10.1016/j.bdr.2021.100209.
• Substantial rewrite of Theodolite's load generator such that it does no longer depend on a running Zookeeper cluster.
• Benchmark implementations for Apache Flink.
• From this release on, our Docker images are provided via GitHub Container Registry.

- Java
Published by SoerenHenning almost 5 years ago

theodolite -

This is the release of Theodolite 0.3.0. The most important features of this release are:

• Entire rewrite of Theodolite in Python instead of Bash
• More efficient execution using configurable search heuristics
• Support for execution as a Kubernetes Job
• Setting custom benchmark environment variables via Theodolite

Other new features are:

• Kubernetes namespace for benchmarks is now configurable
• The documentations have massively been improved
• Benchmark results can be stored in volumes
• Theodolite Docker images are now hosted on GitHub per default

- Java
Published by SoerenHenning about 5 years ago

theodolite -

Most important features:

• Scalable workload generator
• Usage of Avro records instead of Kieker

- Java
Published by SoerenHenning about 5 years ago

theodolite -

Identical to release version 0.1.1, but fixes wrong Docker image tags.

- Java
Published by SoerenHenning about 5 years ago

theodolite -

First release.

- Java
Published by SoerenHenning about 5 years ago