Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of https://github.com/talariadb/talaria

https://github.com/talariadb/talaria - v1.6.1

What's Changed

  • test: use T.TempDir to create temporary test directory by @Juneezee in https://github.com/talariadb/talaria/pull/101
  • Add cocureency ingestion per threads setting by @atlas-comstock in https://github.com/talariadb/talaria/pull/100
  • Update badgerdb and ristretto v0.2.0 to fix bugs on arm by @atlas-comstock in https://github.com/talariadb/talaria/pull/98
  • Add config block cache size for badgerDB by @atlas-comstock in https://github.com/talariadb/talaria/pull/102
  • fix issues with pr #86 by @jeffreylean in https://github.com/talariadb/talaria/pull/106
  • Refactor/streaming v2 by @jeffreylean in https://github.com/talariadb/talaria/pull/107
  • add tables field to ingest grpc contract by @jeffreylean in https://github.com/talariadb/talaria/pull/108
  • Upgrade github CI go version to 1.17 by @atlas-comstock in https://github.com/talariadb/talaria/pull/111
  • Update GitHub action go version by @atlas-comstock in https://github.com/talariadb/talaria/pull/112
  • Fix graceful shutdown by @atlas-comstock in https://github.com/talariadb/talaria/pull/110
  • implement instrumentation in bigquery by @jeffreylean in https://github.com/talariadb/talaria/pull/117
  • lint the code base on https://staticcheck.io/docs/checks/ by @atlas-comstock in https://github.com/talariadb/talaria/pull/119
  • fix streaming inside ingestWithTable method by @jeffreylean in https://github.com/talariadb/talaria/pull/121
  • add grpc call message size configuration by @jeffreylean in https://github.com/talariadb/talaria/pull/123
  • Feature/nats ingress by @jeffreylean in https://github.com/talariadb/talaria/pull/122
  • Spark connector by @tardunge in https://github.com/talariadb/talaria/pull/115
  • fix nats conf tag casing by @jeffreylean in https://github.com/talariadb/talaria/pull/124
  • Bump github.com/aws/aws-sdk-go from 1.30.25 to 1.33.0 by @dependabot in https://github.com/talariadb/talaria/pull/125
  • Hotfix/slow consumer handling by @jeffreylean in https://github.com/talariadb/talaria/pull/127
  • ingest add metrics ingested rows by @atlas-comstock in https://github.com/talariadb/talaria/pull/129
  • Feat/nats multi table support by @jeffreylean in https://github.com/talariadb/talaria/pull/133

New Contributors

  • @Juneezee made their first contribution in https://github.com/talariadb/talaria/pull/101
  • @jeffreylean made their first contribution in https://github.com/talariadb/talaria/pull/106
  • @dependabot made their first contribution in https://github.com/talariadb/talaria/pull/125

Full Changelog: https://github.com/talariadb/talaria/compare/v1.6.0...v1.6.1

- Go
Published by tardunge over 2 years ago

https://github.com/talariadb/talaria - v1.6.0

What's Changed

  • Refactor Sink Writer to accept blocks inplace of bytes by @tardunge in https://github.com/kelindar/talaria/pull/86

Full Changelog: https://github.com/kelindar/talaria/compare/v1.5.1...v1.6.0

- Go
Published by kelindar over 3 years ago

https://github.com/talariadb/talaria - v1.5.1

What's Changed

  • Fix issue #88: Use env to set Azure auth if existed by @atlas-comstock in https://github.com/kelindar/talaria/pull/91

Full Changelog: https://github.com/kelindar/talaria/compare/v1.5.0...v1.6.0

- Go
Published by kelindar over 3 years ago

https://github.com/talariadb/talaria - v1.5.0

What's Changed

  • adding go plugin as a new method in Computed by @atlas-comstock in https://github.com/kelindar/talaria/pull/87

New Contributors

  • @atlas-comstock made their first contribution in https://github.com/kelindar/talaria/pull/87

Full Changelog: https://github.com/kelindar/talaria/compare/v1.4.0...v1.5.0

- Go
Published by kelindar almost 4 years ago

https://github.com/talariadb/talaria - v1.4.0

Changelog

  • Added support for managed identities on Azure and improved monitoring https://github.com/kelindar/talaria/pull/85

- Go
Published by kelindar about 4 years ago

https://github.com/talariadb/talaria - v1.3.1

This release fixes the protobuf errors that occur with the new gRPC.

Changelog

  • Gogo protobuf is incompatible with grpc-go API/v2.
  • Replaced proto-gen-gogo-slick with proto-gen-go-grpc and proto-gen-go-vtproto plugins.
  • Modified python code generation script.

- Go
Published by kelindar over 4 years ago

https://github.com/talariadb/talaria - v1.3.0

This release upgrades to badger/v3 which has a lot of memory-related changes under the hood. See https://github.com/dgraph-io/badger/blob/master/CHANGELOG.md

Changelog

  • (Breaking) Rebasing on badger/v3 which makes data incompatible https://github.com/kelindar/talaria/pull/78

- Go
Published by kelindar over 4 years ago

https://github.com/talariadb/talaria - v1.2.9

This releases introduces fixes of parquet ingestion and expands the number of keys that can be added as partition keys.

Changelog

  • Expand the range of data types that can be used for hashBy column https://github.com/kelindar/talaria/pull/75
  • Fix parquet ingestion for missing columns https://github.com/kelindar/talaria/pull/77

- Go
Published by kelindar over 4 years ago

https://github.com/talariadb/talaria - v1.2.8

This release includes azure weighted storage when using multi-account (https://github.com/kelindar/talaria/pull/70) and parquet encoder (egress) https://github.com/kelindar/talaria/pull/67

Changelog

  • Added support for writing to Azure with weighted choice
  • Added support for parquet encoding when writing data out of Talaria during compaction
  • Added an entry point to allow loading env variable and potentially and generic init procedure within the container. I believe this is useful because if you are using something like Vault to store your secrets, the secrets are mounted as files within the container. With this entry point it will be able to load those secrets.
  • Updated Docker build pattern, I noticed there was the attempt to have a "build" pattern however it was commented out. I believe having the Dockerfile in this way is better because people can test easily locally to see if the image builds correctly.

- Go
Published by kelindar almost 5 years ago

https://github.com/talariadb/talaria - v1.2.7

This release contains major throughput improvements to Azure writer and Talaria now writes to all sinks in parallel. See #66 for more details. Special thanks to @qiaowei-g and @ocassetti for this PR.

Changelog

  • Add Azure Multi Storage Account Writer, use Azure AD credential for authentication.
  • Change multi.writer to call writers in parallel.
  • Add Datadog metric for writer latency and error stats.

- Go
Published by kelindar almost 5 years ago

https://github.com/talariadb/talaria - v1.2.6

This release contains the Parquet support for the ingestion (https://github.com/kelindar/talaria/pull/62) , allowing Talaria to directly read and ingest Parquet files into its storage layer.

- Go
Published by kelindar almost 5 years ago

https://github.com/talariadb/talaria - v1.2.5

This release contains the upgrade of ORC reader in order to fix the reader of float64 as well as a fix for some of our unit tests https://github.com/kelindar/talaria/pull/61.

- Go
Published by kelindar almost 5 years ago

https://github.com/talariadb/talaria - v1.2.4

This release contains various bug fixes.

  • Added support for map[string]interface{} to being auto-serialized into JSON for Golang client #54
  • Fix filter to return nil when data is filtered out #53 by
  • Fix sub streaming not starting #52

- Go
Published by kelindar about 5 years ago

https://github.com/talariadb/talaria - v1.2.3

This release adds real-time sub-streaming capability to Talaria. This can be configured per-table by using streams configuration.

Example configuration tables: eventlog: streams: - pubsub: project: my-gcp-project topic: my-topic filter: "gcs://my-bucket/my-function.lua" encoder: json ...

Changelog

  • Streaming capability for Sinks.
  • Added Google Pub/Sub as a possible sub-streaming sink.

- Go
Published by kelindar over 5 years ago

https://github.com/talariadb/talaria - v1.2.2

This release adds WithLoadBalancer option to the Golang client of talaria, allowing to specify clientside load balancing strategy.

- Go
Published by kelindar over 5 years ago

https://github.com/talariadb/talaria - v1.2.1

Changelog

This release fixes the following changes:

  • When querying, empty frames will be ignored which previously resulted in a panic.
  • When ingesting a varchar column, allow for schema re-mapping so that it can be converted to JSON, int32 or other types.
  • Golang client for Talaria now includes kubernetes resolver, allowing to Dial() with kubernetes schema. See: https://github.com/sercand/kuberesolver

- Go
Published by kelindar over 5 years ago

https://github.com/talariadb/talaria - v1.2.0

This version adds a support for multiple tables and ultimately introduces ability to re-partition data automatically by multiple different dimensions, during the ingestion process. For example, data can be partitioned by event, booking, and user_id by providing a following configuration: tables: events: hashBy: "event" sortBy: "ingested_at" bookings: hashBy: "booking_id" sortBy: "ingested_at" users: hashBy: "user_id" sortBy: "ingested_at"

Breaking changes:

  • Environment variable configuration TALARIA_CONF prefix is replaced with TALARIA
  • Table configuration has been revamped (see /config/sample_config.yaml)
  • Removed the ability to configure logs and nodes tables as they were never actually changed
  • Compaction configuration is now per table

- Go
Published by kelindar over 5 years ago

https://github.com/talariadb/talaria - v1.1.18

  • Added Talaria Sink in order to re-publish compacted data into a second Talaria

- Go
Published by kelindar over 5 years ago

https://github.com/talariadb/talaria - v1.1.17

  • Support JSON string input if desired static schema column is JSON https://github.com/kelindar/talaria/pull/39

- Go
Published by kelindar over 5 years ago

https://github.com/talariadb/talaria - v1.1.16

  • Increased compression chunk in ORC in order to support large strings.

- Go
Published by kelindar over 5 years ago

https://github.com/talariadb/talaria - v1.1.15

  1. Use zlib compression
  2. Add statsd metrics for keys added and keys deleted
  3. Improved compaction and key deletion latency
  4. Expose badger options as follows: badger: levelOneSize: 204800000 maxLevels: 3 syncWrites: false

- Go
Published by kelindar over 5 years ago

https://github.com/talariadb/talaria - v1.1.14

  • Added compaction prior to server shutdown

- Go
Published by kelindar over 5 years ago

https://github.com/talariadb/talaria - v1.1.13

  • Fix the compaction for the key whose hash key value is different from the last merged key.
  • Change the compression to snappy for faster writes.
  • Add stats for error in ingestion and compaction

- Go
Published by kelindar over 5 years ago

https://github.com/talariadb/talaria - v1.1.12

  • Added an ability to write to multiple sinks
  • Added net module for LUA exposing a way to query an IP/MAC addresses

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.11

  • Added liveness & readiness probe server endpoint for envoy
  • Fixed BigQuery writer which defaulted to CSV instead of ORC
  • Fixed timestamp column in ORC
  • Changed make://timestamp to output the value as a timestamp instead of int64

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.10

  • Fixed an issue where computed columns were not computed for all rows under very load.
  • Changed the docker image workflows. The edge tag is now built only for PRs, latest for every master and :v1.x.x for every release.

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.9

  • A unique event identifier can be generated by using make://identifier function in a computed column.
  • An ingestion time stamp can be generated by using make://timestamp function in a computed column.

In order to use it, simply declare a computed column as such:

computed: - name: ingested_at func: "make://timestamp" - name: id func: "make://identifier"

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.8

  • Added better initialization logs

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.7

  • Added support for loading computed columns and configs from Google Cloud Storage. This can be done by using gs://bucket/prefix scheme when specifying the url.

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.6

  • Added ingestion by CSV and URL (for both CSV and ORC files)
  • Added certs to the docker image

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.5

  • Added the ability to load the entire config from TALARIA_CONF environment variable.
  • Fixed an issue with environment config loader for nil structs.

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.4

  • Changed the path to kelindar/talaria
  • Fixed default name function for the compaction
  • Rebased docker image on debian

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.3

  • Fixed an issue with loader not having a default region
  • Fixed an issue with debug and info logs
  • Added a default name function for compaction

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.2

Adds a flexible schema url which can be dynamically reloaded if changes.

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.1

  • Adds support for local file sink.
  • Adds support for configurable S3 sink which can now be used for DigitalOcean and Minio.

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.1.0

  • Breaking change: s3Compact section from the configuration is removed
  • Adds support for Big Query, Google Cloud Storage and Azure as compaction sinks.

- Go
Published by kelindar almost 6 years ago

https://github.com/talariadb/talaria - v1.0.0

- Go
Published by kelindar almost 6 years ago