https://github.com/awslabs/kinesis-aggregation

AWS libraries/modules for working with Kinesis aggregated record data

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
○
.zenodo.json file
○
DOI references
○
Academic publication links
○
Committers with academic emails
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (12.0%) to scientific vocabulary

Keywords from Contributors

observability interactive archival projection generic sequences tracing charts network-simulation human

Last synced: 10 months ago · JSON representation

Repository

AWS libraries/modules for working with Kinesis aggregated record data

Basic Info

Host: GitHub
Owner: awslabs
License: apache-2.0
Language: Java
Default Branch: master
Homepage:
Size: 271 MB

Statistics

Stars: 377
Watchers: 67
Forks: 149
Open Issues: 49
Releases: 8

Created over 10 years ago · Last pushed almost 2 years ago

Metadata Files

Readme Contributing License Code of conduct

Kinesis Record Aggregation & Deaggregation Modules for AWS Lambda

[!CAUTION] DO NOT use Kinesis Aggregation module for use cases where data loss of small number of messages is critical. DATA LOSS CAN OCCUR.

If you use Kinesis Aggregation module outside of Kinesis Producer Library (KPL), you should use it only for low-value messages where loss of a small number of messages is not critical. Aggregation results in messages being tagged to Shards, and in cases where the Stream is mutating during Aggregation, this can result in messages being rejected by the PutRecords API. Kinesis Deaggregation is compatible with all KPL published data and can be used safely.

The Amazon Kinesis Producer Library (KPL) gives you the ability to write data to Amazon Kinesis with a highly efficient, asyncronous delivery model that can improve performance. The KPL is extremely powerful, but is currently only available as a Java API wrapper around a C++ executable which may not be suitable for all deployment environments. Similarly, the Kinesis Client Library (KCL) provides automatic deaggregation of KPL aggregated records, but not all Kinesis consumer applications, such as those running on AWS Lambda, are currently capable of leveraging this deaggregation capability.

KPL Message Format How the Kinesis Producer Library Publishes Data

The components in this project give you the ability to process and create KPL compatible serialised data within AWS Lambda, in Java, Node.js and Python. These components can also be used as part of the Kinesis Client Library a multi-lang KCL application. However, this project has several limitations:

It only generates data, and doesn't perform delivery to the stream like the KPL. You still have to call PutRecord(s) to push data to Kinesis Data Streams
It doesn't manage data across multiple streams like KPL - the interface assumes that all data is sent to a single Stream

Aggregation

One of the main advantages of the KPL is its ability to use record aggregation to increase payload size and improve throughput. While this project is not a replacement for the full KPL, it does provide you the ability to easily aggregate multiple user records into larger aggregated records that make more efficient use of available bandwidth and reduce cost. This data is encoded using Google Protocol Buffers, and returned to the calling function for subsequent use. You can then publish to Kinesis and the data is compatible with consumers using the KCL or these Deaggregation modules.

Processing Model

Deaggregation

The components in this library allow you to efficiently deaggregate protocol buffer encoded aggregated records in any application, including AWS Lambda.

Processing Model

Language Specific Implementations

AWS Lambda supports Java, Node.js, Python and Go as programming languages. We have included support for those languages so that you can create and process UserRecords via standalone modules. Documentation is provided for each language:

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Owner

Name: Amazon Web Services - Labs
Login: awslabs
Kind: organization
Location: Seattle, WA

Website: http://amazon.com/aws/
Repositories: 914
Profile: https://github.com/awslabs

AWS Labs

GitHub Events

Total

Issues event: 1
Watch event: 1

Last Year

Issues event: 1
Watch event: 1

Committers

Last synced: about 1 year ago

All Time

Total Commits: 284
Total Committers: 40
Avg Commits per committer: 7.1
Development Distribution Score (DDS): 0.694

Past Year

Commits: 2
Committers: 1
Avg Commits per committer: 2.0
Development Distribution Score (DDS): 0.0

Top Committers

Name	Email	Commits
IanMeyers	m**i@a**m	87
brentnash	b**h@a**m	76
dependabot[bot]	4****]	32
Jerome Guibert	j**t@g**m	10
Subhi Al Hasan	j**a@g**m	7
stevegoossens	s**s@b**k	6
Joe Bowen	J**n@s**a	6
pablo	p**z@g**m	5
Christopher Strydom	c**m@e**m	5
brentnash	b**h@g**m	4
jeff	j**f@g**m	3
shyyko.serhiy	s**y@g**m	3
Andrej Leitner	a**r@g**m	3
Andrew Kursar	a**r@g**m	3
Oliver Yang	o**g@d**m	2
Armando Segnini	a**g@g**m	2
Henri Yandell	h**l@a**m	2
Jeffrey Barrus	b**j@g**m	2
Minu Hong	3****s	2
Paul	p**l@e**k	2
Xavier Thierry	x**y@c**m	2
drosenst	d**k@i**m	2
borlafu	b****u	1
Toltar	m**w@m**m	1
Takuto Komazaki	t**i@g**t	1
Shimon Tolts	s**n@d**o	1
Bill Letson	b**n@g**m	1
Chris Cmolik	g**t@c**e	1
Zenon Colaco	z**o@o**m	1
james	j**e@b**m	1
and 10 more...

Committer Domains (Top 20 + Academic)

amazon.com: 4 bbc.co.uk: 1 sjrb.ca: 1 expedia.com: 1 giftbit.com: 1 disney.com: 1 e-swoosh.co.uk: 1 carbonblack.com: 1 intel.com: 1 matthew-dickens.com: 1 gree.net: 1 datree.io: 1 gsngames.com: 1 ccmo.me: 1 orionhealth.com: 1 bravenewcoin.com: 1 ringieraxelspringer.pl: 1 sportsbet.com.au: 1 truecar.com: 1

Issues and Pull Requests

Last synced: 11 months ago

All Time

Total issues: 59
Total pull requests: 73
Average time to close issues: 9 months
Average time to close pull requests: 13 days
Total issue authors: 48
Total pull request authors: 16
Average comments per issue: 3.85
Average comments per pull request: 0.77
Merged pull requests: 63
Bot issues: 0
Bot pull requests: 44

Past Year

Issues: 2
Pull requests: 1
Average time to close issues: N/A
Average time to close pull requests: 2 days
Issue authors: 1
Pull request authors: 1
Average comments per issue: 0.0
Average comments per pull request: 0.0
Merged pull requests: 1
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

cosmincatalin (3)
lcmgh (2)
jesshart (2)
ferozed (2)
rrbarbosa (1)
itamarla (1)
reneincer (1)
pushplay (1)
LilTwo (1)
andrejleitner (1)
greylurk (1)
NewstHet (1)
gerodp (1)
mfulleratlassian (1)
ShyykoSerhiy (1)

Pull Request Authors

dependabot[bot] (39)
ShyykoSerhiy (3)
stevegoossens (2)
oyang (2)
akursar (2)
minuhong-aws (2)
andrejleitner (2)
Strydom (2)
IanMeyers (2)
plopezlpz (1)
slaboszm (1)
maxispeicher (1)
jnewbigin (1)
mnunna-vmware (1)
mnunna-broadcom (1)

Top Labels

Issue Labels

Pull Request Labels

dependencies (39) java (20) javascript (15) go (2) python (2)

Packages

Total packages: 8
Total downloads:
- npm 71,609 last-month
- pypi 337,959 last-month
Total docker downloads: 771,289,615

Total dependent packages: 197
(may contain duplicates)
Total dependent repositories: 534
(may contain duplicates)
Total versions: 63
Total maintainers: 4

proxy.golang.org: github.com/awslabs/kinesis-aggregation/go

Homepage: https://github.com/awslabs/kinesis-aggregation
Documentation: https://pkg.go.dev/github.com/awslabs/kinesis-aggregation/go#section-documentation
License: apache-2.0
Latest release: v0.0.0-20230808105340-e631fe742486
published almost 3 years ago

Versions: 12
Dependent Packages: 156
Dependent Repositories: 322
Docker Downloads: 771,289,615

Rankings

Docker downloads count: 0.1%

Dependent packages count: 0.4%

Dependent repos count: 0.4%

Average: 1.2%

Forks count: 2.1%

Stargazers count: 2.9%

Last synced: 10 months ago

proxy.golang.org: github.com/awslabs/kinesis-aggregation/go/v2

Homepage: https://github.com/awslabs/kinesis-aggregation
Documentation: https://pkg.go.dev/github.com/awslabs/kinesis-aggregation/go/v2#section-documentation
License: apache-2.0
Latest release: v2.0.0-20230808105340-e631fe742486
published almost 3 years ago

Versions: 10
Dependent Packages: 16
Dependent Repositories: 30

Rankings

Dependent repos count: 1.0%

Dependent packages count: 1.5%

Average: 1.9%

Forks count: 2.2%

Stargazers count: 3.0%

Last synced: over 1 year ago

npmjs.org: aws-kinesis-agg

Node.js module to simplify working with Amazon Kinesis Records using Protcol Buffers encoding

Homepage: https://github.com/awslabs/kinesis-aggregation
License: Apache-2.0
Latest release: 4.2.8
published almost 3 years ago

Versions: 20
Dependent Packages: 13
Dependent Repositories: 38
Downloads: 71,586 Last month

Rankings

Downloads: 0.7%

Dependent packages count: 1.9%

Dependent repos count: 2.1%

Average: 2.1%

Forks count: 2.5%

Stargazers count: 3.3%

Maintainers (1)

ianmeyers

Last synced: 11 months ago

pypi.org: aws-kinesis-agg

Python module to assist in taking advantage of the Kinesis message aggregation format for both aggregation and deaggregation.

Homepage: http://github.com/awslabs/kinesis-aggregation
Documentation: https://aws-kinesis-agg.readthedocs.io/
License: Apache-2.0
Latest release: 1.2.3
published about 4 years ago

Versions: 14
Dependent Packages: 0
Dependent Repositories: 14
Downloads: 337,959 Last month

Rankings

Downloads: 0.7%

Stargazers count: 3.3%

Dependent repos count: 3.9%

Forks count: 3.9%

Average: 4.4%

Dependent packages count: 10.0%

Maintainers (1)

IanMeyers

Last synced: 11 months ago

repo1.maven.org: com.amazonaws:amazon-kinesis-aggregator

A library for performing in-memory aggregation of Kinesis stream records.

Homepage: https://aws.amazon.com/kinesis
Documentation: https://appdoc.app/artifact/com.amazonaws/amazon-kinesis-aggregator/
License: Amazon Software License
Latest release: 1.0.3
published over 8 years ago

Versions: 1
Dependent Packages: 7
Dependent Repositories: 115

Rankings

Dependent repos count: 1.8%

Dependent packages count: 8.4%

Average: 10.9%

Forks count: 15.6%

Stargazers count: 17.9%

Last synced: 10 months ago

npmjs.org: aws-kinesis-agg-fix

Node.js module to simplify working with Amazon Kinesis Records using Protcol Buffers encoding

Homepage: http://github.com/awslabs/kinesis-aggregation
License: SEE LICENSE IN LICENSE.txt
Latest release: 4.0.2
published over 8 years ago

Versions: 3
Dependent Packages: 2
Dependent Repositories: 1
Downloads: 22 Last month

Rankings

Forks count: 2.5%

Stargazers count: 3.3%

Dependent repos count: 10.4%

Average: 16.3%

Dependent packages count: 21.0%

Downloads: 44.3%

Maintainers (1)

geronimo-iia

Last synced: 11 months ago

repo1.maven.org: com.amazonaws:amazon-kinesis-deaggregator

A library for performing in-memory deaggregation of Kinesis aggregated stream records.

Homepage: https://aws.amazon.com/kinesis
Documentation: https://appdoc.app/artifact/com.amazonaws/amazon-kinesis-deaggregator/
License: Amazon Software License
Latest release: 1.0.3
published over 8 years ago

Versions: 1
Dependent Packages: 1
Dependent Repositories: 13

Rankings

Dependent repos count: 6.8%

Forks count: 15.6%

Stargazers count: 17.9%

Average: 18.3%

Dependent packages count: 33.0%

Last synced: 10 months ago

npmjs.org: aws-kinesis-agg-temp

Node.js module to simplify working with Amazon Kinesis Records using Protcol Buffers encoding

Homepage: http://github.com/awslabs/kinesis-aggregation
License: SEE LICENSE IN LICENSE.txt
Latest release: 4.0.3
published about 7 years ago

Versions: 2
Dependent Packages: 2
Dependent Repositories: 1
Downloads: 1 Last month

Rankings

Forks count: 2.5%

Stargazers count: 3.3%

Dependent repos count: 10.4%

Average: 18.6%

Dependent packages count: 21.0%

Downloads: 55.9%

Maintainers (1)

jodevsa

Last synced: 11 months ago

Dependencies

go/go.mod go

github.com/aws/aws-sdk-go v1.19.48
github.com/golang/protobuf v1.3.1
github.com/stretchr/testify v1.3.0

go/go.sum go

github.com/aws/aws-sdk-go v1.19.48
github.com/davecgh/go-spew v1.1.0
github.com/golang/protobuf v1.3.1
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.0
github.com/stretchr/testify v1.3.0

go/v2/go.mod go

github.com/aws/aws-sdk-go-v2/service/kinesis v1.6.0
github.com/golang/protobuf v1.5.2
github.com/stretchr/testify v1.7.0

go/v2/go.sum go

github.com/aws/aws-sdk-go-v2 v1.9.0
github.com/aws/aws-sdk-go-v2/service/kinesis v1.6.0
github.com/aws/smithy-go v1.8.0
github.com/davecgh/go-spew v1.1.0
github.com/golang/protobuf v1.5.0
github.com/golang/protobuf v1.5.2
github.com/google/go-cmp v0.5.4
github.com/google/go-cmp v0.5.5
github.com/google/go-cmp v0.5.6
github.com/jmespath/go-jmespath v0.4.0
github.com/jmespath/go-jmespath/internal/testify v1.5.1
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.0
github.com/stretchr/testify v1.7.0
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
google.golang.org/protobuf v1.26.0-rc.1
google.golang.org/protobuf v1.26.0
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
gopkg.in/yaml.v2 v2.2.8
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c

java/KinesisAggregator/pom.xml maven

com.amazonaws:amazon-kinesis-client 1.13.3
com.fasterxml.jackson.core:jackson-core 2.10.2
com.fasterxml.jackson.core:jackson-databind [2.10.5.1,)
com.fasterxml.jackson.dataformat:jackson-dataformat-cbor 2.10.2
commons-logging:commons-logging 1.1.1
org.apache.commons:commons-lang3 3.10
commons-codec:commons-codec 1.11 test
junit:junit 4.13.1 test

java/KinesisAggregatorV2/pom.xml maven

com.fasterxml.jackson.core:jackson-core 2.10.2
com.fasterxml.jackson.core:jackson-databind [2.10.5.1,)
com.fasterxml.jackson.dataformat:jackson-dataformat-cbor 2.11.4
commons-logging:commons-logging 1.1.1
org.apache.commons:commons-lang3 3.10
software.amazon.kinesis:amazon-kinesis-client 2.2.11
commons-codec:commons-codec 1.11 test
junit:junit 4.13.1 test

java/KinesisDeaggregator/pom.xml maven

com.amazonaws:amazon-kinesis-client 1.13.3
com.amazonaws:aws-lambda-java-core 1.2.1
com.amazonaws:aws-lambda-java-events 3.1.0
com.fasterxml.jackson.core:jackson-databind 2.13.4.2
org.apache.commons:commons-lang3 3.10
com.amazonaws:amazon-kinesis-aggregator 1.1.2 test
junit:junit 4.13.1 test

java/KinesisDeaggregatorV2/pom.xml maven

com.amazonaws:aws-lambda-java-core 1.2.1
com.amazonaws:aws-lambda-java-events 3.1.0
com.fasterxml.jackson.core:jackson-databind 2.13.4.2
org.apache.commons:commons-lang3 3.10
software.amazon.kinesis:amazon-kinesis-client 2.2.11
com.amazonaws:amazon-kinesis-aggregator 1.1.2 test
junit:junit 4.13.1 test

java/KinesisLambdaForwarder/pom.xml maven

com.amazonaws:amazon-kinesis-aggregator 1.0.3
com.amazonaws:amazon-kinesis-deaggregator 1.0.3
com.amazonaws:aws-java-sdk-kinesis 1.11.251
com.amazonaws:aws-java-sdk-sts 1.11.251
com.amazonaws:aws-lambda-java-core 1.1.0
com.amazonaws:aws-lambda-java-events 1.1.0

java/KinesisTestConsumers/pom.xml maven

com.amazonaws:amazon-kinesis-client 1.13.3
com.amazonaws:amazon-kinesis-deaggregator 1.1.3
com.amazonaws:aws-lambda-java-core 1.2.1
com.amazonaws:aws-lambda-java-events 3.1.0

java/KinesisTestProducers/pom.xml maven

com.amazonaws:amazon-kinesis-aggregator 1.0.3
com.amazonaws:amazon-kinesis-producer 0.10.2
com.amazonaws:aws-java-sdk-kinesis 1.11.251

java/pom-aggregate/pom.xml maven

com.amazonaws:amazon-kinesis-aggregator 1.0.3 compile
com.amazonaws:amazon-kinesis-deaggregator 1.0.3 compile
com.amazonaws:amazon-kinesis-lambda-forwarder 1.0.3 compile
com.amazonaws:amazon-kinesis-test-consumers 1.0.3 compile
com.amazonaws:amazon-kinesis-test-producers 1.0.3 compile

java/pom.xml maven

node/package-lock.json npm

176 dependencies

node/package.json npm

@types/aws-lambda ^8.10.64 development
aws-sdk >=2.814.0 development
jshint ^2.11.0 development
mocha ^9.1.4 development
node-uuid ^1.4.8 development
should ^13.2.1 development
should-sinon ^0.0.6 development
sinon ^4.2.2 development
typescript ^3.1.3 development
agentkeepalive ^4.1.4
ansi-regex >=5.0.1
async 2.6.4
minimist >=0.2.1
protobufjs 7.2.4

python/aws_kinesis_agg.egg-info/requires.txt pypi

protobuf *

python/requirements.txt pypi

protobuf ==4.21.6
six ==1.11.0

python/setup.py pypi

protobuf *

https://github.com/awslabs/kinesis-aggregation

Science Score: 13.0%

Keywords from Contributors

Repository

Basic Info

Statistics

Metadata Files

README.md

Kinesis Record Aggregation & Deaggregation Modules for AWS Lambda

Aggregation

Deaggregation

Language Specific Implementations

Owner

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

proxy.golang.org: github.com/awslabs/kinesis-aggregation/go

Rankings

proxy.golang.org: github.com/awslabs/kinesis-aggregation/go/v2

Rankings

npmjs.org: aws-kinesis-agg

Rankings

Maintainers (1)

pypi.org: aws-kinesis-agg

Rankings

Maintainers (1)

repo1.maven.org: com.amazonaws:amazon-kinesis-aggregator

Rankings

npmjs.org: aws-kinesis-agg-fix

Rankings

Maintainers (1)

repo1.maven.org: com.amazonaws:amazon-kinesis-deaggregator

Rankings

npmjs.org: aws-kinesis-agg-temp

Rankings

Maintainers (1)

Dependencies