https://github.com/52north/postgis-kafka-cdc
Change Data Capture (CDC) with Debezium, Postgis and Kafka
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (6.2%) to scientific vocabulary
Keywords
Repository
Change Data Capture (CDC) with Debezium, Postgis and Kafka
Basic Info
Statistics
- Stars: 1
- Watchers: 9
- Forks: 0
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
ARCHIVED
This project is no longer maintained and will not receive any further updates. If you plan to continue using it, please be aware that future security issues will not be addressed.
Kafka / Postgis Change Data Capture examples
Change Data Capture (CDC) with Debezium, Postgis and Kafka.
Deployment
Currently, images are not yet pushed to docker hub. Please compile locally (it's easy, docker-compose required):
cd kafka
docker-compose build
cd ../kafka-connect-debezium
docker-compose build
cd ../postgis-debezium
docker-compose build
cd ..
Run with docker-compose rm -f && docker-compose up --build.
Wait for the container to boot up. To initialize the connector,
send an HTTP POST (application/json)
to http://localhost:8083/connectors/:
json
{
"name": "inventory-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "postgres",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgres",
"database.dbname" : "postgres",
"database.server.name": "fullfillment",
"table.whitelist": "public.inventory",
"snapshot.mode": "never"
}
}
Known Limitations
Work in progress, so do not expect production stability.
Kafka Version
The currently used debezium Kafka connect plugin is compiled against the 0.10.2.0 API of Kafka. It seems to work with Kafka 0.11.0.0 as well, but we cannot guarantee stable behavior. Once debezium is available for version 0.11.0.0, we will update the Docker images.
Unexpected Behavior
It may happen that changes are only available after a delay (we observed 1-2 minutes). This may be related to topic management, as it does not seem to happen if a consumer first (i.e. before POSTing the above connector config) subscribes to the topic that is later used by debezium to publish changes.
Acknowledgement
These Docker images have been derived from debezium/docker-images.
Owner
- Name: 52°North Spatial Information Research GmbH
- Login: 52North
- Kind: organization
- Email: info@52north.org
- Location: Münster
- Website: https://52north.org/
- Twitter: fivetwon
- Repositories: 261
- Profile: https://github.com/52North
Advancing spatial information infrastructures to foster open science
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: about 1 year ago
Dependencies
- 52north/kafka 0.11.0.0
- 52north/kafka-connect-debezium 0.11.0.0
- 52north/postgis-debezium 9.5
- 52north/sos-configured 4.4.0
- zookeeper 3.4
- openjdk 8-jdk-alpine build
- 52north/kafka 0.11.0.0
- 52north/kafka 0.11.0.0 build
- 52north/kafka-connect-debezium 0.11.0.0
- mdillon/postgis 9.5 build
- 52north/postgis-debezium 9.5