iot-processing-architecture
Architecture for processing and storing Internet of Things (IoT) data in a scalable way.
Science Score: 44.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.9%) to scientific vocabulary
Repository
Architecture for processing and storing Internet of Things (IoT) data in a scalable way.
Basic Info
- Host: GitHub
- Owner: nyamak
- Language: Python
- Default Branch: main
- Size: 27.9 MB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 2
- Releases: 0
Metadata Files
README.md
IoT Processing Architecture
A simple scalable architecture of an IoT data processing pipeline, using common components.
Made as a final project for my Electrical Engineering bachelor's degree at Universidade Federal de Santa Catarina (UFSC).
Project structure
This project can be divided into Infrastructure (/infrastructure),
Microservices (/microservices) and Testing (/testing).
Infrastructure
Written in Terraform, this folder contains all files related to infratructure deployment.
Microservices
Written in Python, this folder contains all code related to application servers.
Testing
This folder contains all code related to load testing, written using Locust.io.
How to use
Setup
To start, you need to have the latest version of Terraform (v1.4.6 as of the
writing of this document) and a free-tier Sendgrid
account, with an API key
and a verified email.
Deployment
If you simply want to deploy the latest version of this application using
Kubernetes, you may update your .kubeconfig file with the desired context
and add its name on ./infrastructure/providers.tf, under both config_context
parameters. You may use minikube for local deployment or your cloud provider
of choice.
After, you may simply go to the .\infrastructure folder (cd infrastructure)
and execute terraform init, followed by terraform apply, setting you correct
credentials and variables.
Load testing
To execute the load tests, you must first install the requirements with:
cd testing
pip install -r requirements.txt
Then you must update create a .env file, replacing any values in the
.env.example file.
To execute the load tests, run the following line:
locust -f load_testing.py
Access the URL shown in the command line and set the parameters.
Sample payload
{'unit_id':123,'created_at':'2022-05-18T11:40:22.519222','is_defective': true,'machine_id':123,'machine_temperature':110.0,'machine_pressure':1.2}
With mosquitto installed, to send a payload to the app use:
mosquitto_pub -p 1883 -h localhost -m "{'unit_id':123,'created_at':'2022-05-18T11:39:22.519222','is_defective': true,'machine_id':123,'machine_temperature':110.0,'machine_pressure':1.2}" -t payloads
Owner
- Name: Nícolas Yamakoshi
- Login: nyamak
- Kind: user
- Location: Brazil
- Repositories: 3
- Profile: https://github.com/nyamak
Citation (CITATION.cff)
# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!
cff-version: 1.2.0
title: IoT Processing Architecture
message: >-
If you use this software, please cite it using the
metadata from this file.
type: software
authors:
- given-names: Nicolas Vitor Yuki
family-names: Obara Yamakoshi
email: nicolas.yamakoshi@gmail.com
repository-code: 'https://github.com/nyamak/iot-processing-architecture'
license: MIT
date-released: '2023-06-02'
GitHub Events
Total
- Push event: 1
Last Year
- Push event: 1
Dependencies
- python 3.11.3-slim-buster build
- python 3.11.3-slim-buster build
- flask_api ==3.0.post1
- python-dotenv ==1.0
- redis ==4.5.5
- sendgrid ==6.10.0
- paho-mqtt ==1.6.1
- psycopg2-binary ==2.9.3
- python-dotenv ==1.0.0
- requests ==2.30.0
- locust ==2.15.1 test
- locust-plugins ==3.3.1 test
- paho-mqtt ==1.6.1 test