mmkit-features
A multimodal architecture to build multimodal knowledge graphs with flexible multimodal feature extraction and dynamic multimodal concept generation
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.1%) to scientific vocabulary
Keywords
Repository
A multimodal architecture to build multimodal knowledge graphs with flexible multimodal feature extraction and dynamic multimodal concept generation
Basic Info
- Host: GitHub
- Owner: dhchenx
- License: mit
- Language: Python
- Default Branch: main
- Homepage: https://pypi.org/project/mmkit-features/
- Size: 324 MB
Statistics
- Stars: 10
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
MMKit-Features: Multimodal Feature Extraction Toolkit
Traditional knowledge graphs (KGs) are usually comprised of entities, relationships, and attributes. However, they are not designed to effectively store or represent multimodal data. This limitation prevents them from capturing and integrating information from different modes of data, such as text, images, and audio, in a meaningful and holistic way.
The MMKit-Features project proposes a multimodal architecture to build multimodal knowledge graphs with flexible multimodal feature extraction and dynamic multimodal concept generation.
Project Goal
- To extract, store, and fuse various multimodal features from multimodal datasets efficiently;
- To achieve generative adversarial network(GAN)-based multimodal knowledge representation dynamically in multimodal knowledge graphs;
- To provide a common deep learning-based architecture to enhance multimodal knowledge reasoning in real life.
Installation
You can install this toolkit using our PyPi package.
pip install mmkit-features
Design Science Framework
Figure 1: Multimodal Computational Sequence
Figure 2: GAN-based Multimodal Concept Generation
Modalities
- Text/Language modality
- Image modality
- Video modality
- Audio modality
- Cross-modality among above
Usage
A toy example showing how to build a multimodal feature (MMF) library is here:
python
from mmkfeatures.fusion.mm_features_lib import MMFeaturesLib
from mmkfeatures.fusion.mm_features_node import MMFeaturesNode
import numpy as np
if __name__ == "__main__":
# 1. create an empty multimodal features library with root and dataset names
feature_lib = MMFeaturesLib(root_name="test features",dataset_name = "test_features")
# 2. set short names for each dimension for convenience
feature_lib.set_features_name(["feature1","feature2","feature3"])
# 3. set a list of content IDs
content_ids = ["content1","content2","content3"]
# 4. according to IDs, assign a group of features with interval to corresponding content ID
features_dict = {}
for id in content_ids:
mmf_node = MMFeaturesNode(id)
mmf_node.set_item("name",str(id))
mmf_node.set_item("features",np.array([[1,2,3]]))
mmf_node.set_item("intervals",np.array([[0,1]]))
features_dict[id] = mmf_node
# 5. set the library's data
feature_lib.set_data(features_dict)
# 6. save the features to disk for future use
feature_lib.save_data("test6_feature.csd")
# 7. check structure of lib file with the format of h5py
feature_lib.show_structure("test6_feature.csd")
# 8. have a glance of features content within the dataset
feature_lib.show_sample_data("test6_feature.csd")
# 9. Finally, we construct a simple multimodal knowledge base.
Further instructions on the toolkit refers to here.
Applications
Here are some examples of using our work in real life with codes and documents.
1. Multimodal Features Extractors
- Text Features Extraction
- Speech Features Extraction
- Image Features Extractoin
- Video Features Extraction
- Transformer-based Features Extraction
2. Multimodal Feature Library (MMFLib)
3. Multimodal Knowledge Bases
- Multimodal Birds Feature Library
- Multimodal Disease Coding Feature Library
- Multimodal ROCO Feature Library
4. Multimodal Indexing and Querying
Credits
The project includes some source codes from various open-source contributors. Here is a list of their contributions.
- A2Zadeh/CMU-MultimodalSDK
- aishoot/SpeechFeatureExtraction
- antoine77340/videofeatureextractor
- jgoodman8/py-image-features-extractor
- v-iashin/Video Features
License
The mmkit-features project is provided by Donghua Chen with MIT license.
Citation
Please cite our project if the project is used in your research.
Chen, D. (2023). MMKit-Features: Multimodal Features Extraction Toolkit (Version 0.0.2) [Computer software]
Owner
- Name: Donghua Chen
- Login: dhchenx
- Kind: user
- Location: Beijing, China
- Company: Department of Artificial Intelligence, University of International Business and Economics
- Website: https://dhchenx.github.io/
- Repositories: 22
- Profile: https://github.com/dhchenx
His research areas focus on Natural Language Processing, Knowledge Modeling, Big Data Analysis, and Artificial Intelligence in Medical Informatics.
GitHub Events
Total
- Watch event: 4
Last Year
- Watch event: 4
Committers
Last synced: almost 3 years ago
All Time
- Total Commits: 17
- Total Committers: 1
- Avg Commits per committer: 17.0
- Development Distribution Score (DDS): 0.0
Top Committers
| Name | Commits | |
|---|---|---|
| Donghua Chen | d****n@1****m | 17 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 2
-
Total downloads:
- pypi 54 last-month
-
Total dependent packages: 0
(may contain duplicates) -
Total dependent repositories: 2
(may contain duplicates) - Total versions: 6
- Total maintainers: 1
pypi.org: mmkit-features
A multimodal architecture to build multimodal knowledge graphs with flexible multimodal feature extraction and dynamic multimodal concept generation.
- Homepage: https://github.com/dhchenx/mmkit-features
- Documentation: https://mmkit-features.readthedocs.io/
- License: MIT
-
Latest release: 0.0.1a4
published almost 3 years ago
Rankings
Maintainers (1)
pypi.org: mmk-features
Extract and fuse multimodal features for deep learning
- Homepage: https://github.com/dhchenx/mmk-features
- Documentation: https://mmk-features.readthedocs.io/
- License: MIT
-
Latest release: 0.0.1.dev0
published about 4 years ago
Rankings
Maintainers (1)
Dependencies
- ffmpeg-python *
- h5py *
- librosa *
- nexusformat *
- numpy *
- opencv-python *
- requests *
- sklearn *
- spacy *
- tensorflow *
- torch *
- torchvision *
- tqdm *
- validators *