https://github.com/caskade-automation/llm-capability-generation
Dataset of a study on the use of LLMs for generating a capability ontology
https://github.com/caskade-automation/llm-capability-generation
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (8.8%) to scientific vocabulary
Repository
Dataset of a study on the use of LLMs for generating a capability ontology
Basic Info
- Host: GitHub
- Owner: CaSkade-Automation
- License: mit
- Language: Python
- Default Branch: main
- Size: 244 KB
Statistics
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
LLM-Capability-Generation
A study on the use of LLMs for generating a capability ontology. This repository contains the complete data set, including all prompts and results, of the paper "On the Use of Large Language Models to Generate Capability Ontologies" by Luis Miguel Vieira da Silva, Aljosha Köcher, Felix Gehlhoff and Alexander Fay. A preprint of the paper is available on arXiv: https://arxiv.org/abs/2404.17524
Repository structure
capability-models: This folder contains both the manually modeled as well as the LLM-generated capability models. The manually modeled ones serve as a benchmark to validate all generated models against. Inside capability-models/generated, there is a substructure of folders for the LLMs used, the temperature values tested as well as for the different prompt techniques used. To give one example: capability-models/generated/claude/temp-0/0_zero-shot contains all models generated by Claude with a temperature of 0 and the zero-shot prompt. All other subfolders are analogous. Please note that we only tested a restricted set of prompts with temp = 1 and these were not used in the paper.
examples: This folder contains examples used in the one-shot and few-shot prompts. For every example, there is a natural-language task description as well as an ontology.
metaprompts: Contains the three meta prompts used, i.e., zero-shot, one-shot and few-shot. Please note that the meta prompts contain placeholders that are filled to obtain the actual prompts.
prompts: Contains the actual prompts that were sent to ChatGPT and Claude. These prompts are obtained by using the python script inside the prompts folder. It takes the three meta prompts and adds the examples as well as task descriptions to create individual prompts for each meta prompt and task
shapes: Contains the SHACL shapes used to evaluate the generated results against certain criteria that were defined beforehand
task-descriptions: This folder contains the seven task descriptions that were used to generate capabilities. For each task description, meta prompt and LLM used, there is a result inside of capability-models/generated.
Owner
- Name: CaSkade
- Login: CaSkade-Automation
- Kind: organization
- Repositories: 3
- Profile: https://github.com/CaSkade-Automation
Capability- and Skill-based Automation
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: over 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0