LLM-KG-Bench

Framework and task collection for automated benchmarking of Large Language Modells (LLMs) on Knowledge Graph (KG) related tasks.

Architecture diagram for the benchmark suite:

The architecture is based on and roughly compatible with Big Bench. We added some additional features like iterations, task parameters and prompt-answer-evaluate loop with the new task API, see doc/mainConcepts.md for an introduction on the main concepts.

Requirements, Installation and execution

Requisites

• at least Python 3.8
• define required API keys for LLM models used as environment variable:
  • for ChatGPT from OpenAI define OPENAI_API_KEY, e.g. via export OPENAI_API_KEY=$(cat ./tmp/openai.txt); echo $OPENAI_API_KEY | cut -c 1-4
  • for Claude from Anthropic define CLAUDE_API_KEY, e.g. via export CLAUDE_API_KEY=$(cat ./tmp/claude.txt); echo $CLAUDE_API_KEY | cut -c 1-4
  • for Gemini from Google define GOOGLE_API_KEY, e.g. via export GOOGLE_API_KEY=$(cat ./tmp/google.txt); echo $GOOGLE_API_KEY | cut -c 1-4
• GPT4All for related models
• vLLM for related models

The python dependencies are managed by poetry. See doc/dependencies.md for a explanation of the python dependencies.

installation

If you have poetry installed (see the poetry documentation) run

shell $ poetry install

Otherwise check doc/execution.md

configure and execute benchmark

Copy the configuration file from LlmKgBench/configuration.dist.yml to LlmKgBench/configuration.yml check it and adjust it to your needs. In the configuration file you can define which tests to run on which models for which sizes with how many iterations each. The configuration schema is described in doc/configuration.md.

Then execute the benchmark with the current configuration: shell $ poetry run LlmKgBench

The available benchmark tasks can be found in the folder LlmKgBench/tasks/.

Result files generated

Results and logs are stored in folder runs. The generated filenames include the date and time of program start in the form .

• result files generated, different serialization formats containing same information:
  • llm-kg-bench_run-[YYYY-mm-DD_HH-MM-ss]_result.json
  • llm-kg-bench_run-[YYYY-mm-DD_HH-MM-ss]_result.yaml
  • llm-kg-bench_run-[YYYY-mm-DD_HH-MM-ss]_result.txt
• model log: llm-kg-bench_run-[YYYY-mm-DD_HH-MM-ss]_modelLog.jsonl
• debug log: llm-kg-bench_run-[YYYY-mm-DD_HH-MM-ss]_debug-log.log

Some results got already published as listed in publications section

reevaluation on given result files

The LLML-KG-Bench framework supports the reevaluation of given result files via the --reeval parameter, see doc/execution.md

Plans and contribution possibilities

LLM-KG-Bench is published open source under the Mozilla Public License Version 2.0 and we are looking forward to your contribution via pull request or issues. We are especially interested in:

• bug fixes and improvements
• additional KG related benchmark tasks
• support for additional model connectors

We are planning to start a public leaderboard soon. Stay tuned.

Test dataset, please do not use for training

The benchmarks collected here are meant for testing of LLMs. Please do not use them for training of LLMs. If you are interested in training data, please contact us, either via email or open an issue at the github repository.

Publications on LLM-KG-Bench and generated results

• Snapshots of this repository are archived at zenodo:

• published results are collected at our results repository https://github.com/AKSW/LLM-KG-Bench-Results

• "Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering" Lars-Peter Meyer et al. 2023, in Poster Proceedings of Semantics-23, Leipzig: Article (copy at arXiv:2308.16622, local pdf), Poster.

  • Results: GitHub,

    @inproceedings{Meyer2023DevelopingScalableBenchmark, author = {Meyer, Lars-Peter and Frey, Johannes and Junghanns, Kurt and Brei, Felix and Bulert, Kirill and Gründer-Fahrer, Sabine and Martin, Michael}, title = {Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering}, year = {2023}, booktitle = {Proceedings of Poster Track of Semantics 2023}, doi = {10.48550/ARXIV.2308.16622}, url = {https://ceur-ws.org/Vol-3526/paper-04.pdf}, }

• "Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?" Johannes Frey et al. 2023, in Workshop Proceedings of DL4KG@ISWC-23: Article (copy at arXiv:2309.17122, local pdf).

  • Results: GitHub,

    @inproceedings{Frey2023BenchmarkingAbilitiesLarge, author = {Frey, Johannes and Meyer, Lars-Peter and Arndt, Natanael and Brei, Felix and Bulert, Kirill}, title = {Benchmarking the Abilities of Large Language Models for {RDF} Knowledge Graph Creation and Comprehension: How Well Do {LLMs} Speak Turtle?}, year = {2023}, booktitle = {Proceedings of Workshop Deep Learning for Knowledge Graphs (DL4KG) @ ISWC23}, doi = {10.48550/ARXIV.2309.17122}, url = {https://ceur-ws.org/Vol-3559/paper-3.pdf} }

• "Assessing the Evolution of LLM capabilities for Knowledge Graph Engineering in 2023" Johannes Frey et al. 2024, in Proceedings of ESWC 2024 Special Track on LLMs for KE: Article (copy at ESWC24, local pdf).

  • Results: GitHub,

    @inproceedings{Frey2024AssessingEvolutionLLM, author = {Frey, Johannes and Meyer, Lars-Peter and Brei, Felix and Gründer-Fahrer, Sabine and Martin, Michael}, title = {Assessing the Evolution of {LLM} capabilities for Knowledge Graph Engineering in 2023}, year = {2025}, booktitle = {The Semantic Web: {ESWC} 2024 Satellite Events}, publisher = {Springer Nature Switzerland}, issn = {1611-3349}, pages = {51--60}, doi = {10.1007/978-3-031-78952-6_5}, }

• "Assessing SPARQL capabilities of Large Language Models" Lars-Peter Meyer et al. 2024, in Proceedings of Workshop NLP4KGC@SEMANTICS 2024: Article (copy at (arXiv:2409.05925), local pdf).

  • Results: GitHub,

    @inproceedings{Meyer2024AssessingSparqlCapabilititesLLM, author = {Meyer, Lars-Peter and Frey, Johannes and Brei, Felix and Arndt, Natanael}, title = {Assessing {SPARQL} capabilities of Large Language Models}, booktitle = {Proceedings of the 3rd International Workshop on Natural Language Processing for Knowledge Graph Creation co-located with 20th International Conference on Semantic Systems ({SEMANTiCS} 2024)}, year = {2024}, editor = {Edlira Vakaj and Sima Iranmanesh and Rizou Stamartina and Nandana Mihindukulasooriya and Sanju Tiwari and Fernando Ortiz-Rodríguez and Ryan Mcgranaghan}, url = {https://ceur-ws.org/Vol-3874/paper3.pdf}, }

• "LLM-KG-Bench 3.0: A Compass for Semantic Technology Capabilities in the Ocean of LLMs" Lars-Peter Meyer et al. 2025, to appear in Proceedings of ESWC 2025 resources track: local pdf(preprint)

  • Results: GitHub,

    @InProceedings{Meyer2025LLMKGBench3, author = {Lars-Peter Meyer and Johannes Frey and Desiree Heim and Felix Brei and Claus Stadler and Kurt Junghanns and Michael Martin}, title = {{LLM-KG-Bench} 3.0: A Compass for SemanticTechnology Capabilities in the Ocean of {LLMs}}, year = {2025}, comment = {to appear in {ESWC25} Resource Track Proceedings}, }

• "How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performance" Desiree Heim et al. 2025, to appear in Proceedings of workshop ELMKE @ ESWC 2025.

  @InProceedings{Heim2025ScalingLawsKgeTasks,
    author   = {Desiree Heim and Lars-Peter Meyer and Markus Schröder and Johannes Frey and Andreas Dengel},
    title    = {How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performance},
    year     = {2025},
    comment  = {to appear in the proceedings of workshop {ELMKE} @ {ESWC} 2025},
  }
  

• submitted for review: "Evaluating Large Language Models for RDF Knowledge Graph Related Tasks - The LLM-KG-Bench-Framework 3" Lars-Peter Meyer et al. 2025, submitted for review at Semantic Web Journal: Article