syndisco
A lightweight, simple and specialized framework used for creating, storing, annotating and analyzing synthetic discussions between LLMs.
Science Score: 44.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (16.5%) to scientific vocabulary
Repository
A lightweight, simple and specialized framework used for creating, storing, annotating and analyzing synthetic discussions between LLMs.
Basic Info
Statistics
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
SynDisco: Automated experiment creation and execution using only LLM agents
A lightweight, simple and specialized framework used for creating, storing, annotating and analyzing synthetic discussions between Large Language Model (LLM) user-agents in the context of online discussions.
This framework is designed for academic use, mainly for simulating Social Science experiments with multiple participants. It is finetuned for heavy server-side use and multi-day computations with limited resources. It has been tested on both simulated debates and online fora.
⚠ Warning: Active Development
This project is currently in active development. The API is subject to change at any time without prior notice.
We recommend keeping an eye on updates and version releases if you're using this project in your applications. Any bug reports or feature requests are welcome!.
Usage
Have a look at the online documentation for high-level descriptions, API documentation and tutorials.
Features
Automated Experiment Generation
SynDisco generates a randomized set of discussion templates. With only a handful of configurations, the researcher can run hundreds or thousands of unique experiments.
Synthetic Group Discussion Generation
SynDisco accepts an arbitrarily large number of LLM user-agent profiles and possible Original Posts (OPs). Each experiment involves a random selection of these user-agents replying to a randomly selected OP. The researcher can determine how these participants behave, whether there is a moderator present and even how the turn-taking is determined.
Synthetic Annotation Generation with multiple annotators
The researcher can create multiple LLM annotator-agent profiles. Each of these annotators will process each generated discussion at the comment-level, and annotate according to the provided instruction prompt, enabling an arbitrary selection of metrics to be used.
Native Transformers support
The framework supports most Hugging Face Transformer models out-of-the-box. Support for models managed by other libraries can be easily achieved by extending a single class.
Native logging and fault tolerance
Since SynDisco is expected to possibly run for days at a time in remote servers, it keeps detailed logs both on-screen and on-disk. Should any experiment fail, the next one will be loaded with no intermittent delays. Results are intermittently saved to the disk, ensuring no data loss or corruption on even catastrophic errors.
Installation
You can download the package from PIP:
bash
pip install syndisco
Or build from source:
bash
git clone https://github.com/dimits-ts/syndisco.git
pip install -r requirements.txt
pip install .
If you want to contribute to the project, or modify the library's code you may use:
bash
git clone https://github.com/dimits-ts/syndisco.git
pip install -r requirements.dev.txt
pip install -e .
Owner
- Name: Dimitris Tsirmpas
- Login: dimits-ts
- Kind: user
- Repositories: 1
- Profile: https://github.com/dimits-ts
I like playing around with data and building stuff.
Citation (CITATION.cff)
cff-version: 1.2.0
title: SynDisco
message: >-
If you use this software, please cite it using the
metadata from this file.
type: software
authors:
- given-names: Dimitris
family-names: Tsirmpas
email: tsirbasdim@gmail.com
affiliation: Athens University of Economics and Business
orcid: 'https://orcid.org/0000-0002-5675-3939'
repository-code: 'https://github.com/dimits-ts/syndisco/'
abstract: >-
A lightweight, simple and specialized framework used for
creating, storing, annotating and analyzing synthetic
discussions between LLM users in the context of online
discussions.
This framework is designed for academic use, mainly for
simulating Social Science experiments with multiple
participants. It is finetuned for heavy server-side use
and multi-day computations with limited resources. It has
been tested on both simulated debates and online fora.
keywords:
- llms
- simulation
- dataset
- llm-annotation
- moderation
- online discussions
license: GPL-3.0
GitHub Events
Total
- Watch event: 2
- Delete event: 6
- Push event: 26
- Pull request event: 9
- Create event: 3
Last Year
- Watch event: 2
- Delete event: 6
- Push event: 26
- Pull request event: 9
- Create event: 3
Committers
Last synced: 9 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Dimitris Tsirmpas | t****m@g****m | 313 |
Issues and Pull Requests
Last synced: 8 months ago
All Time
- Total issues: 0
- Total pull requests: 11
- Average time to close issues: N/A
- Average time to close pull requests: less than a minute
- Total issue authors: 0
- Total pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.0
- Merged pull requests: 11
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 11
- Average time to close issues: N/A
- Average time to close pull requests: less than a minute
- Issue authors: 0
- Pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.0
- Merged pull requests: 11
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
- dimits-ts (15)