Data

Tools

Indexes

Applications

Experiments

Open Source Science

llm4qual

HuggingFace + Langchain based framework for Qualitative Research

https://github.com/msamogh/llm4qual

Science Score: 44.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (17.5%) to scientific vocabulary
Last synced: 10 months ago · JSON representation ·

Repository

HuggingFace + Langchain based framework for Qualitative Research

Basic Info
  • Host: GitHub
  • Owner: msamogh
  • License: mit
  • Language: Python
  • Default Branch: main
  • Homepage:
  • Size: 81.1 KB
Statistics
  • Stars: 2
  • Watchers: 1
  • Forks: 1
  • Open Issues: 0
  • Releases: 0
Created over 2 years ago · Last pushed over 2 years ago
Metadata Files
Readme License Citation

README.md

llm4qual

Welcome to the llm4qual repository, your go-to toolkit for enhancing qualitative research with Large Language Models (LLMs). Our mission is to harness the power of LLMs to streamline and support various tasks in qualitative research.

Supported Workflows

As of Jan 2024, LLM4Qual supports a single workflow called llm-proxy, which uses LLMs to simulate human annotators in text labeling tasks. But we plan to add support for additional workflows.

llm-proxy

The llm-proxy workflow is designed to act as a stand-in for human annotators in text-annotation tasks. By leveraging LLMs, llm-proxy provides a sophisticated and automated approach to evaluating qualitative data across multiple dimensions.

Features

  • Dataset Loading: Integrate seamlessly with Langchain + HuggingFace to import your qualitative datasets.
  • Prompt Templating: Organize your annotation prompts with ease using our structured YAML file templates, compatible with Langchain.
  • Few-Shot Examples: Incorporate few-shot learning by providing examples within your YAML files to enhance the LLM's annotation accuracy.
  • Evaluation Suite Setup: Utilize helper classes and functions to establish a HuggingFace EvaluationSuite tailored to your research needs.
  • Result Reporting: After analysis, neatly save and report the evaluation outcomes for further review and action.

Getting Started

To get started with llm4qual, follow these simple steps:

Prerequisites

Ensure you have the latest versions of Langchain and HuggingFace libraries installed. You can install these using pip:

bash pip install langchain huggingface_hub

Installation

  1. Clone the LLM4Qual repository to your local machine: bash git clone https://github.com/msamogh/llm4qual.git cd llm4qual

  2. Navigate to the llm-proxy directory and install any necessary dependencies: bash cd llm-proxy poetry shell poetry install

  3. Install Transformers4Qual: bash git clone https://github.com/msamogh/transformers4qual.git pip install -e transformers4qual

  4. Install Evaluate4Qual: bash git clone https://github.com/msamogh/evaluate4qual.git pip install -e evaluate4qual

Usage

  1. Load Your Dataset: Use Langchain to load your dataset into the framework.
  2. Create Prompt Templates: Define your annotation dimensions and write prompt templates in YAML format.
  3. Few-Shot Examples: Add few-shot examples to your YAML files to guide the LLM in annotation tasks.
  4. Initialize Evaluation Suite: With the provided helper classes, set up your EvaluationSuite.
  5. Run Evaluation: Execute the evaluation and collect results.
  6. Save & Report: Utilize our reporting tools to save and visualize the analysis outcomes.

License

Distributed under the MIT License. See LICENSE for more information.

Contact

For any additional questions or comments, please reach out to msamogh@gmail.com or submit an issue.

Owner

  • Name: Amogh Mannekote
  • Login: msamogh
  • Kind: user
  • Location: Bellevue, WA
  • Company: Amazon

Dialogue Systems | LLMs | HCI | Education | Open Source

Citation (CITATION.cff) 

cff-version: 1.0.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Mannekote"
  given-names: "Amogh"
  orcid: "https://orcid.org/0000-0003-4066-5456"
title: "LLM4Qual"
version: 1.0.0
date-released: 2024-01-30
url: "https://github.com/msamogh/llm4qual"

GitHub Events

Total
  • Watch event: 1
Last Year
  • Watch event: 1

Issues and Pull Requests

Last synced: about 2 years ago

All Time
  • Total issues: 0
  • Total pull requests: 1
  • Average time to close issues: N/A
  • Average time to close pull requests: 1 minute
  • Total issue authors: 0
  • Total pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 1
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 1
  • Average time to close issues: N/A
  • Average time to close pull requests: 1 minute
  • Issue authors: 0
  • Pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 1
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
  • msamogh (1)
Top Labels
Issue Labels
Pull Request Labels

Dependencies

poetry.lock pypi
  • aiohttp 3.9.1
  • aiosignal 1.3.1
  • annotated-types 0.6.0
  • anyio 4.2.0
  • async-timeout 4.0.3
  • attrs 23.2.0
  • certifi 2023.11.17
  • charset-normalizer 3.3.2
  • colorama 0.4.6
  • dataclasses-json 0.6.3
  • datasets 2.16.1
  • dill 0.3.7
  • evaluate 0.4.1
  • exceptiongroup 1.2.0
  • filelock 3.13.1
  • frozenlist 1.4.1
  • fsspec 2023.10.0
  • greenlet 3.0.3
  • huggingface-hub 0.20.2
  • idna 3.6
  • jsonpatch 1.33
  • jsonpointer 2.4
  • langchain 0.1.0
  • langchain-community 0.0.11
  • langchain-core 0.1.9
  • langsmith 0.0.79
  • marshmallow 3.20.2
  • multidict 6.0.4
  • multiprocess 0.70.15
  • mypy-extensions 1.0.0
  • numpy 1.26.3
  • packaging 23.2
  • pandas 2.1.4
  • pyarrow 14.0.2
  • pyarrow-hotfix 0.6
  • pydantic 2.5.3
  • pydantic-core 2.14.6
  • python-dateutil 2.8.2
  • pytz 2023.3.post1
  • pyyaml 6.0.1
  • requests 2.31.0
  • responses 0.18.0
  • six 1.16.0
  • sniffio 1.3.0
  • sqlalchemy 2.0.25
  • tenacity 8.2.3
  • tqdm 4.66.1
  • typing-extensions 4.9.0
  • typing-inspect 0.9.0
  • tzdata 2023.4
  • urllib3 2.1.0
  • xxhash 3.4.1
  • yarl 1.9.4
pyproject.toml pypi