Data

Tools

Indexes

Applications

Experiments

Open Source Science

linear-relational-concepts

https://github.com/chanind/linear-relational-concepts

Science Score: 54.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (13.0%) to scientific vocabulary
Last synced: 10 months ago · JSON representation ·

Repository

Basic Info
  • Host: GitHub
  • Owner: chanind
  • License: mit
  • Language: Python
  • Default Branch: main
  • Size: 545 KB
Statistics
  • Stars: 3
  • Watchers: 2
  • Forks: 0
  • Open Issues: 1
  • Releases: 1
Created over 2 years ago · Last pushed almost 2 years ago
Metadata Files
Readme License Citation

README.md

Linear Relational Concepts

This is the code for experiments that accompany the paper: "Identifying Linear Relational Concepts in Large Language Models".

If you're interested in using Linear Relational Concepts (LRCs) or Linear Relational Embeddings (LREs) in your own work, check out the linear-relational library. Linear-relational contains the core reusable ideas from this paper, and is packaged in a Python library that can be installed with pip.

Setup

This project uses Poetry to manage dependencies. Make sure you have Poetry installed, and run:

poetry install

Running experiments

There are a number of experiments from the paper in the "linearrelationalconcepts/experiments" dir. These are meant to be imported and run in a Jupyter notebook or an interactive Python shell. If you want to run these as bash scripts, you'll need to build a wrapper yourself to do this.

For example, to run the benchmark_llama2 experiment, run the following:

```python from linearrelationalconcepts.experiments.benchmarkllama2 import benchmarkllama2

this will run the main benchmark with default values, using llama2 from huggingface

benchmarkresults = benchmarkllama2()

results are a dict, with each method as a key and IterationResults as values

for methodname, iterationresults in benchmarkresults.items(): print(f"{methodname} - classification accuracy: {iterationresults.accuracy}, causality: {iterationresults.causality}") ```

Development

This project uses Black for code formatting, Flake8 for linting, and Pytest for tests, and Mypy for type checking.

Run tests with: poetry run pytest

Citation

If you used this code in your research, please cite the following paper:

@article{chanin2023identifying, title={Identifying Linear Relational Concepts in Large Language Models}, author={David Chanin and Anthony Hunter and Oana-Maria Camburu}, journal={arXiv preprint arXiv:2311.08968}, year={2023} }

Owner

  • Name: David Chanin
  • Login: chanind
  • Kind: user
  • Location: London, UK
  • Company: UCL

Citation (CITATION.cff) 

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
  - family-names: "Chanin"
    given-names: "David"
  - family-names: "Hunter"
    given-names: "Anthony"
  - family-names: "Camburu"
    given-names: "Oana-Maria"
title: "Identifying Linear Relational Concepts in Large Language Models"
doi: 10.48550/arXiv.2311.08968
date-released: 2023-11-15
url: "https://arxiv.org/abs/2311.08968"

GitHub Events

Total
  • Watch event: 4
Last Year
  • Watch event: 4

Issues and Pull Requests

Last synced: over 1 year ago

All Time
  • Total issues: 0
  • Total pull requests: 3
  • Average time to close issues: N/A
  • Average time to close pull requests: 23 minutes
  • Total issue authors: 0
  • Total pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 2
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 3
  • Average time to close issues: N/A
  • Average time to close pull requests: 23 minutes
  • Issue authors: 0
  • Pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 2
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
  • chanind (4)
Top Labels
Issue Labels
Pull Request Labels

Dependencies

.github/workflows/ci.yaml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
  • snok/install-poetry v1 composite
poetry.lock pypi
  • aiohttp 3.9.3
  • aiosignal 1.3.1
  • async-timeout 4.0.3
  • attrs 23.2.0
  • black 23.12.1
  • certifi 2024.2.2
  • charset-normalizer 3.3.2
  • click 8.1.7
  • cmake 3.28.3
  • colorama 0.4.6
  • contourpy 1.2.0
  • cycler 0.12.1
  • dataclasses-json 0.6.4
  • exceptiongroup 1.2.0
  • filelock 3.13.1
  • flake8 6.1.0
  • fonttools 4.49.0
  • frozenlist 1.4.1
  • fsspec 2024.2.0
  • huggingface-hub 0.20.3
  • idna 3.6
  • iniconfig 2.0.0
  • isort 5.13.2
  • jinja2 3.1.3
  • joblib 1.3.2
  • kiwisolver 1.4.5
  • lightning-utilities 0.10.1
  • linear-relational 0.2.1
  • lit 17.0.6
  • markupsafe 2.1.5
  • marshmallow 3.20.2
  • matplotlib 3.8.3
  • mccabe 0.7.0
  • mpmath 1.3.0
  • multidict 6.0.5
  • mypy 1.8.0
  • mypy-extensions 1.0.0
  • networkx 3.2.1
  • numpy 1.26.4
  • nvidia-cublas-cu11 11.10.3.66
  • nvidia-cuda-cupti-cu11 11.7.101
  • nvidia-cuda-nvrtc-cu11 11.7.99
  • nvidia-cuda-runtime-cu11 11.7.99
  • nvidia-cudnn-cu11 8.5.0.96
  • nvidia-cufft-cu11 10.9.0.58
  • nvidia-curand-cu11 10.2.10.91
  • nvidia-cusolver-cu11 11.4.0.1
  • nvidia-cusparse-cu11 11.7.4.91
  • nvidia-nccl-cu11 2.14.3
  • nvidia-nvtx-cu11 11.7.91
  • packaging 23.2
  • pandas 2.2.0
  • pathspec 0.12.1
  • pillow 10.2.0
  • platformdirs 4.2.0
  • pluggy 1.4.0
  • protobuf 4.25.3
  • pycodestyle 2.11.1
  • pyflakes 3.1.0
  • pyparsing 3.1.1
  • pytest 7.4.4
  • python-dateutil 2.8.2
  • pytorch-lightning 2.2.0.post0
  • pytz 2024.1
  • pyyaml 6.0.1
  • regex 2023.12.25
  • requests 2.31.0
  • safetensors 0.4.2
  • scikit-learn 1.4.1.post1
  • scipy 1.12.0
  • seaborn 0.13.2
  • sentencepiece 0.1.99
  • setuptools 69.1.0
  • six 1.16.0
  • sympy 1.12
  • syrupy 4.6.1
  • threadpoolctl 3.3.0
  • tokenizers 0.15.2
  • tomli 2.0.1
  • torch 2.0.0
  • torchmetrics 1.3.1
  • tqdm 4.66.2
  • transformers 4.37.2
  • triton 2.0.0
  • typing-extensions 4.9.0
  • typing-inspect 0.9.0
  • tzdata 2024.1
  • urllib3 2.2.0
  • wheel 0.42.0
  • yarl 1.9.4
pyproject.toml pypi
  • black ^23.9.1 develop
  • flake8 ^6.1.0 develop
  • isort ^5.12.0 develop
  • mypy ^1.6.0 develop
  • pytest ^7.4.2 develop
  • syrupy ^4.5.0 develop
  • torch 2.0.0 develop
  • dataclasses-json ^0.6.1
  • linear-relational ^0.2.1
  • matplotlib ^3.8.1
  • protobuf ^4.24.4
  • python ^3.10
  • pytorch-lightning ^2.1.0
  • scikit-learn ^1.3.2
  • seaborn ^0.13.0
  • sentencepiece ^0.1.99
  • tqdm ^4.66.1
  • transformers ^4.34.0