Data

Tools

Indexes

Applications

Experiments

Open Source Science

inspect-ai

Inspect: A framework for large language model evaluations

https://github.com/ukgovernmentbeis/inspect_ai

Science Score: 44.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.8%) to scientific vocabulary
Last synced: 7 months ago · JSON representation ·

Repository

Inspect: A framework for large language model evaluations

Basic Info
  • Host: GitHub
  • Owner: UKGovernmentBEIS
  • License: mit
  • Language: Python
  • Default Branch: main
  • Homepage: https://inspect.aisi.org.uk/
  • Size: 136 MB
Statistics
  • Stars: 1,281
  • Watchers: 11
  • Forks: 286
  • Open Issues: 75
  • Releases: 0
Created over 2 years ago · Last pushed 8 months ago
Metadata Files
Readme Changelog License Citation

README.md

Welcome to Inspect, a framework for large language model evaluations created by the UK AI Security Institute.

Inspect provides many built-in components, including facilities for prompt engineering, tool usage, multi-turn dialog, and model graded evaluations. Extensions to Inspect (e.g. to support new elicitation and scoring techniques) can be provided by other Python packages.

To get started with Inspect, please see the documentation at https://inspect.aisi.org.uk/.

To work on development of Inspect, clone the repository and install with the -e flag and [dev] optional dependencies:

bash git clone https://github.com/UKGovernmentBEIS/inspect_ai.git cd inspect_ai pip install -e ".[dev]"

Optionally install pre-commit hooks via

bash make hooks

Run linting, formatting, and tests via

bash make check make test

If you use VS Code, you should be sure to have installed the recommended extensions (Python, Ruff, and MyPy). Note that you'll be prompted to install these when you open the project in VS Code.

Owner

  • Name: Department of Business, Energy and Industrial Strategy
  • Login: UKGovernmentBEIS
  • Kind: organization

Citation (CITATION.cff) 

cff-version: 1.2.0
title: "Inspect AI: Framework for Large Language Model Evaluations"
message: >-
  If you cite this software, please do so using the
  metadata from this file.
type: software
authors:
  - name: UK AI Security Institute
    website: "https://www.aisi.gov.uk/"
repository-code: "https://github.com/UKGovernmentBEIS/inspect_ai"
url: "https://inspect.aisi.org.uk/"
license: MIT
date-released: "2024-05-10"

Issues and Pull Requests

Last synced: 7 months ago

All Time
  • Total issues: 496
  • Total pull requests: 1,613
  • Average time to close issues: 9 days
  • Average time to close pull requests: about 24 hours
  • Total issue authors: 192
  • Total pull request authors: 155
  • Average comments per issue: 1.75
  • Average comments per pull request: 0.33
  • Merged pull requests: 1,218
  • Bot issues: 0
  • Bot pull requests: 33
Past Year
  • Issues: 396
  • Pull requests: 1,403
  • Average time to close issues: 9 days
  • Average time to close pull requests: about 21 hours
  • Issue authors: 153
  • Pull request authors: 128
  • Average comments per issue: 1.54
  • Average comments per pull request: 0.27
  • Merged pull requests: 1,056
  • Bot issues: 0
  • Bot pull requests: 30
View more stats
Top Authors
Issue Authors
  • jjallaire (36)
  • tadamcz (31)
  • JasonGross (25)
  • max-kaufmann (14)
  • andrew-aisi (14)
  • dragonstyle (11)
  • kaifronsdal (11)
  • rusheb (11)
  • us2547 (9)
  • sohaibimran7 (9)
  • jjallaire-aisi (8)
  • jeremyschlatter (8)
  • Lovkush-A (7)
  • skinnerjc (7)
  • tekumara (6)
Pull Request Authors
  • jjallaire (475)
  • dragonstyle (345)
  • jjallaire-aisi (180)
  • epatey (75)
  • lorenzoAtBEIS (34)
  • craigwalton-dsit (34)
  • dependabot[bot] (33)
  • art-dsit (31)
  • tadamcz (22)
  • max-kaufmann (20)
  • rusheb (18)
  • andrew-aisi (16)
  • sdtblckgov (11)
  • rasmusfaber (11)
  • aisi-inspect (11)
Top Labels
Issue Labels
web browser (5) bug (1)
Pull Request Labels
dependencies (33) web browser (1) python (1) enhancement (1) benchmark (1) javascript (1)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 1,939,719 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 128
  • Total maintainers: 2
pypi.org: inspect-ai

Framework for large language model evaluations

  • Versions: 128
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 1,939,719 Last month
Rankings
Dependent packages count: 9.6%
Average: 36.5%
Dependent repos count: 63.3%
Maintainers (2)
jjallaire aisi-inspect
Last synced: 7 months ago

Dependencies

.github/workflows/build.yml actions
  • actions/checkout v4 composite
  • actions/setup-python v4 composite
  • chartboost/ruff-action v1 composite
  • hynek/build-and-inspect-python-package v1 composite
.github/workflows/docs.yml actions
  • actions/checkout v4 composite
  • quarto-dev/quarto-actions/publish v2 composite
  • quarto-dev/quarto-actions/setup v2 composite
.github/workflows/pypi.yml actions
  • actions/checkout v4 composite
  • actions/setup-python v5 composite
  • pypa/gh-action-pypi-publish release/v1 composite
src/inspect_ai/_view/www/package-lock.json npm
  • @bcherny/json-schema-ref-parser 10.0.5-fork development
  • @jsdevtools/ono 7.1.3 development
  • @types/glob 7.2.0 development
  • @types/json-schema 7.0.15 development
  • @types/lodash 4.14.202 development
  • @types/minimatch 5.1.2 development
  • @types/node 20.11.20 development
  • @types/prettier 2.7.3 development
  • any-promise 1.3.0 development
  • argparse 2.0.1 development
  • balanced-match 1.0.2 development
  • brace-expansion 1.1.11 development
  • call-me-maybe 1.0.2 development
  • cli-color 2.0.3 development
  • concat-map 0.0.1 development
  • d 1.0.1 development
  • es5-ext 0.10.63 development
  • es6-iterator 2.0.3 development
  • es6-symbol 3.1.3 development
  • es6-weak-map 2.0.3 development
  • esniff 2.0.1 development
  • event-emitter 0.3.5 development
  • ext 1.7.0 development
  • fs.realpath 1.0.0 development
  • get-stdin 8.0.0 development
  • glob 7.2.3 development
  • glob-promise 4.2.2 development
  • inflight 1.0.6 development
  • inherits 2.0.4 development
  • is-extglob 2.1.1 development
  • is-glob 4.0.3 development
  • is-promise 2.2.2 development
  • js-yaml 4.1.0 development
  • json-schema-to-typescript 13.1.2 development
  • lodash 4.17.21 development
  • lru-queue 0.1.0 development
  • memoizee 0.4.15 development
  • minimatch 3.1.2 development
  • minimist 1.2.8 development
  • mkdirp 1.0.4 development
  • mz 2.7.0 development
  • next-tick 1.1.0 development
  • object-assign 4.1.1 development
  • once 1.4.0 development
  • path-is-absolute 1.0.1 development
  • prettier 2.8.8 development
  • thenify 3.3.1 development
  • thenify-all 1.6.0 development
  • timers-ext 0.1.7 development
  • type 2.7.2 development
  • type 1.2.0 development
  • undici-types 5.26.5 development
  • wrappy 1.0.2 development
  • htm 3.1.1
  • preact 10.19.6
src/inspect_ai/_view/www/package.json npm
  • json-schema-to-typescript ^13.1.2 development
  • htm ^3.1.1
  • preact ^10.20.1
examples/agents/langchain/requirements.txt pypi
  • inspect_ai *
  • langchain *
  • langchainhub *
  • openai *
  • wikipedia *
pyproject.toml pypi
requirements.txt pypi
  • beautifulsoup4 *
  • click *
  • debugpy *
  • docstring-parser *
  • fsspec *
  • httpx *
  • jsonlines *
  • nest_asyncio *
  • numpy *
  • platformdirs *
  • pydantic >=2
  • python-dotenv *
  • pyyaml *
  • rich *
  • s3fs >=2023
  • semver *
  • shortuuid *
  • tenacity *
  • typing_extensions *