colibri-core

Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way. At the core is the tool ``colibri-patternmodeller`` whi ch allows you to build, view, manipulate and query pattern models.

https://github.com/proycon/colibri-core

Science Score: 49.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • CITATION.cff file
  • codemeta.json file
    Found codemeta.json file
  • .zenodo.json file
    Found .zenodo.json file
  • DOI references
    Found 6 DOI reference(s) in README
  • Academic publication links
    Links to: zenodo.org
  • Committers with academic emails
  • Institutional organization owner
  • JOSS paper metadata
  • Scientific vocabulary similarity
    Low similarity (16.4%) to scientific vocabulary

Keywords

c-plus-plus computational-linguistics corpus library linguistics ngram ngrams nlp pattern-recognition python skipgram text-processing

Keywords from Contributors

learning-algorithm historical-linguistics language-detection language-identification
Last synced: 4 months ago · JSON representation

Repository

Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way. At the core is the tool ``colibri-patternmodeller`` whi ch allows you to build, view, manipulate and query pattern models.

Basic Info
Statistics
  • Stars: 129
  • Watchers: 11
  • Forks: 20
  • Open Issues: 9
  • Releases: 29
Topics
c-plus-plus computational-linguistics corpus library linguistics ngram ngrams nlp pattern-recognition python skipgram text-processing
Created over 12 years ago · Last pushed about 1 year ago
Metadata Files
Readme Changelog License Authors Codemeta

README.md

Colibri Core

GitHub C++ build GitHub Python build DOI GitHub release Project Status: Active – The project has reached a stable, usable state and is being actively developed. PyPi

by Maarten van Gompel, proycon@anaproy.nl, Radboud University Nijmegen

Licensed under GPLv3 (See http://www.gnu.org/licenses/gpl-3.0.html)

Colibri Core is software to quickly and efficiently count and extract patterns from large corpus data, to extract various statistics on the extracted patterns, and to compute relations between the extracted patterns. The employed notion of pattern or construction encompasses the following categories:

  • n-gram -- n consecutive words
  • skipgram -- An abstract pattern of predetermined length with one or multiple gaps (of specific size).
  • flexgram -- An abstract pattern with one or more gaps of variable-size.

N-gram extraction may seem fairly trivial at first, with a few lines in your favourite scripting language, you can move a simple sliding window of size n over your corpus and store the results in some kind of hashmap. This trivial approach however makes an unnecessarily high demand on memory resources, this often becomes prohibitive if unleashed on large corpora. Colibri Core tries to minimise these space requirements in several ways:

  • Compressed binary representation -- Each word type is assigned a numeric class, which is encoded in a compact binary format in which highly frequent classes take less space than less frequent classes. Colibri core always uses this representation rather than a full string representation, both on disk and in memory.
  • Informed iterative counting -- Counting is performed more intelligently by iteratively processing the corpus in several passes and quickly discarding patterns that won't reach the desired occurrence threshold.

Skipgram and flexgram extraction are computationally more demanding but have been implemented with similar optimisations. Skipgrams are computed by abstracting over n-grams, and flexgrams in turn are computed either by abstracting over skipgrams, or directly from n-grams on the basis of co-occurrence information (mutual pointwise information).

At the heart of the sofware is the notion of pattern models. The core tool, to be used from the command-line, is colibri-patternmodeller which enables you to build pattern models, generate statistical reports, query for specific patterns and relations, and manipulate models.

A pattern model is simply a collection of extracted patterns (any of the three categories) and their counts from a specific corpus. Pattern models come in two varieties:

  • Unindexed Pattern Model -- The simplest form, which simply stores the patterns and their count.
  • Indexed Pattern Model -- The more informed form, which retains all indices to the original corpus, at the cost of more memory/diskspace.

The Indexed Pattern Model is much more powerful, and allows more statistics and relations to be inferred.

The generation of pattern models is optionally parametrised by a minimum occurrence threshold, a maximum pattern length, and a lower-boundary on the different types that may instantiate a skipgram (i.e. possible fillings of the gaps).

Technical Details

Colibri Core is available as a collection of standalone command-line tools, as a C++ library, and as a Python library.

Please consult the full documentation at https://proycon.github.io/colibri-core

Installation

Python binding

For the Colibri Core Python library, just install using:

pip install colibricore

We strongly recommend you use a Virtual Environment for this. Do note that this is only available for unix-like systems, Windows is not supported.

Installation from packages

For the command-line tools, check if your distribution has a package available. There are packages for Alpine Linux (apk add colibri-core) and for macOS with homebrew (brew tap fbkarsdorp/homebrew-lamachine && brew install colibri-core). Note that these do not contain the Python binding!

Otherwise you will need to either use the container image or to build and install from source.

Installation from source

If no packages are available, you will need to compile from source or use the container build (e.g. Docker) as explained later on.

In order to do so, you need a sane build environment, install the necessary dependencies for your distribution:

For Debian/Ubuntu::

$ sudo apt-get install make gcc g++ pkg-config autoconf-archive libtool autotools-dev libbz2-dev zlib1g-dev libtar-dev python3 python3-dev cython3

For RedHat-based systems (run as root)::

```

yum install pkgconfig libtool autoconf automake autoconf-archive make gcc gcc-c++ libtar libtar-devel python3 python3-devel zlib zlib-devel python3-pip bzip2 bzip2-devel cython3

```

For macOS with homebrew:

$ brew install autoconf automake libtool autoconf-archive python3 pkg-config

Then clone this repository and install as follows:

$ bash bootstrap $ ./configure $ make $ sudo make install

Container usage

The Colibri Core command-line tools are also available as an OCI/Docker container.

A pre-made container image can be obtained from Docker Hub as follows:

docker pull proycon/colibri-core

You can also build a container image yourself as follows, make sure you are in the root of this repository:

docker build -t proycon/colibri-core .

This builds the latest stable release, if you want to use the latest development version from the git repository instead, do:

docker build -t proycon/colibri-core --build-arg VERSION=development .

Run the frog container interactively as follows, it will dump you into a shell where the various command line tools are available:

docker run -t -i proycon/colibri-core

Add the -v /path/to/your/data:/data parameter if you want to mount your data volume into the container at /data.

Demo

Colibri Core Demo

Publication

This software is extensively described in the following peer-reviewed publication:

van Gompel, M and van den Bosch, A (2016)
Efficient n-gram, Skipgram and Flexgram Modelling with Colibri Core.
*Journal of Open Research Software*
4: e30, DOI: http://dx.doi.org/10.5334/jors.105

Access the publication here and please cite it if you make use of Colibri Core in your work.

Owner

  • Name: Maarten van Gompel
  • Login: proycon
  • Kind: user
  • Location: Eindhoven, the Netherlands
  • Company: KNAW Humanities Cluster & CLST, Radboud University

Research software engineer - NLP - AI - 🐧 Linux & open-source enthusiast - 🐍 Python/ 🌊C/C++ / 🦀 Rust / 🐚 Shell - 🔐 InfoSec - https://git.sr.ht/~proycon

CodeMeta (codemeta.json)

{
  "@context": [
    "https://doi.org/10.5063/schema/codemeta-2.0",
    "http://schema.org",
    "https://w3id.org/software-types"
  ],
  "@type": "SoftwareSourceCode",
  "identifier": "colibricore",
  "name": "Colibri Core",
  "version": "2.5.9",
  "description": "Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way. ",
  "license": "https://spdx.org/licenses/GPL-3.0-only",
  "url": "https://proycon.github.io/colibri-core",
  "author": [
    {
      "@id": "https://orcid.org/0000-0002-1046-0006",
      "@type": "Person",
      "givenName": "Maarten",
      "familyName": "van Gompel",
      "email": "proycon@anaproy.nl",
      "affiliation": {
        "@id": "https://www.ru.nl/cls",
        "@type": "Organization",
        "name": "Centre for Language Studies",
        "url": "https://www.ru.nl/cls",
        "parentOrganization": {
          "@id": "https://www.ru.nl",
          "name": "Radboud University",
          "@type": "Organization",
          "url": "https://www.ru.nl",
          "location": {
            "@type": "Place",
            "name": "Nijmegen"
          }
        }
      }
    }
  ],
  "programmingLanguage": [
    {
      "@type": "ComputerLanguage",
      "identifier": "c++",
      "name": "C++"
    },
    {
      "@type": "ComputerLanguage",
      "identifier": "cython",
      "name": "Cython"
    }
  ],
  "operatingSystem": [
    "Linux",
    "BSD",
    "macOS"
  ],
  "codeRepository": "https://github.com/proycon/colibri-core",
  "softwareRequirements": [],
  "softwareHelp": "https://proycon.github.io/colibri-core/doc/",
  "readme": "https://github.com/proycon/colibri-core/blob/master/README.md",
  "issueTracker": "https://github.com/proycon/colibri-core/issues",
  "contIntegration": "https://travis-ci.org/proycon/colibri-core",
  "releaseNotes": "https://github.com/proycon/colibri-core/releases",
  "developmentStatus": "https://www.repostatus.org/#active",
  "keywords": [
    "nlp",
    "natural language processing",
    "ngrams",
    "skipgrams",
    "pattern recognition",
    "language modelling"
  ],
  "referencePublication": [
    {
      "@id": "https://dx.doi.org/10.5334/jors.105",
      "@type": "TechArticle",
      "name": "Efficient n-gram, Skipgram and Flexgram Modelling with Colibri Core",
      "author": [
        "Maarten van Gompel",
        "Antal van den Bosch"
      ],
      "isPartOf": {
        "@type": "PublicationIssue",
        "datePublised": "2016",
        "name": "Journal of Open Research Software",
        "issue": "4"
      },
      "url": "https://dx.doi.org/10.5334/jors.105"
    }
  ],
  "dateCreated": "2013-09-15",
  "targetProduct": [
    {
      "@type": "CommandLineApplication",
      "name": "colibri-classencode",
      "executableName": "colibri-classencode",
      "description": "Encodes a plain text corpus to a binary encoded corpus and a class file"
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-classdecode",
      "executableName": "colibri-classdecode",
      "description": "Decodes a binary encoded corpus and a class file to a plain text corpus"
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-patternmodeller",
      "executableName": "colibri-patternmodeller",
      "description": "Extract, model and compare recurring patterns (n-grams, skipgrams, flexgrams) and their frequencies in text corpus data. This is the main tool of Colibri Core."
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-cooc",
      "executableName": "colibri-cooc",
      "description": "Computes co-occurrence statistics (absolute co-cooccurrence or pointwise mutual information) between patterns in a corpus"
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-ngrams",
      "executableName": "colibri-ngrams",
      "description": "Extract n-grams of a particular size by moving a sliding window over the corpus. This is a high-level convenience script over underlying tools."
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-freqlist",
      "executableName": "colibri-freqlist",
      "description": "Extract n-grams (and optionally skipgrams) with their counts from one or more plain-text corpus files. This is a high-level convenience script over underlying tools."
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-coverage",
      "executableName": "colibri-coverage",
      "description": "Computes the coverage of training/background corpus on a particular test/foreground corpus, i.e how many of the patterns in the test corpus were found during training, how many tokens are covered, and how is this all distributed?. This is a high-level convenience script over underlying tools."
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-findpatterns",
      "executableName": "colibri-findpatterns",
      "description": "Find patterns in corpus data based on a presupplied list of patterns (one per line). This is a high-level convenience script over underlying tools."
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-histogram",
      "executableName": "colibri-histogram",
      "description": "Computes a histogram for ngram occurrences (and optionally skipgrams) in the corpus. This is a high-level convenience script over underlying tools."
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-loglikelihood",
      "executableName": "colibri-loglikelihood",
      "description": "Compares the frequency of patterns between two or more corpus files (plain text) by computing log likelihood, following the methodology of Rayson and Garside (2000), Comparing corpora using frequency profiling. In proceedings of the workshop on Comparing Corpora, held in conjunction with the 38th annual meeting of the Association for Computational Linguistics (ACL 2000). 1-8 October 2000, Hong Kong, pp. 1 - 6: http://www.comp.lancs.ac.uk/~paul/publications/rg_acl2000.pdf. This is a high-level convenience script over underlying tools."
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-ngramstats",
      "executableName": "colibri-ngramstats",
      "description": "Computes a summary report on the count of ngrams (and optionally skipgrams) in the corpus. This is a high-level convenience script over underlying tools."
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-queryngrams",
      "executableName": "colibri-queryngrams",
      "description": "Interactive command line  tool to  n-grams with their counts from one or more plain-text corpus files. This is a high-level convenience script over underlying tools."
    },
    {
      "@type": "CommandLineApplication",
      "name": "colibri-reverseindex",
      "executableName": "colibri-reverseindex",
      "description": "Computes and prints reverse index of the corpus, for each token position in the corpus, all patterns that start at that position are shown. This is a high-level convenience script over underlying tools."
    }
  ]
}

GitHub Events

Total
  • Watch event: 7
  • Push event: 34
Last Year
  • Watch event: 7
  • Push event: 34

Committers

Last synced: 8 months ago

All Time
  • Total Commits: 1,403
  • Total Committers: 7
  • Avg Commits per committer: 200.429
  • Development Distribution Score (DDS): 0.092
Past Year
  • Commits: 54
  • Committers: 2
  • Avg Commits per committer: 27.0
  • Development Distribution Score (DDS): 0.037
Top Committers
Name Email Commits
Maarten van Gompel p****n@a****l 1,274
Ko van der Sloot K****t@z****l 98
Ko van der Sloot K****t@l****l 22
louis l****s@n****l 6
Prasasto Adi p****i 1
Vladislavs Dovgalecs u****t@i****v 1
Maarten van Gompel p****n@c****l 1
Committer Domains (Top 20 + Academic)

Issues and Pull Requests

Last synced: 4 months ago

All Time
  • Total issues: 37
  • Total pull requests: 8
  • Average time to close issues: 3 months
  • Average time to close pull requests: 3 months
  • Total issue authors: 10
  • Total pull request authors: 4
  • Average comments per issue: 1.41
  • Average comments per pull request: 0.5
  • Merged pull requests: 6
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Top Authors
Issue Authors
  • proycon (25)
  • naiaden (2)
  • andreasvc (2)
  • svetlana21 (1)
  • KasperFyhn (1)
  • mkrump (1)
  • redadmiral (1)
  • mikkokotila (1)
  • ngawangtrinley (1)
  • manrock007 (1)
Pull Request Authors
  • kosloot (5)
  • usptact (1)
  • manrock007 (1)
  • prasastoadi (1)
Top Labels
Issue Labels
bug (14) enhancement (11) low priority (6) investigate (6) PRIORITY (5) question (3) expired (2) wontfix (1) waiting (1) in progress (1) help wanted (1)
Pull Request Labels

Packages

  • Total packages: 12
  • Total downloads: unknown
  • Total dependent packages: 0
    (may contain duplicates)
  • Total dependent repositories: 0
    (may contain duplicates)
  • Total versions: 43
  • Total maintainers: 1
proxy.golang.org: github.com/proycon/colibri-core
  • Versions: 30
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 6.4%
Average: 6.7%
Dependent repos count: 6.9%
Last synced: 4 months ago
alpine-edge: colibri-core

NLP tool and C++ library to quickly and efficiently count and extract patterns from large corpus data

  • Versions: 2
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Average: 14.2%
Dependent packages count: 14.6%
Stargazers count: 19.6%
Forks count: 22.7%
Maintainers (1)
Last synced: 4 months ago
alpine-edge: colibri-core-libs

NLP tool and C++ library to quickly and efficiently count and extract patterns from large corpus data (libraries)

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Average: 14.2%
Dependent packages count: 14.6%
Stargazers count: 19.6%
Forks count: 22.7%
Maintainers (1)
Last synced: 4 months ago
alpine-edge: colibri-core-dev

NLP tool and C++ library to quickly and efficiently count and extract patterns from large corpus data (development files)

  • Versions: 2
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Average: 14.2%
Dependent packages count: 14.6%
Stargazers count: 19.6%
Forks count: 22.7%
Maintainers (1)
Last synced: 4 months ago
alpine-v3.21: colibri-core-dev

NLP tool and C++ library to quickly and efficiently count and extract patterns from large corpus data (development files)

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
Last synced: 4 months ago
alpine-v3.19: colibri-core-dev

NLP tool and C++ library to quickly and efficiently count and extract patterns from large corpus data (development files)

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
Last synced: 4 months ago
alpine-v3.21: colibri-core

NLP tool and C++ library to quickly and efficiently count and extract patterns from large corpus data

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
Last synced: 4 months ago
alpine-v3.22: colibri-core

NLP tool and C++ library to quickly and efficiently count and extract patterns from large corpus data

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
Last synced: 4 months ago
alpine-v3.22: colibri-core-dev

NLP tool and C++ library to quickly and efficiently count and extract patterns from large corpus data (development files)

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
Last synced: 4 months ago
alpine-v3.20: colibri-core-dev

NLP tool and C++ library to quickly and efficiently count and extract patterns from large corpus data (development files)

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
Last synced: 4 months ago
alpine-v3.20: colibri-core

NLP tool and C++ library to quickly and efficiently count and extract patterns from large corpus data

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
Last synced: 4 months ago
alpine-v3.19: colibri-core

NLP tool and C++ library to quickly and efficiently count and extract patterns from large corpus data

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
Last synced: 4 months ago

Dependencies

setup.py pypi
  • Cython *