tlda

Tools for language data analysis (R package)

https://github.com/lsoenning/tlda

Repository

Tools for language data analysis (R package)

README.Rmd

---
output: github_document
---



```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
```

# tlda: Tools for language data analysis 


[![CRAN status](https://www.r-pkg.org/badges/version/tlda)](https://CRAN.R-project.org/package=tlda)


This package includes a number of utility functions and resources for analyzing linguistic data. At the moment, the focus is on corpus-linguistic **dispersion analysis** (see Gries 2020; Sönning 2025), which quantifies how widely and/or evenly and item is distributed across corpus parts. This kind of analysis requires two variables:

- the *subfrequencies*, i.e. the number of times the item is found in each corpus part
- the *size of the corpus parts* (number of word tokens)

The package includes functions that allow you to calculate seven different parts-based dispersion measures, including their frequency-adjusted version. Subfrequencies and part sizes can be supplied in two forms, either as vectors or as a term-document matrix. For some measures, different formulas are found in the literature, and the user can choose among them. The following indices are implemented:

- *Range* (Keniston 1920)
- *D* (Juilland & Chang-Rodriguez 1964)
- *D~2~* (Carroll 1970)
- *S* (Rosengren 1971)
- *D~P~* (Gries 2008; modification: Egbert et al. 2020)
- *D~A~* (Burch et al. 2017)
- *D~KL~* (Gries 2020, 2021)



## Usage

The function `disp()` calculates seven dispersion measures based on two vectors:

- `subfreq` a set of subfrequencies, i.e. the number of occurrences of the item in each corpus part
- `partsize` a vector with the size of the corpus parts

The argument `directionality` controls the scaling of the scores:

- `conventional`: higher values reflect a **more even** distribution
- `gries`: higher values reflect a **less even** distribution

The function prints information about the directionality of scaling and details about the formula used.

As an example, we will use data from Lyne's (1985) classic study and consider the distribution of the French lemma ALLEMAND across the ten (nearly) equal-sized parts ('Tenths') of his corpus of French business correspondence. The part sizes are taken from Figure 1 (p. 85) and the subfrequencies from Appendix I (p. 299).

```{r}
library(tlda)

x <- c(2, 0, 1, 1, 3, 0, 3, 0, 0, 0)
y <- c(8143, 8058, 8271, 8125, 7959, 7941, 8146, 8001, 8003, 7930)

disp(
  subfreq = x,
  partsize = y,
  directionality = "conventional"
)
```

If we prefer the reversed scaling used by Gries (2008), we can change the value of the argument `directionality`, like so:

```{r}
disp(
  subfreq = x,
  partsize = y,
  directionality = "gries"
)
```

To calculate dispersion for multiple items, it makes sense to provide the data in the form of a term-document matrix. In this tabular arrangement,

- each row represents an item
- each column represents a corpus part.

A number of example data sets are shipped with the `tlda` package, including `biber150_ice_gb`, a term-document matrix recording the text-level subfrequencies for Biber et al.'s (2016) 150 lexical items in ICE-GB (Nelson et al. 2002). Importantly, the first row gives the number of word tokens in the text file. This is an excerpt from the matrix:

```{r}
biber150_ice_gb[1:5, 1:5]
```

The function `disp_tdm()` calculates seven dispersion measures for each item in the matrix. The output is therefore also a matrix. In this function, the arguments `subfreq` and `partsize` are replaced by the two following:

- `tdm` A term-document matrix, where rows represent items and columns corpus parts; it must also contain a row giving the size of the corpus parts (first or last row in the TDM)
- `row_partsize` Character string indicating which row in the TDM contains the size of the corpus parts.

The following calculates dispersion scores for the first ten items in the term-document matrix (rounded to two decimal places).

```{r}
disp_tdm(
    tdm = biber150_ice_gb[1:11,], 
    row_partsize = "first",
    digits = 2,
    print_score = TRUE,
    verbose = FALSE)
```



## Installation

Install the latest `tlda` release from CRAN:

```r
install.packages("tlda")
```

Or the development version of `tlda` from [GitHub](https://github.com/) with:

``` r
# install.packages("pak")
pak::pak("lsoenning/tlda")
```


## References

Biber, Douglas, Randi Reppen, Erin Schnur & Romy Ghanem. 2016. On the (non)utility of Juilland’s *D* to measure lexical dispersion in large corpora. *International Journal of Corpus Linguistics* 21(4). 439–464. doi: [10.1075/ijcl.21.4.01bib](https://doi.org/10.1075/ijcl.21.4.01bib) 

Burch, Brent, Jesse Egbert & Douglas Biber. 2017. Measuring and interpreting lexical dispersion in corpus linguistics. *Journal of Research Design and Statistics in Linguistics and Communication Science* 3(2). 189–216. doi: [10.1558/jrds.33066](https://doi.org/10.1558/jrds.33066)

Carroll, John B. 1970. An alternative to Juilland’s usage coefficient for lexical frequencies and a proposal for a standard frequency index. *Computer Studies in the Humanities and Verbal Behaviour* 3(2). 61–65. doi: [10.1002/j.2333-8504.1970.tb00778.x](https://doi.org/10.1002/j.2333-8504.1970.tb00778.x)

Egbert, Jesse, Brent Burch & Douglas Biber. 2020. Lexical dispersion and corpus design. *International Journal of Corpus Linguistics* 25(1). 89–115. doi: [10.1075/ijcl.18010.egb](https://doi.org/10.1075/ijcl.18010.egb)

Gries, Stefan Th. 2008. Dispersions and adjusted frequencies in corpora. *International Journal of Corpus Linguistics* 13(4). 403–437. doi: [10.1075/ijcl.13.4.02gri](https://doi.org/10.1075/ijcl.13.4.02gri)

Gries, Stefan Th. 2020. Analyzing dispersion. In Magali Paquot & Stefan Th. Gries (eds.), *A practical handbook of corpus linguistics*, 99–118. New York: Springer. doi: [10.1007/978-3-030-46216-1_5](https://doi.org/10.1007/978-3-030-46216-1_5)

Gries, Stefan Th. 2021. A new approach to (key) keywords analysis: Using frequency, and now also dispersion. *Research in Corpus Linguistics* 9(2). 1−33. doi: [10.32714/ricl.09.02.02](https://doi.org/10.32714/ricl.09.02.02)

Juilland, Alphonse G. & Eugenio Chang-Rodriguez. 1964. *Frequency dictionary of Spanish words.* The Hague: Mouton de Gruyter. doi: [10.1515/9783112415467](https://doi.org/10.1515/9783112415467)

Keniston, Hayward. 1920. Common words in Spanish. *Hispania* 3(2). 85–96. doi: [10.2307/331305](https://doi.org/10.2307/331305)

Lijffijt, Jefrey & Stefan Th. Gries. 2012. Correction to Stefan Th. Gries’ ‘Dispersions and adjusted frequencies in corpora’. *International Journal of Corpus Linguistics* 17(1). 147–149. doi: [10.1075/ijcl.17.1.08lij](https://doi.org/10.1075/ijcl.17.1.08lij)

Lyne, Anthony A. 1985. *The vocabulary of French business correspondence*. Paris: Slatkine-Champion.

Nelson, Gerald, Sean Wallis & Bas Aarts. 2002. *Exploring Natural Language: Working with the British Component of the International Corpus of English*. Amsterdam: John Benjamins. doi: [10.1075/veaw.g29](https://doi.org/10.1075/veaw.g29)

Rosengren, Inger. 1971. The quantitative concept of language and its relation to the structure of frequency dictionaries. *Etudes de linguistique appliquee (Nouvelle Serie)* 1. 103–127.

Sönning, Lukas. 2025. Advancing our understanding of dispersion measures in corpus research. *Corpora*. 

Owner

GitHub Events

Packages