TextDescriptives

A Python library for calculating a large variety of metrics from text(s) using spaCy v.3 pipeline components and extensions.

🔧 Installation

pip install textdescriptives

📰 News

• We now have a TextDescriptives-powered web-app so you can extract and downloads metrics without a single line of code! Check it out here
• Version 2.0 out with a new API, a new component, updated documentation, and tutorials! Components are now called by "textdescriptives/{metric_name}. New coherence component for calculating the semantic coherence between sentences. See the documentation for tutorials and more information!
  

⚡ Quick Start

Use extract_metrics to quickly extract your desired metrics. To see available methods you can simply run: ```python import textdescriptives as td td.getvalidmetrics()

{'quality', 'readability', 'all', 'descriptivestats', 'dependencydistance', 'posproportions', 'informationtheory', 'coherence'}

```

Set the spacy_model parameter to specify which spaCy model to use, otherwise, TextDescriptives will auto-download an appropriate one based on lang. If lang is set, spacy_model is not necessary and vice versa.

Specify which metrics to extract in the metrics argument. None extracts all metrics.

```py import textdescriptives as td

text = "The world is changed. I feel it in the water. I feel it in the earth. I smell it in the air. Much that once was is lost, for none now live who remember it."

will automatically download the relevant model (´encoreweb_lg´) and extract all metrics

df = td.extract_metrics(text=text, lang="en", metrics=None)

specify spaCy model and which metrics to extract

df = td.extractmetrics(text=text, spacymodel="encoreweb_lg", metrics=["readability", "coherence"]) ```

Usage with spaCy

To integrate with other spaCy pipelines, import the library and add the component(s) to your pipeline using the standard spaCy syntax. Available components are descriptive_stats, readability, dependency_distance, pos_proportions, coherence, and quality prefixed with textdescriptives/.

If you want to add all components you can use the shorthand textdescriptives/all.

```py import spacy import textdescriptives as td

load your favourite spacy model (remember to install it first using e.g. python -m spacy download en_core_web_sm)

nlp = spacy.load("encorewebsm") nlp.addpipe("textdescriptives/all") doc = nlp("The world is changed. I feel it in the water. I feel it in the earth. I smell it in the air. Much that once was is lost, for none now live who remember it.")

access some of the values

doc..readability doc..token_length ```

TextDescriptives includes convenience functions for extracting metrics from a Doc to a Pandas DataFrame or a dictionary.

py td.extract_dict(doc) td.extract_df(doc) | | text | firstordercoherence | secondordercoherence | pospropDET | pospropNOUN | pospropAUX | pospropVERB | pospropPUNCT | pospropPRON | pospropADP | pospropADV | pospropSCONJ | fleschreadingease | fleschkincaidgrade | smog | gunningfog | automatedreadabilityindex | colemanliauindex | lix | rix | nstopwords | alpharatio | meanwordlength | doclength | proportionellipsis | proportionbulletpoints | duplicatelinechrfraction | duplicateparagraphchrfraction | duplicate5-gramchrfraction | duplicate6-gramchrfraction | duplicate7-gramchrfraction | duplicate8-gramchrfraction | duplicate9-gramchrfraction | duplicate10-gramchrfraction | top2-gramchrfraction | top3-gramchrfraction | top4-gramchrfraction | symbol#towordratio | containslorem ipsum | passedqualitycheck | dependencydistancemean | dependencydistancestd | propadjacentdependencyrelationmean | propadjacentdependencyrelationstd | tokenlengthmean | tokenlengthmedian | tokenlengthstd | sentencelengthmean | sentencelengthmedian | sentencelengthstd | syllablespertokenmean | syllablespertokenmedian | syllablespertokenstd | ntokens | nuniquetokens | proportionuniquetokens | ncharacters | nsentences | | ---: | :------------------------ | --------------------: | ---------------------: | -----------: | ------------: | -----------: | ------------: | -------------: | ------------: | -----------: | -----------: | -------------: | ------------------: | -------------------: | ------: | ----------: | --------------------------: | -----------------: | ------: | ---: | -----------: | ----------: | ---------------: | ---------: | ------------------: | -----------------------: | --------------------------: | -------------------------------: | ----------------------------: | ----------------------------: | ----------------------------: | ----------------------------: | ----------------------------: | -----------------------------: | ----------------------: | ----------------------: | ----------------------: | ---------------------: | :------------------- | :------------------- | -----------------------: | ----------------------: | -------------------------------------: | ------------------------------------: | ----------------: | ------------------: | ---------------: | -------------------: | ---------------------: | ------------------: | -----------------------: | -------------------------: | ----------------------: | -------: | --------------: | -----------------------: | -----------: | ----------: | | 0 | The world is changed(...) | 0.633002 | 0.573323 | 0.097561 | 0.121951 | 0.0731707 | 0.170732 | 0.146341 | 0.195122 | 0.0731707 | 0.0731707 | 0.0487805 | 107.879 | -0.0485714 | 5.68392 | 3.94286 | -2.45429 | -0.708571 | 12.7143 | 0.4 | 24 | 0.853659 | 2.95122 | 41 | 0 | 0 | 0 | 0 | 0.232258 | 0.232258 | 0 | 0 | 0 | 0 | 0.0580645 | 0.174194 | 0 | 0 | False | False | 1.77524 | 0.553188 | 0.457143 | 0.0722806 | 3.28571 | 3 | 1.54127 | 7 | 6 | 3.09839 | 1.08571 | 1 | 0.368117 | 35 | 23 | 0.657143 | 121 | 5 |

📖 Documentation

TextDescriptives has a detailed documentation as well as a series of Jupyter notebook tutorials. All the tutorials are located in the docs/tutorials folder and can also be found on the documentation website.

| Documentation | | | -------------------------- | ---------------------------------------------------------------------------------- | | 📚 Getting started | Guides and instructions on how to use TextDescriptives and its features. | | 👩‍💻 Demo | A live demo of TextDescriptives. | | 😎 Tutorials | Detailed tutorials on how to make the most of TextDescriptives | | 📰 News and changelog | New additions, changes and version history. | | 🎛 API References | The detailed reference for TextDescriptive's API. Including function documentation | | 📄 Paper | The preprint of the TextDescriptives paper. |