https://github.com/alan-turing-institute/readabilipy

A simple HTML content extractor in Python. Can be run as a wrapper for Mozilla's Readability.js package or in pure-python mode.

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
○
.zenodo.json file
○
DOI references
○
Academic publication links
✓
Committers with academic emails
2 of 13 committers (15.4%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (14.3%) to scientific vocabulary

Keywords

hut23 hut23-134 python readability

Keywords from Contributors

hut23-612 hut23-448 hut23-190 hut23-1304 data-safe-haven archival sequences interactive projection hacking

Last synced: 6 months ago · JSON representation

Repository

A simple HTML content extractor in Python. Can be run as a wrapper for Mozilla's Readability.js package or in pure-python mode.

Basic Info

Host: GitHub
Owner: alan-turing-institute
License: mit
Language: HTML
Default Branch: main
Homepage:
Size: 653 KB

Statistics

Stars: 337
Watchers: 16
Forks: 41
Open Issues: 15
Releases: 3

Topics

hut23 hut23-134 python readability

Created over 7 years ago · Last pushed about 1 year ago

Metadata Files

Readme Changelog License

ReadabiliPy

ReadabiliPy contains a Python wrapper for Mozilla's Readability.js Node.js package, as well as article extraction routines written in pure Python.

This package augments the output of Readability.js to also return a list of plain text representations of article paragraphs.

ReadabiliPy comes with a handy command line application: readabilipy.

Installation

To use the Readability.js wrapper you need to have a working Node.js installation of version 14 or higher. Make sure to install Node.js before installing this package, as this ensures Readability.js will be installed. If you only want to use the Python-based article extraction, you do not need to install Node.js.

ReadabiliPy can be installed simply from PyPI:

$ pip install readabilipy

Note that to update to a new version of Readability.js you can simply reinstall ReadabiliPy.

Usage

ReadabiliPy can be used either as a command line application or as a Python library.

Command line application

The readabilipy command line application can be used to extract an article from an HTML source file.

For example, if you have the article saved as input.html in the current directory then you can run:

$ readabilipy -i ./input.html -o article.json

The extracted article can then be found in the article.json file. By default ReadabiliPy will use the Readability.js functionality to extract the article, provided this is available. If instead you'd like to use the Python-based extraction, run:

$ readabilipy -p -i ./input.html -o article.json

The complete help text of the command line application is as follows:

``` $ readabilipy -h usage: readabilipy [-h] -i INPUTFILE -o OUTPUTFILE [-c] [-n] [-p] [-V]

Extract article data from a HTML file using either Mozilla's Readability.js package or a simplified python-only alternative.

optional arguments: -h, --help show this help message and exit -i INPUTFILE, --input-file INPUTFILE Path to input file containing HTML. -o OUTPUTFILE, --output-file OUTPUTFILE Path to file to output the article data to as JSON. -c, --content-digests Add a 'data-content-digest' attribute containing a SHA256-based digest of the element's contents to each HTML element in the plaincontent output. -n, --node-indexes Add a 'data-node-index' attribute containing a hierarchical representation of the element's position in the HTML structure each HTML element in the plaincontent output. -p, --use-python-parser Use the pure-python 'plain_html' parser included in this project rather than Mozilla's Readability.js. -V, --version Show version and exit ```

Library

ReadabiliPy can also be used as a Python package. The main routine is called simple_json_from_html_string and expects the HTML article as a string. Here is an example of extracting an article after downloading the page using requests:

```python

import requests from readabilipy import simplejsonfromhtmlstring req = requests.get('https://en.wikipedia.org/wiki/Readability') article = simplejsonfromhtmlstring(req.text, use_readability=True) ```

Note that you need to use the flag use_readability=True to use Readability.js, otherwise the Python-based extraction is used.

The simple_json_from_html_string function returns a dictionary with the following fields:

title: The article title
byline: Author information
content: A simplified HTML representation of the article, with all article text contained in paragraph elements.
plain_content: A "plain" version of the simplified Readability.js article HTML present in the content field. This attempts to retain only the plain text content of the article, while preserving the HTML structure.
plain_text: A list containing plain text representations of each paragraph (<p>) or list (<ol> or <ul>) present in the simplified Readability.js article HTML in the content field. Each paragraph or list is represented as a single string. List strings look like "* item 1, * item 2, * item 3," for both ordered and unordered lists (note the trailing ,).

Note further that:

All fields are guaranteed to be present. If individual fields are missing from the output of Readability.js, the value of these fields will be None. If no article data is returned by Readability.js, the value of all fields will be None.
All text in the plain_content and plain_text fields is encoded as unicode normalised using the "NFKC" normal form. This normal form is used to try and ensure as much as possible that things that appear visually the same are encoded with the same unicode representation (the K part) and characters are represented as a single composite character where possible (the C part).
An optional content_digests flag can be passed to the Python wrapper. When this is set to True, each HTML element in the plain_content field has a data-content-digest attribute, which holds the SHA-256 hash of its plain text content. For "leaf" nodes (containing only plain text in the output), this is the SHA-256 hash of their plain text content. For nodes containing other nodes, this is the SHA-256 hash of the concatenated SHA-256 hashes of their child nodes.
An optional node_indexes flag can be passed to the Python wrapper. When this is set to True, each HTML element in the plain_content field has a data-node-indexes attribute, which holds a hierarchical index describing the location of element within the plain_content HTML structure.
An optional use_readability flag can be passed to the Python wrapper. When this is set to True, Mozilla's Readability.js will be used as the parser. If it is set to False then the pure-python parser in plain_html.py will be used instead.

The second top-level function exported by ReadabiliPy is simple_tree_from_html_string. This returns a cleaned, parsed HTML tree of the article as a BeautifulSoup object.

Notes

License: MIT License, see the LICENSE file.

If you encounter any issues or have any suggestions for improvement, please open an issue on Github. You're helping to make this project better for everyone!

Owner

Name: The Alan Turing Institute
Login: alan-turing-institute
Kind: organization
Email: info@turing.ac.uk

Website: https://turing.ac.uk
Repositories: 477
Profile: https://github.com/alan-turing-institute

The UK's national institute for data science and artificial intelligence.

GitHub Events

Total

Create event: 5
Issues event: 3
Release event: 1
Watch event: 95
Delete event: 1
Issue comment event: 5
Push event: 8
Pull request review event: 8
Pull request review comment event: 6
Pull request event: 7
Fork event: 3

Last Year

Create event: 5
Issues event: 3
Release event: 1
Watch event: 95
Delete event: 1
Issue comment event: 5
Push event: 8
Pull request review event: 8
Pull request review comment event: 6
Pull request event: 7
Fork event: 3

Committers

Last synced: 9 months ago

All Time

Total Commits: 424
Total Committers: 13
Avg Commits per committer: 32.615
Development Distribution Score (DDS): 0.597

Past Year

Commits: 17
Committers: 3
Avg Commits per committer: 5.667
Development Distribution Score (DDS): 0.176

Top Committers

Name	Email	Commits
Ed Chalstrey	e**y@M**k	171
James Robinson	j**n@g**m	138
Gertjan van den Burg	g**g@g**m	53
Martin O'Reilly	d**r@m**t	38
Nelson Liu	n**u@n**e	12
dependabot[bot]	4****]	3
Alexander Ryabov	a**v@n**m	2
James Robinson	j**n@l**k	2
Luka Lodrant	l**t@g**m	1
Giovanni Garifo	g**o@o**m	1
hanzalajamash	h**h@g**m	1
eric	e**c@g**r	1
Inzamam Anwar	i**m@c**e	1

Committer Domains (Top 20 + Academic)

csms.ae: 1 gupi.fr: 1 laptop-ati0233.corp.turing.ac.uk: 1 nvidia.com: 1 nelsonliu.me: 1 mtspace.net: 1 mac-ati0379.corp.turing.ac.uk: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 46
Total pull requests: 64
Average time to close issues: 3 months
Average time to close pull requests: 2 months
Total issue authors: 19
Total pull request authors: 12
Average comments per issue: 1.52
Average comments per pull request: 1.13
Merged pull requests: 63
Bot issues: 0
Bot pull requests: 3

Past Year

Issues: 3
Pull requests: 4
Average time to close issues: N/A
Average time to close pull requests: 25 days
Issue authors: 3
Pull request authors: 3
Average comments per issue: 2.33
Average comments per pull request: 1.25
Merged pull requests: 4
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

jemrobinson (21)
martintoreilly (5)
sgibson91 (2)
ducnguyenphanhoai (2)
cayolblake (2)
EngTurtle (1)
swetepete (1)
SunLnx (1)
fernand0 (1)
westurner (1)
malicialab (1)
econaxis (1)
kjoshi (1)
cmdcam (1)
orange391224 (1)

Pull Request Authors

jemrobinson (27)
martintoreilly (12)
edwardchalstrey1 (11)
GjjvdBurg (3)
dependabot[bot] (3)
haron (3)
erpic (2)
nelson-liu (2)
lodrantl (2)
InzamamAnwar (1)
giovannigarifo (1)
hanzalajamash (1)

Top Labels

Issue Labels

bug (12) future (7) feature (7) not a bug (1)

Pull Request Labels

dependencies (3) bug (1)

Packages

Total packages: 1
Total downloads:
- pypi 633,319 last-month
Total docker downloads: 752

Total dependent packages: 3
Total dependent repositories: 7
Total versions: 2
Total maintainers: 1

pypi.org: readabilipy

Python wrapper for Mozilla's Readability.js

Homepage: https://github.com/alan-turing-institute/ReadabiliPy
Documentation: https://readabilipy.readthedocs.io/
License: MIT
Latest release: 0.3.0
published about 1 year ago

Versions: 2
Dependent Packages: 3
Dependent Repositories: 7
Downloads: 633,319 Last month
Docker Downloads: 752

Rankings

Dependent packages count: 2.3%

Docker downloads count: 3.5%

Downloads: 4.6%

Average: 5.0%

Dependent repos count: 5.6%

Stargazers count: 6.5%

Forks count: 7.8%

Maintainers (1)

jemrobinson

Last synced: 6 months ago

Dependencies

readabilipy/javascript/package.json npm

jsdom >=12.2.0
minimist ^1.2.3

.github/workflows/lint.yml actions

actions/checkout v3 composite
actions/setup-python v4 composite

.github/workflows/test.yml actions

actions/checkout v3 composite
actions/setup-node v3 composite
actions/setup-python v4 composite

benchmarks/Dockerfile docker

python 3 build

setup.py pypi

https://github.com/alan-turing-institute/readabilipy

Science Score: 23.0%

Keywords

Keywords from Contributors

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

ReadabiliPy

Installation

Usage

Command line application

Library

Notes

Owner

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: readabilipy

Rankings

Maintainers (1)

Dependencies