Data

Tools

Indexes

Applications

Experiments

Open Source Science

mdr

A python library detect and extract listing data from HTML page.

https://github.com/scrapinghub/mdr

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ✓
    DOI references
    Found 1 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: arxiv.org, acm.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (9.4%) to scientific vocabulary

Keywords

data-science
Last synced: 11 months ago · JSON representation

Repository

A python library detect and extract listing data from HTML page.

Basic Info
  • Host: GitHub
  • Owner: scrapinghub
  • Language: C
  • Default Branch: master
  • Size: 434 KB
Statistics
  • Stars: 108
  • Watchers: 24
  • Forks: 29
  • Open Issues: 6
  • Releases: 0
Topics
data-science
Created about 12 years ago · Last pushed about 9 years ago
Metadata Files
Readme Changelog

README.rst 

===
MDR
===

.. image:: https://travis-ci.org/scrapinghub/mdr.svg?branch=master
    :target: https://travis-ci.org/scrapinghub/mdr

MDR is a library detect and extract listing data from HTML page. It implemented base on the `Finding and Extracting Data Records from Web Pages `_ but
change the similarity to tree alignment proposed by `Web Data Extraction Based on Partial Tree Alignment `_ and `Automatic Wrapper Adaptation by Tree Edit Distance Matching `_.


Requires
========

``numpy`` and ``scipy`` must be installed to build this package.

Usage
=====

Detect listing data
~~~~~~~~~~~~~~~~~~~

MDR assume the data record close to the elements has most text nodes::

    [1]: import requests
    [2]: from mdr import MDR
    [3]: mdr = MDR()
    [4]: r = requests.get('http://www.yelp.co.uk/biz/the-ledbury-london')
    [5]: candidates, doc = mdr.list_candidates(r.text.encode('utf8'))
    ...

    [8]: [doc.getpath(c) for c in candidates[:10]]
     ['/html/body/div[2]/div[3]/div[2]/div/div[1]/div[1]/div[2]/div[1]/div[2]/ul',
     '/html/body/div[2]/div[3]/div[2]/div/div[1]/div[2]',
     '/html/body/div[2]/div[3]/div[2]/div/div[1]/div[2]/div[2]',
     '/html/body/div[2]/div[3]/div[1]/div/div[4]/div[1]/div/div[1]/div/div[2]/div[1]/div[1]/div',
     '/html/body/div[2]/div[3]/div[1]/div/div[4]/div[2]/div/div[3]',
     '/html/body/div[2]/div[3]/div[1]/div/div[4]/div[1]/div/div[2]/ul/li[2]/div/div/ul',
     '/html/body/div[2]/div[3]/div[2]/div/div[1]/div[1]/div[2]/div[1]',
     '/html/body/div[2]/div[3]/div[2]/div/div[1]/div[2]/div[2]/div[1]/table/tbody',
     '/html/body/div[2]',
     '/html/body/div[2]/div[4]/div/div[1]']

Extract data record
~~~~~~~~~~~~~~~~~~~

MDR can find the repetiton patterns by using tree matching under certain candidate DOM tree, then it builds a mapping from HTML element to other matched elements of the DOM tree.

Used with annotation (optional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can annotate the seed elements with any tools (e.g. scrapely_) you like, then mdr will be able to find the other matched elements on the page.

e.g. you can find this demo page here_. the colored data in first row are annotated manually, the rest are extracted by MDR.

Author
======

Terry Peng 

License
=======

MIT

.. _scrapely: https://github.com/scrapy/scrapely
.. _here: http://ibc.scrapinghub.com/tmp/h.html

Owner

  • Name: Scrapinghub
  • Login: scrapinghub
  • Kind: organization
  • Location: The Internet

Turn web content into useful data

GitHub Events

Total
Last Year

Committers

Last synced: almost 3 years ago

All Time
  • Total Commits: 23
  • Total Committers: 2
  • Avg Commits per committer: 11.5
  • Development Distribution Score (DDS): 0.043
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
tpeng p****o@g****m 22
Shane Evans s****s@g****m 1

Issues and Pull Requests

Last synced: about 1 year ago

All Time
  • Total issues: 6
  • Total pull requests: 2
  • Average time to close issues: about 1 month
  • Average time to close pull requests: N/A
  • Total issue authors: 2
  • Total pull request authors: 1
  • Average comments per issue: 2.0
  • Average comments per pull request: 0.0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • dportabella (4)
  • whatsdis (2)
Pull Request Authors
  • dportabella (2)
Top Labels
Issue Labels
Pull Request Labels

Dependencies

requirements.txt pypi
  • lxml *
  • numpy *
  • scipy *
setup.py pypi
  • lxml *