Data

Tools

Indexes

Applications

Experiments

Open Source Science

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

https://github.com/scrapy/scrapy

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (9.9%) to scientific vocabulary

Keywords

crawler crawling framework hacktoberfest python scraping web-scraping web-scraping-python
Last synced: 6 months ago · JSON representation

Repository

Scrapy, a fast high-level web crawling & scraping framework for Python.

Basic Info
  • Host: GitHub
  • Owner: scrapy
  • License: bsd-3-clause
  • Language: Python
  • Default Branch: master
  • Homepage: https://scrapy.org
  • Size: 27.4 MB
Statistics
  • Stars: 58,095
  • Watchers: 1,774
  • Forks: 11,028
  • Open Issues: 685
  • Releases: 43
Topics
crawler crawling framework hacktoberfest python scraping web-scraping web-scraping-python
Created almost 16 years ago · Last pushed 6 months ago
Metadata Files
Readme Changelog Contributing License Code of conduct Security Authors

README.rst 

|logo|

.. |logo| image:: https://raw.githubusercontent.com/scrapy/scrapy/master/docs/_static/logo.svg
   :target: https://scrapy.org
   :alt: Scrapy
   :width: 480px

|version| |python_version| |ubuntu| |macos| |windows| |coverage| |conda| |deepwiki|

.. |version| image:: https://img.shields.io/pypi/v/Scrapy.svg
   :target: https://pypi.org/pypi/Scrapy
   :alt: PyPI Version

.. |python_version| image:: https://img.shields.io/pypi/pyversions/Scrapy.svg
   :target: https://pypi.org/pypi/Scrapy
   :alt: Supported Python Versions

.. |ubuntu| image:: https://github.com/scrapy/scrapy/workflows/Ubuntu/badge.svg
   :target: https://github.com/scrapy/scrapy/actions?query=workflow%3AUbuntu
   :alt: Ubuntu

.. |macos| image:: https://github.com/scrapy/scrapy/workflows/macOS/badge.svg
   :target: https://github.com/scrapy/scrapy/actions?query=workflow%3AmacOS
   :alt: macOS

.. |windows| image:: https://github.com/scrapy/scrapy/workflows/Windows/badge.svg
   :target: https://github.com/scrapy/scrapy/actions?query=workflow%3AWindows
   :alt: Windows

.. |coverage| image:: https://img.shields.io/codecov/c/github/scrapy/scrapy/master.svg
   :target: https://codecov.io/github/scrapy/scrapy?branch=master
   :alt: Coverage report

.. |conda| image:: https://anaconda.org/conda-forge/scrapy/badges/version.svg
   :target: https://anaconda.org/conda-forge/scrapy
   :alt: Conda Version

.. |deepwiki| image:: https://deepwiki.com/badge.svg
   :target: https://deepwiki.com/scrapy/scrapy
   :alt: Ask DeepWiki

Scrapy_ is a web scraping framework to extract structured data from websites.
It is cross-platform, and requires Python 3.9+. It is maintained by Zyte_
(formerly Scrapinghub) and `many other contributors`_.

.. _many other contributors: https://github.com/scrapy/scrapy/graphs/contributors
.. _Scrapy: https://scrapy.org/
.. _Zyte: https://www.zyte.com/

Install with:

.. code:: bash

    pip install scrapy

And follow the documentation_ to learn how to use it.

.. _documentation: https://docs.scrapy.org/en/latest/

If you wish to contribute, see Contributing_.

.. _Contributing: https://docs.scrapy.org/en/master/contributing.html

Owner

  • Name: Scrapy project
  • Login: scrapy
  • Kind: organization

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 639
  • Total pull requests: 1,363
  • Average time to close issues: over 1 year
  • Average time to close pull requests: 8 months
  • Total issue authors: 347
  • Total pull request authors: 389
  • Average comments per issue: 4.08
  • Average comments per pull request: 2.26
  • Merged pull requests: 645
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 170
  • Pull requests: 584
  • Average time to close issues: 8 days
  • Average time to close pull requests: 5 days
  • Issue authors: 72
  • Pull request authors: 116
  • Average comments per issue: 1.28
  • Average comments per pull request: 1.59
  • Merged pull requests: 310
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • wRAR (125)
  • Gallaecio (52)
  • mohmad-null (26)
  • kmike (18)
  • Prometheus3375 (11)
  • Ehsan-U (11)
  • elacuesta (5)
  • GeorgeA92 (5)
  • pawelmhm (5)
  • starrify (4)
  • ddebernardy (4)
  • Urahara (4)
  • jtoallen (4)
  • lopuhin (4)
  • zuozhehao (3)
Pull Request Authors
  • wRAR (392)
  • Gallaecio (123)
  • Laerte (50)
  • Rotzbua (14)
  • jxlil (14)
  • mlmsmith (14)
  • GeorgeA92 (13)
  • Rohitkr117 (10)
  • Sintivrousai (10)
  • MehrazRumman (9)
  • elramen (9)
  • mery16q (9)
  • kumar-sanchay (9)
  • thalissonvs (9)
  • LucasSD (8)
Top Labels
Issue Labels
enhancement (219) bug (104) good first issue (68) docs (63) discuss (58) cleanup (45) CI (37) asyncio (27) help wanted (18) needs more info (17) media pipelines (17) logging (15) upstream issue (12) performance (11) http (10) https (9) security (7) link extraction (5) not reproducible (5) typing (5) Windows (5) contracts (4) HTTP/2 (3) backward-incompatible (2) stale (2) gsoc-candidate (2) patch available (1) macOS (1) waiting feedback (1) S3 (1)
Pull Request Labels
enhancement (197) CI (158) cleanup (136) bug (73) typing (70) docs (52) asyncio (46) spam (18) discuss (13) media pipelines (8) needs more info (7) hacktoberfest-accepted (7) backward-incompatible (5) logging (5) security (4) https (3) upstream issue (3) Windows (3) link extraction (3) in progress (3) contracts (2) help wanted (2) S3 (1) item loaders (1) waiting feedback (1) stale (1) http (1) macOS (1)

Packages

  • Total packages: 8
  • Total downloads:
    • pypi 2,096,674 last-month
  • Total docker downloads: 588,557
  • Total dependent packages: 142
    (may contain duplicates)
  • Total dependent repositories: 2,831
    (may contain duplicates)
  • Total versions: 161
  • Total maintainers: 8
  • Total advisories: 15
pypi.org: scrapy

A high-level Web Crawling and Web Scraping framework

  • Versions: 104
  • Dependent Packages: 136
  • Dependent Repositories: 2,753
  • Downloads: 2,096,401 Last month
  • Docker Downloads: 588,557
Rankings
Stargazers count: 0.0%
Forks count: 0.0%
Dependent packages count: 0.2%
Dependent repos count: 0.2%
Downloads: 0.3%
Average: 0.4%
Docker downloads count: 1.4%
Maintainers (4)
dangra kmike pablohoffman scrapy
Advisories (15)
Last synced: 6 months ago
conda-forge.org: scrapy

Scrapy is an open source and collaborative framework for extracting the data you need from websites in a fast, simple, yet extensible way.

  • Homepage: https://scrapy.org/
  • License: BSD-3-Clause-Clear
  • Latest release: 2.7.1
    published over 3 years ago
  • Versions: 31
  • Dependent Packages: 4
  • Dependent Repositories: 38
Rankings
Stargazers count: 0.2%
Forks count: 0.9%
Average: 4.9%
Dependent repos count: 5.9%
Dependent packages count: 12.5%
Last synced: 6 months ago
pypi.org: pylab-utils

python utility tools

  • Versions: 2
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 18 Last month
Rankings
Stargazers count: 0.0%
Forks count: 0.0%
Dependent packages count: 7.4%
Average: 9.5%
Downloads: 18.0%
Dependent repos count: 22.2%
Maintainers (1)
alannesta
Last synced: 6 months ago
anaconda.org: scrapy

Scrapy is an open source and collaborative framework for extracting the data you need from websites in a fast, simple, yet extensible way.

  • Homepage: https://scrapy.org
  • License: BSD-3-Clause
  • Latest release: 2.13.3
    published 6 months ago
  • Versions: 14
  • Dependent Packages: 2
  • Dependent Repositories: 38
Rankings
Stargazers count: 0.7%
Forks count: 2.5%
Average: 12.5%
Dependent packages count: 20.5%
Dependent repos count: 26.4%
Last synced: 6 months ago
pypi.org: scrapy-hls

scrapy integration for m3u8 files

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 9 Last month
Rankings
Stargazers count: 0.0%
Forks count: 0.0%
Dependent packages count: 7.4%
Average: 18.9%
Dependent repos count: 22.2%
Downloads: 64.7%
Maintainers (1)
alannesta
Last synced: 6 months ago
pypi.org: bf-scrapy-base

A high-level Web Crawling and Web Scraping framework

  • Versions: 3
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 112 Last month
Rankings
Dependent packages count: 8.6%
Average: 28.6%
Dependent repos count: 48.6%
Maintainers (1)
liuyu_59344
Last synced: 6 months ago
pypi.org: scrapy-qfm

A high-level Web Crawling and Web Scraping framework

  • Versions: 2
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 9 Last month
Rankings
Dependent packages count: 10.0%
Average: 33.2%
Dependent repos count: 56.4%
Maintainers (1)
qi2017
Last synced: 6 months ago
pypi.org: aminer-scrapy

A high-level Web Crawling and Web Scraping framework

  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 125 Last month
Rankings
Dependent packages count: 10.0%
Average: 38.1%
Dependent repos count: 66.2%
Maintainers (1)
coderflying
Last synced: 6 months ago

Dependencies

.github/workflows/checks.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
  • pre-commit/action v3.0.0 composite
.github/workflows/publish.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
  • pypa/gh-action-pypi-publish v1.6.4 composite
.github/workflows/tests-macos.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
.github/workflows/tests-ubuntu.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
.github/workflows/tests-windows.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
docs/requirements.txt pypi
  • sphinx ==5.0.2
  • sphinx-hoverxref ==1.1.1
  • sphinx-notfound-page ==0.8
  • sphinx-rtd-theme ==1.0.0
setup.py pypi
  • 23.8.0 *
  • Twisted >=18.9.0,<23.8.0
  • cryptography >=36.0.0
  • cssselect >=0.9.1
  • itemadapter >=0.1.0
  • itemloaders >=1.0.1
  • lxml >=4.4.1
  • packaging *
  • parsel >=1.5.0
  • protego >=0.1.15
  • pyOpenSSL >=21.0.0
  • queuelib >=1.4.2
  • service_identity >=18.1.0
  • setuptools *
  • tldextract *
  • w3lib >=1.17.0
  • zope.interface >=5.1.0
tests/requirements.txt pypi
  • attrs * test
  • bpython * test
  • brotli ==1.0.9 test
  • brotli * test
  • ipython * test
  • pyftpdlib * test
  • pytest * test
  • pytest-cov ==4.0.0 test
  • pytest-xdist * test
  • pywin32 * test
  • testfixtures * test
  • uvloop * test
  • zstandard * test