Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/ahmedshahriar/bd-ponno

Scrapy Mongodb Djongo integrated API that scrapes popular e-commerce sites (10+) from Bangladesh

https://github.com/ahmedshahriar/bd-ponno

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (4.3%) to scientific vocabulary

Keywords

django django-rest-framework djongo mongodb mongodb-atlas python python37 restful-api scrapy web-scraping
Last synced: 5 months ago · JSON representation

Repository

Scrapy Mongodb Djongo integrated API that scrapes popular e-commerce sites (10+) from Bangladesh

Basic Info
  • Host: GitHub
  • Owner: ahmedshahriar
  • License: apache-2.0
  • Language: Python
  • Default Branch: main
  • Homepage:
  • Size: 93.8 KB
Statistics
  • Stars: 6
  • Watchers: 1
  • Forks: 2
  • Open Issues: 0
  • Releases: 0
Topics
django django-rest-framework djongo mongodb mongodb-atlas python python37 restful-api scrapy web-scraping
Created almost 5 years ago · Last pushed over 1 year ago
Metadata Files
Readme License

README.md

bd-ponno

Scrapy MongoDB Django integrated API that scrapes popular e-commerce sites (10+) from Bangladesh.\ Added some extra functionalities for the API, ex-custom price, and product tag filter

The project is no longer maintained.

Built With

Django==3.1.7 django-cors-headers==3.7.0 django-filter==2.4.0 django-heroku==0.3.1 djangorestframework==3.12.2 djongo==1.3.4 Scrapy==2.4.1 scrapy-djangoitem==1.1.1 pymongo==3.11.3

Remarks

MongoDB Atlas is not a good option for djongo or remote web scraping, because djongo is just a connector for the underlying relational mapper with Django which causes additional delay, hence bad performance for remote scraping with Atlas. Also, you might run into lots of unintentional issues like - 'list' object has no attribute '_meta' while checking the product details view in the admin panel (ArrayReferenceField class (Category model)).

API preview

bd_ponno_products

Owner

  • Name: Ahmed Shahriar Sakib
  • Login: ahmedshahriar
  • Kind: user
  • Location: Ontario, Canada
  • Company: @criticalml-uw

Software Engineer, an expert in web scraping & automation, data analytics, and machine learning. Kaggle Master.

GitHub Events

Total
  • Watch event: 1
Last Year
  • Watch event: 1

Issues and Pull Requests

Last synced: 10 months ago

All Time
  • Total issues: 0
  • Total pull requests: 2
  • Average time to close issues: N/A
  • Average time to close pull requests: 2 minutes
  • Total issue authors: 0
  • Total pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 2
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 1
  • Average time to close issues: N/A
  • Average time to close pull requests: less than a minute
  • Issue authors: 0
  • Pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 1
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
  • ahmedshahriar (3)
Top Labels
Issue Labels
Pull Request Labels
wontfix (2)

Dependencies

requirements.txt pypi
  • Automat ==20.2.0
  • Django ==3.1.7
  • Jinja2 ==3.0.1
  • MarkupSafe ==2.0.1
  • Protego ==0.1.16
  • PyDispatcher ==2.0.5
  • PyHamcrest ==2.0.2
  • PyYAML ==5.4.1
  • Pygments ==2.8.0
  • Scrapy ==2.4.1
  • Twisted ==20.3.0
  • asgiref ==3.3.1
  • attrs ==20.3.0
  • certifi ==2021.5.30
  • cffi ==1.14.5
  • chardet ==4.0.0
  • colorama ==0.4.4
  • commonmark ==0.9.1
  • constantly ==15.1.0
  • coreapi ==2.3.3
  • coreschema ==0.0.4
  • coverage ==5.5
  • cryptography ==3.4.6
  • cssselect ==1.1.0
  • dataclasses ==0.8
  • dj-database-url ==0.5.0
  • django-cors-headers ==3.7.0
  • django-filter ==2.4.0
  • django-heroku ==0.3.1
  • djangorestframework ==3.12.2
  • djongo ==1.3.4
  • dnspython ==2.1.0
  • gunicorn ==20.1.0
  • hyperlink ==21.0.0
  • idna ==2.10
  • importlib-metadata ==4.5.0
  • incremental ==17.5.0
  • inflection ==0.5.1
  • itemadapter ==0.2.0
  • itemloaders ==1.0.4
  • itypes ==1.2.0
  • jmespath ==0.10.0
  • jsonschema ==3.2.0
  • lxml ==4.6.2
  • packaging ==20.9
  • parsel ==1.6.0
  • psycopg2 ==2.8.6
  • pyOpenSSL ==20.0.1
  • pyasn1 ==0.4.8
  • pyasn1-modules ==0.2.8
  • pycparser ==2.20
  • pymongo ==3.11.3
  • pyparsing ==2.4.7
  • pyrsistent ==0.17.3
  • python-dotenv ==0.17.0
  • pytz ==2021.1
  • queuelib ==1.5.0
  • requests ==2.25.1
  • rich ==9.12.0
  • ruamel.yaml ==0.17.9
  • ruamel.yaml.clib ==0.2.2
  • scrapy-djangoitem ==1.1.1
  • service-identity ==18.1.0
  • six ==1.15.0
  • sqlparse ==0.2.4
  • swagger-spec-validator ==2.7.3
  • typing-extensions ==3.7.4.3
  • uritemplate ==3.0.1
  • urllib3 ==1.26.5
  • w3lib ==1.22.0
  • whitenoise ==5.2.0
  • zipp ==3.4.1
  • zope.interface ==5.2.0