Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (6.0%) to scientific vocabulary
Last synced: 11 months ago
·
JSON representation
Repository
Basic Info
- Host: GitHub
- Owner: dytttf
- License: bsd-3-clause
- Language: Python
- Default Branch: main
- Size: 126 KB
Statistics
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Created over 4 years ago
· Last pushed 11 months ago
Metadata Files
Readme
License
README.md
openget
A Spider FrameWork.
Installation
You can install openget by simply doing:
pip install openget
pip install git+https://github.com/dytttf/openget.git@main
pip install git+https://github.com/dytttf/openget.git@dev
Usage
Very Important Thing
This line must be the first line in code
from openget.spiders import *
命令行支持
- 生成配置文件样例
shell # 环境变量类型的配置文件 python -m openget --gen_env_example - 查看配置文件内容
shell # 环境变量类型的配置文件 python -m openget --show_env_config
About Environment Variable
优先级
- $(pwd)/.env
- ~/.openget/.env
docker build
shell
cd docekr
bash build.sh
TODO
- ftp download
- auto add task
- statis crawl speed
- browser support: zhipin.com tmall.com
- sqlite.db rename
- use cmd to create spider template
- 金融许可证 卡死问题处理
- 环境变量定义优化
- 新项目创建优化
- 极简模式、完整模式 减少包的依赖
Owner
- Name: Dytttf
- Login: dytttf
- Kind: user
- Location: China BeiJing
- Website: http://www.dytttf.com
- Repositories: 4
- Profile: https://github.com/dytttf
A Python Coder :)
GitHub Events
Total
- Watch event: 1
- Delete event: 3
- Push event: 15
- Pull request event: 11
Last Year
- Watch event: 1
- Delete event: 3
- Push event: 15
- Pull request event: 11
Committers
Last synced: almost 3 years ago
Top Committers
| Name | Commits | |
|---|---|---|
| dytttf | d****f@f****m | 16 |
Committer Domains (Top 20 + Academic)
foxmail.com: 1
Issues and Pull Requests
Last synced: 11 months ago
All Time
- Total issues: 0
- Total pull requests: 23
- Average time to close issues: N/A
- Average time to close pull requests: 1 minute
- Total issue authors: 0
- Total pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.13
- Merged pull requests: 18
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 7
- Average time to close issues: N/A
- Average time to close pull requests: 1 minute
- Issue authors: 0
- Pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.0
- Merged pull requests: 5
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
- dytttf (23)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 38 last-month
- Total dependent packages: 0
- Total dependent repositories: 0
- Total versions: 10
- Total maintainers: 1
pypi.org: openget
A Spider FrameWork
- Homepage: https://github.com/dytttf/openget
- Documentation: https://openget.readthedocs.io/
- License: BSD
-
Latest release: 0.3.5
published over 1 year ago
Rankings
Dependent packages count: 6.6%
Downloads: 22.5%
Average: 25.9%
Forks count: 30.5%
Dependent repos count: 30.6%
Stargazers count: 39.1%
Maintainers (1)
Last synced:
11 months ago
Dependencies
pyproject.toml
pypi
- Sphinx ^4.3.0 develop
- cx-oracle ^8.3.0 develop
- flask ^2.0.2 develop
- ipython ^7.29.0 develop
- isort ^5.10.1 develop
- pyftpdlib ^1.5.6 develop
- requests-ftp ^0.3.1 develop
- better-exceptions ^0.3.3
- crontab ^0.23.0
- gevent ^21.8.0
- httpx ^0.20.0
- mysqlclient ^2.0.3
- oss2 ^2.15.0
- pymysql ^1.0.2
- python ^3.8
- redis ^3.5.3
- tqdm ^4.62.3
- urllib3 ^1.26.7
- user-agent2 ^2021.11.1
setup.py
pypi
- better-exceptions *
- gevent *
- httpx *
- mysqlclient *
- pymysql *
- redis >=3.0.0
- tqdm *
docker/Dockerfile
docker
- python 3.8 build