https://github.com/chapzq77/personspiderengine

多线程爬取百度，搜狗，bing等浏览器检索的结果，结果保存在轻量级的数据库sqlite中

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (0.6%) to scientific vocabulary

Last synced: 10 months ago · JSON representation

Repository

多线程爬取百度，搜狗，bing等浏览器检索的结果，结果保存在轻量级的数据库sqlite中

Created almost 9 years ago · Last pushed almost 9 years ago

Readme

爬取百度，bing，搜狗等浏览器的检索的返回的网页
使用多线程爬取，多线程的类 Fetcher，是自己写的，可以替代threadpool
爬取的数据的存取，我使用的轻量级的数据库 sqlite
输入的形式 ``` ["person" : "张三", #文件中指定输入的人名 "engine" : ["baidu","sogou","bing"], #爬取的引擎 "path" : "/person_spider/zhangsan.db" # 存储的文件路径和文件名]

可以在task_list 中添加多个query

```
爬取说明

python Spider.py