Data

Tools

Indexes

Applications

Experiments

Open Source Science

crawler-course-example

A fully automated web scraping solution that includes scheduling automatic launches through GitHub Actions, using Puppeteer with Node.js to collect data, and storing the data on GitHub Pages for public access.

https://github.com/pulipulichen/crawler-course-example

Science Score: 67.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 5 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (4.4%) to scientific vocabulary

Keywords

puli-portfolio
Last synced: 10 months ago · JSON representation ·

Repository

A fully automated web scraping solution that includes scheduling automatic launches through GitHub Actions, using Puppeteer with Node.js to collect data, and storing the data on GitHub Pages for public access.

Basic Info
Statistics
  • Stars: 0
  • Watchers: 2
  • Forks: 5
  • Open Issues: 0
  • Releases: 2
Topics
puli-portfolio
Created over 2 years ago · Last pushed over 1 year ago
Metadata Files
Readme License Citation

README.md

DOI

crawler-Course-Example: 自動排程網路爬蟲教材

A fully automated solution for web scraping. It includes automatic scheduling and triggering through GitHub Actions, data collection using Node.js's Puppeteer, and data preservation on GitHub Pages for use by others.

一套網路爬蟲的全自動解決方案。包含了透過GitHub Action自動排程啟動、使用Node.js的Puppet蒐集資料、以GitHub Pages保存資料供其他人使用。

Techniques

  • Node.js
  • GitHub Action: 自動排程執行的DevOps方案。
  • Puppeteer: Node.js的瀏覽器模擬工具。

Slide

Citation

Chen, Y.-T. (2024). Crawler-Course-Example (20240518.210053) [Computer software]. Zenodo. https://doi.org/10.5281/ZENODO.11214115

Memo

最後記得用以下欄位儲存：

  • id: 一定要有id。
  • dc.title
  • dc.creator
  • dc.subject
  • dc.description
  • dc.publisher
  • dc.contributor
  • dc.date: 建議轉換成ISO格式。
  • dc.type
  • dc.format
  • dc.identifier
  • dc.source
  • dc.language
  • dc.relation
  • dc.coverage
  • dc.rights

API

https://pulipulichen.github.io/crawler-Course-Example/data.csv

Owner

  • Name: Pulipuli Chen
  • Login: pulipulichen
  • Kind: user
  • Location: Taiwan

Citation (CITATION.cff) 

cff-version: 1.2.0
message: "If you use this software, please cite it using these metadata."
authors:
  - family-names: Chen
    given-names: Yung-Ting
    orcid: https://orcid.org/0000-0003-3751-610X
    email: https://blog.pulipuli.info
title: "crawler-Course-Example"
license: "MIT"
repository-code: "https://github.com/pulipulichen/crawler-Course-Example"
abstract: "An example to crawl a webpage."
identifiers:
  - type: doi
    value: 10.5281/zenodo.11214115

GitHub Events

Total
Last Year

Committers

Last synced: 12 months ago

All Time
  • Total Commits: 40
  • Total Committers: 1
  • Avg Commits per committer: 40.0
  • Development Distribution Score (DDS): 0.0
Past Year
  • Commits: 2
  • Committers: 1
  • Avg Commits per committer: 2.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Pulipuli Chen p****n@g****m 40

Issues and Pull Requests

Last synced: 12 months ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Dependencies

.github/workflows/crawler.yml actions
  • JamesIves/github-pages-deploy-action v4 composite
  • actions/cache v3 composite
  • actions/checkout v3 composite
  • satackey/action-docker-layer-caching v0.0.11 composite
Dockerfile docker
  • pudding/github-action-app puppeteer-python-14-action-rss-20230826-1641 build
docker-compose.yml docker
  • pudding/github-action-app puppeteer-python-14-crawler-Course-Example-20231106-1201
package-lock.json npm
  • iconv-lite 0.6.3
  • safer-buffer 2.1.2
package.json npm