Data

Tools

Indexes

Applications

Experiments

Open Source Science

foot

Foot is a library that fetches a list of URLs and silly walks through each site to gather information.

https://github.com/cak/foot

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (5.8%) to scientific vocabulary

Keywords

bugbounty crawler scraping
Last synced: 6 months ago · JSON representation

Repository

Foot is a library that fetches a list of URLs and silly walks through each site to gather information.

Basic Info
Statistics
  • Stars: 0
  • Watchers: 2
  • Forks: 0
  • Open Issues: 1
  • Releases: 0
Topics
bugbounty crawler scraping
Created about 7 years ago · Last pushed over 1 year ago
Metadata Files
Readme Changelog Contributing License Authors

README.md

foot

Web Crawling Project

Foot is a library that fetches a list of URLs and silly walks through each site to gather information.

Usage

Install

pip install foot

foot (cli)

  • -u URL(s) (encapsulated in quotes, separated by commas)
  • -f filename (list of urls on new lines)
Options (not required)
  • -c : Specify chunk size (default=10)
  • --recursive : Enable one level of recursion
Examples:

Array of URLS:
foot -u 'http://example.com, http://example1.com'

Array of URLS with Options:
foot 'http://example.com, http://example1.com' -c 5 --recursive

File:
foot -f 'example.txt

File with Options:
foot -f 'example.txt' -c 5 --recursive

Module

Import:
import foot

Functions

The get function takes an array of URLS and options.

javascript foot.get(["http://www.example`.com", "http://www.example2.com"], options); => Data output in ./foot/url.json and foot-date.txt

The file function takes a filename and options.

python foot.file("./test.txt", recursive=True) => Data output in ./foot/url.json and foot-date.txt

Owner

  • Name: caleb
  • Login: cak
  • Kind: user
  • Location: Maryland

code, coffee, and cybersecurity

GitHub Events

Total
  • Pull request event: 1
  • Create event: 1
Last Year
  • Pull request event: 1
  • Create event: 1

Committers

Last synced: 6 months ago

All Time
  • Total Commits: 1
  • Total Committers: 1
  • Avg Commits per committer: 1.0
  • Development Distribution Score (DDS): 0.0
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Caleb Kinney c****y@g****m 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 0
  • Total pull requests: 1
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 1
Past Year
  • Issues: 0
  • Pull requests: 1
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 1
View more stats
Top Authors
Issue Authors
Pull Request Authors
  • dependabot[bot] (2)
Top Labels
Issue Labels
Pull Request Labels
dependencies (2)

Dependencies

setup.py pypi
  • aiohttp ==3.4.4
  • beautifulsoup4 ==4.6.3
  • eg *
  • six >=1.7