Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/arvid-berndtsson/robots-txt-analyzer

Modern robots.txt analyzer with instant analysis, security recommendations, and export capabilities. Built with Qwik and deployed on Cloudflare Pages.

https://github.com/arvid-berndtsson/robots-txt-analyzer

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (12.3%) to scientific vocabulary

Keywords

cloudflare-pages qwik robots-txt robots-txt-parser seo tailwindcss typescript web-crawler web-tools

Keywords from Contributors

archival interactive projection generic sequences observability autograding hacking shellcodes modular
Last synced: 5 months ago · JSON representation

Repository

Modern robots.txt analyzer with instant analysis, security recommendations, and export capabilities. Built with Qwik and deployed on Cloudflare Pages.

Basic Info
  • Host: GitHub
  • Owner: arvid-berndtsson
  • License: mit
  • Language: TypeScript
  • Default Branch: main
  • Homepage: https://robots-txt.arvid.tech
  • Size: 952 KB
Statistics
  • Stars: 1
  • Watchers: 1
  • Forks: 0
  • Open Issues: 8
  • Releases: 1
Topics
cloudflare-pages qwik robots-txt robots-txt-parser seo tailwindcss typescript web-crawler web-tools
Created about 1 year ago · Last pushed 6 months ago
Metadata Files
Readme License

README.md

Robots.txt Analyzer 🤖

A modern web tool for analyzing robots.txt files, built with Qwik and deployed on Cloudflare Pages. Try it out at robots-txt.arvid.tech!

Features

  • 🔍 Instant robots.txt analysis
  • 📊 Comprehensive scoring system
  • 🗺️ Sitemap validation
  • 🚫 Security recommendations
  • 📱 Mobile-friendly interface
  • 💾 Export results as JSON or CSV
  • ⚡️ Built with Qwik for optimal performance

Development

Prerequisites

  • Node.js 16+
  • npm or yarn
  • A .env.local file with: shell ORIGIN="http://localhost:5173" API_KEY="your-api-key" HISTORY_KV="NAME_OF_CLOUDFLARE_KV_NAMESPACE"
  • A wrangler.toml file with: shell [[d1_databases]] binding = "DB" database_name = "robots-txt-analyzer" database_id = "YOUR-DATABASE-ID"

Getting Started

  1. Clone the repository
  2. Install dependencies: shell npm install
  3. Start the development server: shell npm run dev
  4. Visit http://localhost:5173

Building for Production

shell npm run build

Preview Production Build

shell npm run preview

Deployment

This project is deployed on Cloudflare Pages. The build configuration is:

  • Build command: npm run build
  • Build output directory: dist
  • Environment variables:
    • ORIGIN: Your production URL
    • API_KEY: Your API key for the analyzer
    • DB: Your Cloudflare D1 database ID
  • wrangler.toml: Your Cloudflare D1 database ID
    • wrangler.toml.example: Example wrangler.toml file

Notes: - Set the DB variable in the .env.local file to your Cloudflare D1 database ID. - For initially setup, push the DB schema to cloudflare, run npx wrangler d1 execute robots-txt-analyzer --file=./db/schema.sql --remote.

Cleanup

To cleanup old cache entries, you can run the cleanup endpoint. This will delete all cache entries older than 24 hours.

shell curl -X GET https://robots-txt.arvid.tech/api/v1/cleanup

Recommendation is to call this using a cron job.

Project Structure

├── src/ │ ├── components/ # Reusable UI components │ ├── routes/ # Page routes and API endpoints │ └── utils/ # Utility functions and parsers ├── public/ # Static assets └── adapters/ # Cloudflare Pages adapter config

Built With

Author

Built by Arvid Berndtsson

License

This project is open source and available under the MIT License.

Owner

  • Name: Arvid Berndtsson
  • Login: arvid-berndtsson
  • Kind: user
  • Location: Sweden
  • Company: @LimeTip

Based in Malmö, Sweden, I specialize in software development and digital sustainability. Founder of @LimeTip and Co-Founder of @merely-emissions

GitHub Events

Total
  • Create event: 93
  • Issues event: 2
  • Release event: 1
  • Delete event: 87
  • Issue comment event: 140
  • Push event: 56
  • Pull request review event: 2
  • Pull request event: 173
Last Year
  • Create event: 93
  • Issues event: 2
  • Release event: 1
  • Delete event: 87
  • Issue comment event: 140
  • Push event: 56
  • Pull request review event: 2
  • Pull request event: 173

Committers

Last synced: 7 months ago

All Time
  • Total Commits: 40
  • Total Committers: 2
  • Avg Commits per committer: 20.0
  • Development Distribution Score (DDS): 0.15
Past Year
  • Commits: 40
  • Committers: 2
  • Avg Commits per committer: 20.0
  • Development Distribution Score (DDS): 0.15
Top Committers
Name Email Commits
Arvid Berndtsson 1****n 34
dependabot[bot] 4****] 6

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 3
  • Total pull requests: 158
  • Average time to close issues: N/A
  • Average time to close pull requests: 5 days
  • Total issue authors: 2
  • Total pull request authors: 2
  • Average comments per issue: 0.0
  • Average comments per pull request: 1.27
  • Merged pull requests: 56
  • Bot issues: 1
  • Bot pull requests: 107
Past Year
  • Issues: 3
  • Pull requests: 158
  • Average time to close issues: N/A
  • Average time to close pull requests: 5 days
  • Issue authors: 2
  • Pull request authors: 2
  • Average comments per issue: 0.0
  • Average comments per pull request: 1.27
  • Merged pull requests: 56
  • Bot issues: 1
  • Bot pull requests: 107
View more stats
Top Authors
Issue Authors
  • arvid-berndtsson (2)
Pull Request Authors
  • dependabot[bot] (101)
  • arvid-berndtsson (50)
Top Labels
Issue Labels
Pull Request Labels
dependencies (101) javascript (41)

Dependencies

.github/workflows/cc-commit-messages.yml actions
  • actions/checkout v4 composite
.github/workflows/cc-pr-title.yml actions
  • actions/checkout v4 composite
package-lock.json npm
  • 598 dependencies
package.json npm
  • @builder.io/qwik ^1.12.0 development
  • @builder.io/qwik-city ^1.12.0 development
  • @types/eslint 8.56.10 development
  • @types/node 20.14.11 development
  • @typescript-eslint/eslint-plugin 7.16.1 development
  • @typescript-eslint/parser 7.16.1 development
  • autoprefixer ^10.4.19 development
  • eslint 8.57.0 development
  • eslint-plugin-qwik ^1.12.0 development
  • postcss ^8.4.39 development
  • prettier 3.3.3 development
  • prettier-plugin-tailwindcss ^0.5.4 development
  • tailwindcss ^3.4.6 development
  • typescript 5.4.5 development
  • undici * development
  • vite 5.3.5 development
  • vite-tsconfig-paths ^4.2.1 development
  • wrangler ^3.0.0 development