Broken Links Inspector

This project is heavily inspired by stevenvachon/broken-link-checker.

If you want to use this tool and need any help (instructions, bug fixes, features) open an issue!

Features: - inspects a web-page and all its URLs, reports broken ones - can go recursively, inspecting all pages within a domain - makes requests in parallel, shows indication of "work in progress" - does not check URL twice - reports OK, TIMEOUT, ERROR CODE or generic error - support configurable timeout - supports GET and HEAD methods (double checks with GET if HEAD fails) - supports a list of excluded URLs (glob matching) and/or excluded prefixes (e.g. mailto:) - can define OK codes, such as 999 for linkedin - supports different reporting, such as colored console or JUnit file - JUnit report is best used with CI (tested with GitLab) - need a feature, go to issues

How to install and run

``` npm i -g broken-links-inspector

bli inspect https://dbogatov.org -r -t 2000 -s linkedin --reporters console

or

bli inspect file://links.txt

with a URL per line in a file links.txt

```

How to use

``` $ bli inspect -h

Usage: index inspect [options] file://

Check links in the given URL or a text file

Options: -r, --recursive recursively check all links in all URLs within supplied host (ignored for file://) (default: false) -t, --timeout timeout in ms after which the link will be considered broken (default: 2000) -g, --get use GET request instead of HEAD (default: false) -s, --skip URLs to skip defined by globs, like 'linkedin' (default: []) --reporters Reporters to use in processing the results (junit, console) (default: ["console"]) --retries The number of times to retry TIMEOUT URLs (default: 3) --user-agent The User-Agent header (default: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10157) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1 Safari/605.1.15") --ignore-prefixes prefix(es) to ignore (without ':'), like mailto: and tel: (default: ["javascript","data","mailto","sms","tel","geo"]) --accept-codes HTTP response code(s) (beyond 200-299) to accept, like 999 for linkedin (default: [999]) --ignore-skipped Do not report skipped URLs (default: false) --single-threaded Do not enable parallelization (default: false) -v, --verbose log progress of checking URLs (default: false) -h, --help display help for command ```

Return code is 1 if at least one broken link detected, 0 otherwise.

-r, --recursive will instruct inspector to keep checking all URLs in the original domain. Very useful for checking an entire website, such as personal blog. For example, bli inspect https://yoursite.com -r will check yoursite.com and if it finds something like yoursite.com/contact it will check that as well and will keep going. It will check all URLs on all pages, but will not parse "external" pages.

-t, --timeout <number> given in milliseconds sets a timeout for a request. If this timeout is exceeded, the check fails with TIMEOUT.

-g, --get instructs to use GET request instead fo the default HEAD request. If HEAD request fails, the URL will be retried with GET.

-s, --skip <coma-separated-globs> is a list of globs or parts of URL to skip. As an example, -s *linkedin* -s hello will instruct to skip all URLs which contain either linkedin or hello in them.

--reporters <coma-separated-strings> is a list of reporters to process the result. Currently there are two: console and junit. console will print appealing colored report to the console. junit will produce junit-report.xml file in the current directory. JUnit file treats pages as test suites and URLs in a page as test cases.

--retries will instruct the number of times to try a URL before declaring it failed.

--user-agent <string> will use specified User-Agent header (some websites reply with 401 Unauthorized for "bots")

--ignore-prefixes <coma-separated-strings> is a list of prefixes/ schemas to skip, such as mailto:. Provided list should not include colons.

--accept-codes <coma-separated-numbers> is a list of HTTP code to consider successful, like 999 for linkedin.

--ignore-skipped excludes skipped URLs from reports.

--single-threaded mandates a sequential execution (should be used in for debugging).

-v, --verbose currently unused.

How to build

``` npm install # to install dependencies

npm run build # to compile TS (result in ./dist/index.js)

npm run coverage # to run tests and coverage ```