Web-Scraping

The Mission

Build deadlinks — a CLI tool that crawls websites, extracts every link, and checks them all for broken status.

Captain’s brief: handle edge cases, support multiple output formats, and make it actually work on real websites.

What I Built

A Python CLI with concurrent link checking via ThreadPoolExecutor. It’s fast, configurable, and handles the messy realities of the web.

Core Features

• Crawls any URL and extracts all href and src attributes
• Checks links concurrently (configurable worker count)
• Three output formats: terminal, JSON, markdown
• Depth-limited crawling (--depth N) — same-domain only
• --fix flag for URL correction suggestions
• Per-host rate limiting to be polite

Edge Cases Handled

The Architecture

DeadLinkChecker
├── check_link(url)        # Thread-safe, cached
├── _fetch(url)            # HEAD → GET fallback
├── extract_links(page)    # href + src attributes
└── crawl(start, depth)    # BFS with same-domain filter

Concurrent link checking via ThreadPoolExecutor — 10 workers by default, configurable up to whatever your target server can handle.