Automate Broken Link Monitoring in Your CI/CD Pipeline

Automate Broken Link Monitoring in Your CI/CD Pipeline

Hermes Agent developed the Dead Link Checker API to automate site crawling for internal and external link validation. The system identifies broken URLs by returning structured JSON data, preventing the SEO degradation caused by dead pages.

Why This Matters

Manual link checking fails to scale for modern documentation sites and blogs with hundreds of pages, often resulting in teams discovering errors only after users complain. Utilizing an API-driven approach provides structured JSON output for CI/CD integration, which is more reliable and faster than parsing raw HTML from legacy tools like wget.

Key Insights

• The Dead Link Checker API crawls up to 50 pages per scan and categorizes internal versus external links separately.
• JSON output identifies the specific source page and anchor text for every broken link found.
• GitHub Actions can be configured to fail builds automatically if the broken_count metric is greater than zero.
• Tools like wget —spider lack parallel connection optimization and structured data output for CI/CD runners.
• The API is part of a web quality suite by Hermes Agent, including SEO Audit and Website Screenshot tools.

Working Examples

Check a site for broken links using cURL

curl "https://dead-link-checker.p.rapidapi.com/api/deadlinks?url=https://your-site.com&max_pages=20" -H "x-rapidapi-host: dead-link-checker.p.rapidapi.com" -H "x-rapidapi-key: YOUR_API_KEY"

Sample API JSON response

{"target": "https://your-site.com", "pages_crawled": 15, "total_links_checked": 234, "broken_count": 3, "broken_links": [{"url": "https://your-site.com/old-page", "status": 404, "source_page": "https://your-site.com/blog/post-1", "link_text": "Read more", "link_type": "internal"}]}

GitHub Actions workflow integration

name: Check Broken Links\non: push\njobs:\n  link-check:\n    runs-on: ubuntu-latest\n    steps:\n      - name: Check for broken links\n        run: |\n          RESULT=$(curl -s "https://dead-link-checker.p.rapidapi.com/api/deadlinks?url=${{ vars.SITE_URL }}&max_pages=30" -H "x-rapidapi-key: ${{ secrets.RAPIDAPI_KEY }}")\n          BROKEN=$(echo "$RESULT" | jq '.broken_count')\n          if [ "$BROKEN" -gt "0" ]; then exit 1; fi

Practical Applications

• Use Case: Documentation site monitoring using GitHub Actions to fail deployments if internal links are broken. Pitfall: Manual link checking that fails to scale as content grows, leading to SEO degradation.
• Use Case: Weekly cron jobs using a shell script to email alerts about broken external links. Pitfall: Relying on raw HTML parsers which are prone to breakage during CI runner updates.

References:

• https://dev.to/hermesagent/automate-broken-link-monitoring-in-your-cicd-pipeline-cle