See what the crawler found
Inspect discovered pages, source links, response statuses, and redirect outcomes.
Website Audit Tool
Home / Free Website Crawler
Core audit pages
Free website crawler with URL-level audit details.
A crawler-focused page for people who want to inspect discovered URLs, response statuses, redirects, titles, descriptions, and crawl paths.
✓Free during beta✓URL-level evidence✓Private history after signup✓Scheduled monitoring
What this check finds
This page focuses on website crawler in crawl context. The crawler records the fetched URL, response status, discovery source when available, metadata, redirect outcome, and history signals that explain why a finding appears.
Use the check to review discovered URLs, response statuses, redirects, titles, descriptions, and crawl paths. The goal is not just to report a problem, but to show the URL-level evidence needed to decide whether the issue is intentional, accidental, fixed, or still unresolved.
Free Website CrawlerCompleted
Visited704
Indexable619
Errors20
Warnings8
FindingSeverityEvidence
Discovered URLsInfoCrawl path and source page
HTTP terminal statusesWarningStatus code and final URL
Broken linksErrorSource page evidence
Why this check matters
Understand crawl coverage
Review visited, discovered, indexable, and blocked counts from the completed run.
Repeat with consistent settings
Use comparable crawl settings when validating fixes or monitoring changes.
How to fix and verify it
- Run a focused crawl. Start with the affected website or section and keep settings consistent for follow-up comparisons.
- Inspect the evidence. Review affected URLs, source pages, HTTP statuses, metadata values, and crawl lineage before making changes.
- Apply the smallest safe fix. Update the page, template, route, redirect rule, canonical tag, or internal link that caused the finding.
- Rerun the audit. Confirm that the finding is resolved and did not create a new issue elsewhere.
When to monitor it
Small technical issues become expensive when they remain invisible. For website crawler, a repeatable crawl helps distinguish a one-time anomaly from a persistent problem caused by templates, redirects, routes, metadata, or content changes.
Start with the concrete evidence: affected URL, source page, HTTP status, metadata value, redirect target, or comparison row. After the fix is made, run another crawl with comparable settings to confirm the finding is resolved.
Frequently asked questions
What does this page check?
It checks discovered URLs, response statuses, redirects, titles, descriptions, and crawl paths and presents the result as summary counts plus URL-level evidence. That makes the audit useful for deciding what to fix and for verifying the result in a later crawl.
Will the audit start immediately after I enter a URL?
No. The URL is transferred to the crawler seed field so you can review scope, limits, redirects, and sitemap policy before starting.
Can this be monitored over time?
Yes. Registered users can create scheduled audits and review whether findings are new, resolved, or still unresolved across completed crawls.
Is this only for SEO?
No. The same findings are useful for site maintenance, release validation, migration checks, and general quality assurance.
Related audit pages
Run the free website crawler.
Enter a seed URL and review discovered pages, crawl paths, redirects, status codes, and metadata in the live crawler.