Confirm intentional exclusions
Separate expected noindex pages from pages that should be indexable.
Find pages that contain noindex directives and verify whether those exclusions are intentional.
This page focuses on noindex directives in crawl context. The crawler records the fetched URL, response status, discovery source when available, metadata, redirect outcome, and history signals that explain why a finding appears.
Use the check to review pages blocked from indexing by robots metadata. The goal is not just to report a problem, but to show the URL-level evidence needed to decide whether the issue is intentional, accidental, fixed, or still unresolved.
Separate expected noindex pages from pages that should be indexable.
See whether the directive comes from robots meta or related crawl signals.
Detect new noindex pages after deployments or CMS edits.
Small technical issues become expensive when they remain invisible. For noindex directives, a repeatable crawl helps distinguish a one-time anomaly from a persistent problem caused by templates, redirects, routes, metadata, or content changes.
Start with the concrete evidence: affected URL, source page, HTTP status, metadata value, redirect target, or comparison row. After the fix is made, run another crawl with comparable settings to confirm the finding is resolved.
It checks pages blocked from indexing by robots metadata and presents the result as summary counts plus URL-level evidence. That makes the audit useful for deciding what to fix and for verifying the result in a later crawl.
No. The URL is transferred to the crawler seed field so you can review scope, limits, redirects, and sitemap policy before starting.
Yes. Registered users can create scheduled audits and review whether findings are new, resolved, or still unresolved across completed crawls.
No. The same findings are useful for site maintenance, release validation, migration checks, and general quality assurance.
Enter a website URL and inspect noindex directives with URL-level evidence in the live crawler.