Search indexing | Laravel Documentation

Catches settings that hide your site from search engines, like a robots.txt blocking everything or a noindex tag left over from staging. It also flags missing pages that answer 200 instead of 404.

How it works

Once a day, we look for mistakes that hide your site from search engines, usually a staging configuration shipped to production:

robots.txt blocking every crawler (Disallow: /), or answering with a server error, which Google reads as "disallow everything";
a noindex directive on the page, in a <meta name="robots"> tag or an X-Robots-Tag header.

Either one makes your pages disappear from Google within days. We also request a page that does not exist: it should answer 404. A missing page that answers 200 (a "soft 404") gets error pages indexed as duplicate content.

Statuses

Status	Meaning
OK	Search engines may index the site, and missing pages return 404.
Warning	Missing pages return a success status (soft 404), or we could not connect.
Failed	`robots.txt` blocks everything or returns a server error, or the page has `noindex`.

By default you are notified of warnings and failures.

Fixing common problems

robots.txt returns 5xx: serve a valid file, even an empty one, with a 200 status.
robots.txt blocks everything: replace Disallow: / with the paths you really want hidden, or remove the rule.
noindex: remove it from the page template or the server configuration. In WordPress, untick Discourage search engines from indexing this site under Settings › Reading.
Soft 404: make your application or catch-all route return a real 404 status for unknown pages.
After a fix, request indexing again in Google Search Console to speed up recovery.