Key takeaways
- Verify /robots.txt returns text rather than the SPA HTML shell.
- Remove staging rules such as a site-wide Disallow: / before production.
- Reference the absolute production sitemap URL.
- Do not rely on robots.txt to remove sensitive pages from search.
- Re-test the file after deployment configuration changes.
Why staging rules cause production incidents
Teams often block crawling while a site is under development and then carry the same robots file into production. A single site-wide disallow can prevent normal crawling. SearchFix treats this as a critical issue because it can affect the entire site.
robots.txt is not access control
Robots directives are instructions for compliant crawlers, not a security boundary. Private or sensitive pages should be protected with proper authentication and authorization. Likewise, blocking a URL in robots.txt does not guarantee that the URL disappears from an index.
Keep the file simple
Most marketing sites need only straightforward public crawling rules plus a sitemap reference. Complex patterns are harder to test and easier to break. Add restrictions only when there is a clear crawl-management reason.
Frequently asked questions
Where does robots.txt go on a Lovable project?
It needs to be served from the site root at /robots.txt. In many React-style projects that means a public asset, but the exact source path depends on the project's framework and deployment setup.
Should I block admin pages in robots.txt?
Security should come from authentication. You may also use crawl directives where appropriate, but do not expose sensitive pages and assume robots.txt protects them.
Should robots.txt contain my sitemap?
It is useful to include an absolute Sitemap directive pointing to the production sitemap URL.