Exposure · Check
robots.txt blocks all crawlers — confirm this is intentional
A `Disallow: /` for `User-agent: *` blocks every crawler, including Google. This is correct for staging/internal sites but catastrophic for a public production site — it removes you from search results entirely. Verify the rule matches the host's role.
Real-world risk
Indexing is discouraged for well-behaved crawlers; malicious agents may still fetch public URLs.
Fix steps (in order)
- If this is production, review whether a blanket disallow is intended; tighten authentication for anything that must stay private.
- If staging, keep DNS separate from production to avoid accidental discovery.
Verify the fix in 30 seconds
Run a Scorifya scan on the affected host after deploy. The same finding id (robots_txt_disallow_all) clears once the externally-observable signal is in place.