Exposure · Check
robots.txt missing — publishing crawler guidance for your site
A missing robots.txt isn't a security problem on its own, but it's a hygiene signal — automated crawlers and search engines have no canonical place to find your sitemap or crawl preferences. Even an allow-everything `User-agent: *\nAllow: /` plus a `Sitemap:` line is better than nothing.
Real-world risk
Crawler and archive behavior defaults to permissive rules; sitemap hints and crawl delays are not communicated.
Fix steps (in order)
- Publish /robots.txt with intentional rules for User-agent: * and Sitemap: lines if you want discovery hints.
Verify the fix in 30 seconds
Run a Scorifya scan on the affected host after deploy. The same finding id (robots_txt_not_found) clears once the externally-observable signal is in place.