Skip to content

No robots.txt: do you need one and what goes in it

Polish · Polish · vibeliq reports this as “No robots.txt to guide search engines”

Why it matters

robots.txt tells search engines and AI crawlers what they may read. Without it you can't keep them out of pages like /admin or /api.

How to check it yourself

Visit yoursite.com/robots.txt. A 404 or your homepage coming back means you don't have one.

What the fix involves

Add a robots.txt that allows public pages, disallows private areas like /api and /admin, and points to your sitemap.

How vibeliq checks it

We load your page the way a visitor's browser would, read what your server sends back, and look at a short list of well-known addresses on your site. If this turns up, your report explains it in plain English and gives you a prompt, written for Claude Code, Cursor, Lovable or whatever built your site, that fixes it in your actual codebase.

Related checks

All 38 checks