How to Create a Robots.txt File Online — Crawl Rules Guide

A robots.txt file tells crawlers which parts of a website they are allowed or not allowed to crawl. It is placed at the root of a site, such as /robots.txt. Search engines check this file before crawling many URLs.

Utilao's Robots.txt Generator helps you create simple robots.txt rules without writing the syntax manually.

What robots.txt can and cannot do

Robots.txt controls crawling. It does not guarantee that a URL will be removed from search results, and it does not protect private content. If something must stay private, use authentication or remove it from public access.

Robots.txt is useful for blocking low-value paths, admin areas, search result pages, duplicate URLs, generated files or heavy resources that should not be crawled.

How to create robots.txt online

1. Open the Robots.txt Generator.
2. Choose a user-agent.
3. Add Allow or Disallow rules.
4. Add your sitemap URL if available.
5. Copy the generated text.
6. Upload it as /robots.txt at the root of your site.

After creating crawl rules, check that important URLs are still included in your XML sitemap.

Common robots.txt mistakes

Do not block pages you want indexed. Do not use robots.txt as a security feature. Be careful with broad rules like Disallow: /, because that can block your entire website from crawling.

If you add a sitemap line, use the full absolute sitemap URL.

FAQ

Can robots.txt block my whole site?

Yes. User-agent: * plus Disallow: / tells compliant crawlers not to crawl the entire site.

Is robots.txt a privacy tool?

No. Public robots.txt files can be viewed by anyone.

Should robots.txt include a sitemap?

It is a good practice to include a Sitemap line pointing to your XML sitemap.