Free Robots.txt Generator | Create & Test Robots.txt Online

Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

What Is a Robots.txt File and Why It Controls Your SEO

A robots.txt file is a plain-text file stored at your domain's root (yourdomain.com/robots.txt) that gives crawl instructions to search engine bots before they access your site. It doesn't hide pages from users it tells compliant crawlers like Googlebot, Bingbot, and others which URLs they're allowed to request.

This matters for three reasons that most generic explanations skip:

• Crawl budget is finite. Google allocates a limited number of requests per site per day, based on your site's authority and server response speed. Every crawl spent on a filtered URL, an internal search page, or a duplicate parameter page is a crawl not spent on your new blog post or product page.
• Robots.txt does not prevent indexing. This is the single most common misunderstanding. If a disallowed URL has external backlinks, Google can still index the URL (showing it with no snippet, as "indexed, though blocked by robots.txt" in Search Console) because robots.txt blocks crawling, not indexing. To fully deindex a page, use a noindex meta tag or HTTP header, and make sure the page is crawlable so Google can see that tag.
• One misplaced slash can deindex your entire site. Disallow: / blocks everything. This has happened to large, well-known sites during migrations it's not a hypothetical beginner mistake.

How to Write Robots.txt Directives Correctly

Basic Syntax

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/help-center/
Sitemap: https://yourdomain.com/sitemap.xml

• User-agent which crawler the rule applies to (* = all crawlers, or name a specific one like Googlebot)
• Disallow path the crawler should not request
• Allow an exception inside a disallowed folder
• Sitemap points crawlers to your XML sitemap (best practice: always include this)

Common Real-World Rules by Site Type

• Ecommerce: block /cart/, /checkout/, /*?sort=, /*?filter= to stop crawl budget waste on faceted navigation, while keeping category and product pages open
• WordPress: block /wp-admin/ but allow /wp-admin/admin-ajax.php (Google needs this to render pages correctly)
• SaaS platforms: block staging subdomains, internal dashboards, and API routes; keep marketing and docs pages open
• Blogs/publishers: rarely need heavy blocking usually just /tag/ or internal search results pages if they create thin, duplicate content

Robots.txt Mistakes That Actively Hurt Rankings

1. Blocking CSS/JS files. Google renders pages like a browser. If robots.txt blocks the assets needed to render the page, Google may misjudge mobile-friendliness or page layout.
2. Using robots.txt to hide duplicate content instead of canonical tags. Blocking a duplicate page stops Google from crawling it but if Google already indexed it, it stays indexed with no way to consolidate signals to the preferred version.
3. Case sensitivity errors. /Products/ and /products/ are different paths to crawlers. A rule written with the wrong case simply won't match.
4. Forgetting the trailing slash on directory rules, which can accidentally block unintended URLs that merely start with the same string.
5. Deploying a staging robots.txt to production  a Disallow: / copied over from a staging environment is one of the most common causes of sudden, unexplained traffic drops after a site migration.

How This Generator Helps You Avoid Those Mistakes

Instead of hand-writing syntax, this tool lets you:

• Set default crawl access (allow/refuse) for all bots in one toggle
• Apply different rules to 15+ named crawlers (Googlebot, Google Image, Bingbot/MSN, Yahoo, Baidu, Naver, and others) when you need bot-specific behavior for example, allowing Googlebot Image but blocking a scraping bot from image directories
• Set a crawl-delay for servers with limited resources (rarely needed for modern hosting, but useful for shared hosting under heavy bot load)
• Add your sitemap URL directly, so it's declared in the one place crawlers check first
• Generate clean, validated syntax that avoids the case-sensitivity and formatting errors listed above

How to Test Your Robots.txt Before Publishing

After generating your file:

1. Upload it to your site's root directory it must be reachable at https://yourdomain.com/robots.txt, not in a subfolder.
2. Test it in Google Search Console's URL Inspection tool to confirm a specific URL is or isn't blocked.
3. Re-check it after any site migration, replatform, or staging-to-production deployment this is the single most common time robots.txt breaks silently.

Frequently Asked Questions

Does blocking a page in robots.txt remove it from Google's index?

No. Robots.txt blocks crawling, not indexing. A blocked page with external links can still appear in search results without a description. Use a noindex tag on a crawlable page to fully remove it from the index.

Do I need a robots.txt file if I have nothing to block?

It's still good practice to have one even an empty-rules file that only declares your sitemap because its absence can cause some crawlers to log repeated 404 requests, and its presence confirms to Google that no crawl instructions were missed.

Can robots.txt block bad bots or scrapers?

Only bots that voluntarily respect the standard will follow it. Malicious scrapers typically ignore robots.txt entirely for those, you need server level blocking (firewall rules, rate limiting) instead.

How is this different from a meta robots tag?

Robots.txt is a site-wide file controlling crawl access. A meta robots tag (<meta name="robots" content="noindex">) sits on an individual page and controls indexing of that specific page once it's already been crawled. They solve different problems and are often needed together.