Free SEO and Dev Tools

⚙️

Configure Your robots.txt

Fill in your domain and rules, then generate

Your Domain Required

🌐

Used for the Sitemap URL and file signature

Default Crawl Rules

Preset Template

Bot Access Rules

🔍 Allow Googlebot

Allow Google to crawl your site

🔵 Allow Bingbot

Allow Bing to crawl your site

🤖 Block AI Scrapers

Block GPTBot, ClaudeBot, CCBot

🔒 Block Admin Pages

Hide /admin, /wp-admin, /login

🗂️ Block Private Paths

Hide /private, /staging, /api

⏱️ Crawl-delay (10s)

Throttle bot requests

🖼️ Block Image Crawling

Prevent /images/, /uploads/ indexing

🔎 Block Internal Search

Block /?s=, /search?q= pages

Custom Disallow / Allow Rules

Sitemaps

robots.txt

🤖

Your robots.txt will appear here

Fill in your domain and rules on the left,
then click Generate robots.txt

What is a robots.txt file?

A robots.txt file is a plain-text file placed at the root of your website that tells search engine crawlers which pages or sections they can and cannot access.

Every website should have a robots.txt file. Without one, search engines will crawl everything — including admin panels, login pages, staging environments, and duplicate content — which can harm your SEO and waste your crawl budget.

The file lives at yourdomain.com/robots.txt and is the first file crawlers check before indexing your site.

Key directives:
• User-agent — specifies which bot the rule applies to (* = all bots)
• Disallow — blocks a path from being crawled
• Allow — explicitly permits a path (overrides Disallow)
• Sitemap — tells bots where your sitemap is
• Crawl-delay — throttles crawl speed to reduce server load

🎯

Protect Your Crawl Budget

Block low-value pages (admin, search results, staging) so Google spends its budget on your real content.

⚠️

Disallow ≠ Noindex

Blocking a page in robots.txt doesn’t remove it from Google’s index. Use a noindex meta tag for that.

🔒

Not a Security Tool

robots.txt is a convention, not a security barrier. Malicious bots will ignore it. Use server authentication for truly private pages.

📍

Upload to Root Directory

The file must be at yourdomain.com/robots.txt — not in a subdirectory. Most hosts let you upload via FTP, cPanel, or File Manager.

Frequently Asked Questions

Does every website need a robots.txt file? ▼

Not strictly required, but highly recommended. Without one, all bots can crawl everything, including admin pages, login forms, and duplicate URLs. A well-configured robots.txt protects your crawl budget and keeps sensitive paths out of search results.

Will disallowing a page remove it from Google’s index? ▼

No. Disallowing a URL in robots.txt prevents crawling, but Google may still index the URL if it finds links pointing to it — it just won’t know the page content. To fully remove a page from Google’s index, use the noindex meta tag or HTTP header, or use Google Search Console’s URL removal tool.

What’s the difference between Disallow and noindex? ▼

Disallow (robots.txt): prevents bots from crawling the page — they won’t read its content. noindex (meta tag): bots can crawl the page but won’t include it in search results. For pages you want completely out of Google, use both. For pages that are just low-value but not sensitive, noindex alone is usually sufficient.

Should I block AI scrapers like GPTBot in robots.txt? ▼

That’s your choice. Blocking GPTBot, ClaudeBot, and CCBot prevents your content from being used to train AI models — reputable AI companies honor robots.txt. However, it won’t stop all scraping tools. If you want to prevent AI training data collection from your content, adding these blocks is a reasonable first step.

How do I test if my robots.txt is working? ▼

Use Google Search Console → Settings → robots.txt tester, or visit google.com/webmasters/tools/robots-testing-tool. You can also check if your file is live by visiting yourdomain.com/robots.txt in a browser. For advanced testing, fetch the URL with a tool like curl and verify the response headers and content.

🤖 Generated by TrafficTool.in — Free SEO & Web Utilities Toolkit

No signup · No limits · No tracking · Built for SEOs and developers everywhere

Robots.txt Generator