A robots.txt generator creates the robots.txt file that tells search engine crawlers (Google, Bing, Yandex) which parts of your website they are allowed to crawl and index, and which parts they should ignore.
Every website needs a robots.txt file. Without one, crawlers access everything — including admin panels, duplicate content, staging URLs, and internal search results pages that dilute your SEO. This guide explains the syntax, common rules, and how to generate the right robots.txt for your site.
Generate Your robots.txt File
Select your website type, add disallow rules, include your sitemap, and download a ready-to-use robots.txt file.
What Is robots.txt?
robots.txt is a plain text file placed at the root of your website (example.com/robots.txt) that gives instructions to web crawlers. It follows the Robots Exclusion Protocol — an internet standard since 1994.
Basic Syntax
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
Sitemap: https://example.com/sitemap.xml| Directive | Purpose | Example |
|---|---|---|
| User-agent | Which crawler this rule applies to | * (all), Googlebot, Bingbot |
| Disallow | Block this path from crawling | /admin/, /api/, /tmp/ |
| Allow | Override a disallow for a sub-path | /admin/public-page |
| Sitemap | Location of your XML sitemap | https://example.com/sitemap.xml |
| Crawl-delay | Seconds between requests (not Google) | 10 (for Bingbot, Yandex) |
robots.txt prevents crawling, not indexing. If other pages link to a disallowed URL, Google may still index it (showing "No information is available for this page"). Use noindex meta tag or X-Robots-Tag header to prevent indexing.
Common robots.txt Rules by Website Type
Different types of websites need different robots.txt configurations:
WordPress Blog
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /tag/
Disallow: /author/
Disallow: /?s=
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlE-commerce (Shopify/WooCommerce)
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search
Disallow: /collections/*?sort_by=
Disallow: /collections/*?filter=
Sitemap: https://example.com/sitemap.xmlSPA / React / Next.js App
User-agent: *
Disallow: /api/
Disallow: /_next/
Disallow: /dashboard/
Disallow: /settings/
Sitemap: https://example.com/sitemap.xmlWhat to Block and What to Allow
Getting this wrong either wastes your crawl budget or hides important pages. Here is a decision framework:
Block These
| Path | Why Block |
|---|---|
| /admin/, /dashboard/ | Internal admin panels — no SEO value |
| /api/ | API endpoints — not user-facing pages |
| /cart, /checkout | User-specific pages — should not be indexed |
| /search, /?s=, /?q= | Internal search results — creates infinite crawl paths |
| /tmp/, /staging/ | Temporary or test content |
| Duplicate content paths | URL parameters creating duplicate pages (?sort=, ?filter=) |
Never Block These
- CSS and JavaScript files — Google needs these to render your pages correctly. Blocking them hurts mobile-first indexing.
- Images you want indexed — Block only private images, not product photos or blog images.
- Your main content pages — Blog posts, product pages, landing pages should always be crawlable.
Google allocates a crawl budget to each site. Blocking low-value pages (search results, filtered pages, admin) lets Google spend more budget on your important pages.
Common robots.txt Mistakes
These mistakes can tank your SEO or expose private content:
| Mistake | Impact | Fix |
|---|---|---|
| Disallow: / (blocking everything) | Entire site disappears from Google | Remove or limit to specific paths |
| Blocking CSS/JS files | Google cannot render pages, hurts rankings | Allow all CSS/JS |
| No sitemap reference | Google may miss important pages | Add Sitemap: directive |
| Using robots.txt for security | Pages are still accessible via direct URL | Use authentication, not robots.txt |
| Trailing slash confusion | /admin blocks /admin/ and /admin-page | Use /admin/ with trailing slash |
| Forgetting staging subdomain | staging.example.com gets indexed | Add robots.txt to staging with Disallow: / |
Using Disallow: / on your production site. This single line removes your entire website from search engines. Always double-check before deploying.
Testing Your robots.txt
After generating your robots.txt, test it before deploying:
- Google Search Console — Use the robots.txt Tester (Settings > robots.txt) to check if specific URLs are blocked or allowed
- Manual check — Visit yoursite.com/robots.txt in a browser to verify the file is accessible and formatted correctly
- Test specific URLs — For each important page, verify it is not accidentally blocked
- Check after deployment — Confirm the file is at the root domain, not in a subdirectory
Debugging Checklist
- File is at the root: example.com/robots.txt (not example.com/pages/robots.txt)
- File is plain text (not HTML)
- No BOM (Byte Order Mark) at the start of the file
- Directives are case-sensitive: "Disallow" not "disallow"
- Each directive on its own line
Blocking AI Crawlers (GPTBot, CCBot, Google-Extended)
With the rise of AI training, many website owners want to block AI crawlers from scraping their content:
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: anthropic-ai
Disallow: /
# Allow regular search engines
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /This blocks AI companies from using your content for model training while keeping your site fully accessible to search engines for normal indexing and ranking.
robots.txt is advisory — well-behaved crawlers (Google, OpenAI, Anthropic) respect it, but not all scrapers do. For stronger protection, use server-side bot detection and rate limiting.
How to Use the Tool (Step by Step)
- 1
Select Website Type
Choose your platform — WordPress, e-commerce, SPA, or custom — to get recommended rules.
- 2
Add Disallow Rules
Specify which paths to block from crawlers — /admin/, /api/, /cart, /search, etc.
- 3
Add Sitemap URL
Enter the URL of your XML sitemap so crawlers can discover all your pages.
- 4
Generate and Download
Generate the robots.txt file and download it. Upload to your website root directory.
- 5
Test in Google Search Console
Use the robots.txt Tester to verify your rules work correctly for important URLs.
Frequently Asked Questions
What is a robots.txt file?+−
A plain text file at your website root (example.com/robots.txt) that tells search engine crawlers which pages to crawl and which to skip. It follows the Robots Exclusion Protocol.
Does every website need a robots.txt?+−
Yes. Without one, crawlers access everything including admin pages, search results, and duplicate content. Even a simple robots.txt with a sitemap reference improves crawl efficiency.
Does robots.txt block pages from Google search results?+−
No. It prevents crawling, not indexing. Google may still index a blocked URL if other pages link to it. Use a noindex meta tag to prevent a page from appearing in search results.
What happens if I use Disallow: /?+−
This blocks your entire website from all crawlers. Your site will disappear from search results. Never use this on a production site unless intentional (e.g., staging environments).
Should I block CSS and JavaScript in robots.txt?+−
No. Google needs CSS and JS files to render your pages correctly for mobile-first indexing. Blocking them can significantly hurt your search rankings.
How do I block AI crawlers like GPTBot?+−
Add separate User-agent blocks for GPTBot, CCBot, Google-Extended, and anthropic-ai with Disallow: /. This blocks AI training crawlers while keeping search engines allowed.
Where do I upload the robots.txt file?+−
Place it at the root of your domain: example.com/robots.txt. It must be accessible at this exact path — not in a subdirectory.
Is this robots.txt generator free and private?+−
Yes. The generator runs in your browser. No website data or URLs are sent to any server.
Generate Your robots.txt File
Select your website type, add disallow rules, include your sitemap, and download a ready-to-use robots.txt file.
Open Robots.txt Generator ->Related Guides
Meta Tag Generator Guide
Master title tags, meta descriptions, Open Graph, Twitter Cards, and canonical tags — with exact character limits and 2026 Google best practices.
Headline Analyzer Guide
Learn the science behind high-performing headlines — word count, power words, emotion triggers, and proven formulas that boost CTR.
JSON Formatter Guide
A complete developer reference for JSON syntax, common errors, formatting options, and how to validate JSON in any language or tool.