Robots.txt Generator
Generate a valid robots.txt file to control search engine crawling of your website.
Robots.txt Generator
Create a valid robots.txt file to control how search engine crawlers access your site.
Group 1
No disallow rules
Sitemap URLs
No sitemaps added
User-agent: * Allow: /
Robots.txt Syntax Guide
- User-agent:
- Specifies the crawler. Use
*for all bots. - Disallow:
- Blocks a path from crawling.
Disallow: /blocks the entire site. - Allow:
- Overrides a Disallow for a specific sub-path (supported by Google, Bing).
- Crawl-delay:
- Seconds between requests (honored by Bing/Yandex, ignored by Google).
- Sitemap:
- Full URL to your XML sitemap. Can list multiple.
Place the file at the root of your domain: https://example.com/robots.txt
Robots.txt Generator कसरी प्रयोग गर्ने
- 1
Choose a quick preset or start from scratch by adding user-agent groups.
- 2
Select a user-agent (*, Googlebot, Bingbot, or custom) for each group.
- 3
Add Allow and Disallow paths using the + buttons.
- 4
Optionally set a Crawl-delay and add Sitemap URLs.
- 5
Review the live preview, then copy to clipboard or download the file.
Robots.txt Generator बारेमा
Create a production-ready robots.txt file with an intuitive visual editor — control which pages search engine crawlers can or cannot access on your website.
Key Features
- Quick Presets: Allow All, Block All, Block AI Bots (GPTBot, ChatGPT-User, CCBot, Google-Extended), Standard, and WordPress Default.
- Custom Rules: Add multiple user-agent groups with Allow/Disallow paths, crawl-delay, and sitemap URLs.
- Live Preview: See your robots.txt output update in real time as you configure rules.
- Validation: Built-in warnings for conflicting rules.
- Copy & Download: Copy to clipboard or download as a robots.txt file.
Who Is This For?
Perfect for SEO professionals, web developers, and website owners who want to control search engine and AI bot crawling without manually writing syntax.
Privacy & Security
All generation runs in your browser — no data stored, no signup required.