🔍 SEO Tools
Robots.txt Generator
Generate a valid robots.txt file to control search engine crawlers. Allow or block specific bots and paths, then download the file.
Crawl Rules
robots.txt Preview
# robots.txt will appear here…
Advertisement
Free Robots.txt Generator — Create SEO-Optimised Robots.txt
Our free robots.txt generator creates a correctly formatted robots.txt file with crawl directives for Google, Bing and all bots. Block private pages, add your sitemap URL and control bot access with a simple visual interface.
🤖 All Major Bots
Configure rules for Googlebot, Bingbot and all crawlers.
🚫 Block URLs
Disallow specific paths from being crawled.
🗺 Sitemap Link
Automatically includes your sitemap URL.
⬇ Download File
Download the ready-to-upload robots.txt file.
How to Use — Step by Step
1
Choose your crawl policy
Allow all bots, block all, or configure specific rules.
2
Add disallow rules
Enter paths to block — /admin/, /private/, /thank-you/
3
Add your sitemap URL
Include your sitemap URL so search engines can find it.
4
Download and upload
Download robots.txt and upload to your website root directory.
Frequently Asked Questions
Does robots.txt prevent Google from showing pages?
No — it prevents crawling, not indexing. A blocked page can still appear in results if linked externally. Use the noindex meta tag to truly prevent a page from appearing in search results.
What if I have no robots.txt?
Crawlers default to indexing everything accessible. Fine for most small sites but wastes crawl budget on admin and private pages for larger sites.
Can I block specific scrapers?
You can add User-agent rules, but bad actors rarely respect robots.txt. Use server-level or Cloudflare firewall rules to actually stop unwanted traffic.
What is crawl budget?
The number of pages Googlebot crawls per time frame. Block low-value pages (duplicate content, admin URLs, parameters) so Googlebot focuses on important content.
Should I block CSS and JavaScript?
No — Google needs them to render your page. Blocking causes Google to see a broken version, which hurts rankings. Only block truly private paths.