Search engine specifications · Instant generation and verification

Robots.txt Generator Robots.txt generator

Quickly create robots.txt that complies with search engine specifications and supports multiple sets of User-agent and Allow/Disallow rules. Built-in commonly used templates, sitemap settings and real-time verification.

Robots.txt generator
Robots settings
Global settings
Sitemap (can enter multiple entries)
Commonly used templates
robots.txt generated instantly
0 lines 
# Set your robots.txt rules here…
Verification results

After setting the rules, it will be verified immediately...

advertising space

❓ FAQ FAQ

Frequently asked questions about robots.txt

What is robots.txt? Does the website require it?

robots.txt It is a plain text file placed in the root directory of the website to inform search engine crawlers (such as Googlebot, Bingbot)Which pages should be crawled and which ones should not be crawled。

Almost every officially launched website should have a robots.txt, and here’s why:

  • Control crawl budget(Crawl Budget) — Guide crawlers to focus on important pages
  • Avoid sensitive content from being indexed — Admin, shopping cart, etc. should not appear in search results
  • Declare Sitemap location — Help search engines discover all pages faster
  • Reduce server load — Prevent crawlers from crawling meaningless pages

💡 Note: robots.txt is a "courtesy agreement" and is not a mandatory security mechanism. Pages that really need to be protected should use login authentication or noindex tags.

What is the difference between Allow and Disallow?

Disallow Tell the crawler "Do not crawl this path";Allow Tell the crawler "this path can be crawled".

Example:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /images/

The above rule means: crawling is prohibited /admin/ with /private/ path that starts with, but allows /images/(even if it's in a blocked directory).

💡 Allow is often used in scenarios where "first block a large area and then open specific subdirectories", such as blocking /wp-content/ but opening /wp-content/uploads/.

Why set up a Sitemap?

Declare in robots.txt Sitemap: Directives have two main benefits:

  • Accelerate discovery — When search engines crawl robots.txt, they will immediately know the Sitemap location and discover new pages faster.
  • Improve indexing efficiency — Especially for new or large websites, Sitemap ensures that all important pages are considered for indexing

Sitemap directive format:Sitemap: https://example.com/sitemap.xml, you can declare multiple Sitemap files by repeating multiple lines.

💡 It is recommended that every website declares Sitemap in robots.txt, which is a low-cost, high-efficiency SEO best practice. This tool will display a pass if there is a declared Sitemap, otherwise it will display a reminder.

What is crawl-delay? Need to set?

Crawl-delay Tell the crawler "how many seconds to wait between each request" to reduce server load.

  • Google The Crawl-delay command is not supported. Use Search Console's "Crawl Frequency" setting control instead.
  • Bing（Bingbot） Supported, but recommended only when server load is too high
  • Yandex and Baidu Support this command

💡 General websites (with normal traffic and fast page loading) do not need to set Crawl-delay because it may slow down the indexing speed of the page. Only recommended if the server load is too high.

What are some common robots.txt errors?

Here are the most common robots.txt mistakes made by website administrators:

  • Accidentally blocked the entire site — write Disallow: / And there is no Allow rule, resulting in all pages not being indexed (the most serious!)
  • Path does not start with slash — The correct format is Disallow: /admin/, the missing leading slash will be ignored or misinterpreted
  • Wrong case — user-agent、disallow is lowercase; some crawlers are case-sensitive
  • Repeat rules — The same path is announced repeatedly, causing maintenance difficulties and potential conflicts.
  • Multiple groups of the same User-agent — The search engine will only use the first set of rules with the same name, and subsequent ones will be ignored.
  • Use robots.txt for security protection — It is just a polite agreement and cannot prevent malicious access

💡 The verification function of this tool will automatically check the above problems and prompt them in the form of errors/warnings.

Where should robots.txt be placed? How to deploy?

robots.txt must be placed on the websiteroot directory, and the file name must be robots.txt：

  • Correct:https://example.com/robots.txt
  • Error:https://example.com/robots/robots.txt(Subdirectories will not be used by search engines)

Deployment method:

  1. Use this tool to set rules and download robots.txt
  2. Upload to website root directory via FTP/hosting management panel
  3. Open using browser https://您的網域/robots.txt Confirm content is correct
  4. If you need to verify, you can use Google Search Console's "robots.txt Test Tool"
Is this tool safe? Will my rules be uploaded to the server?

Totally safe! Robots.txt Generator is a 100% pure front-end tool：

  • ✅ All rule generation and verification are completed in your browser
  • ✅ No rules or URLs will be uploaded to the server
  • ✅ No need to register, log in or install any software
  • ✅ Can be used completely offline after loading
  • ✅ None of your settings will be saved

💡 Even unpublished site structures or internal paths can be generated here with complete confidence.

📖 The Complete Guide to Robots.txt: Search Engine Specifications and SEO Advice

Learn more about how robots.txt works and best practices

Function introduction

Robots.txt Generator is a free online robots.txt creation tool launched by ToolHub. You can add multiple groups in the graphical interface User-agent(Googlebot, Bingbot, Baiduspider, YandexBot, etc.), settings Allow / Disallow rules, announcements Sitemap, adjust Crawl-delay with Host, the corresponding robots.txt content is instantly generated on the right side, and verification of grammar, blocking range, and duplication rules is automatically performed. It has built-in 7 commonly used templates such as WordPress, Blog, and E-commerce, and you can complete the basic settings with one click.

What is robots.txt?

robots.txt It is the "crawler command file" of the website, located in the root directory of the website. It uses a standardized format to tell search engine crawlers which pages can be crawled and which pages should be avoided. it follows Robots Exclusion Protocol（REP）, is the basic link of SEO technical optimization.

A typical robots.txt consists of one or more "rule groups", each group contains User-agent corresponding to Allow / Disallow Rules:

User-agent: * Disallow: /admin/ Disallow: /private/ Allow: /images/ Sitemap: https://example.com/sitemap.xml

How to use robots.txt?

Rule group (User-agent)

Each rule group starts with User-agent: At the beginning, specify the crawler to which the rule applies. General rules for use * Indicates all crawlers; you can also set exclusive rules for specific crawlers (such as Googlebot). Please note:Google only uses the first matching User-agent group, so the most specific crawler rules should be placed first.

Allow and Disallow

Disallow Block the specified path,Allow Open the specified path (Google natively supports Allow). Path must begin with a slash / Beginning, for example Disallow: /admin/. When Allow and Disallow exist at the same time,The longest matching path rule takes precedence——This is the basis for "blocking large areas first and then opening subdirectories".

Sitemap declaration

Sitemap: The directive tells the crawler the location of the XML Sitemap. This command has nothing to do with User-agent. It is usually placed at the end of the file and can declare multiple transactions. It is recommended that all websites declare a Sitemap in robots.txt.

Crawl-delay and Host

Crawl-delay Set the number of seconds between crawler requests (not supported by Google);Host Declare primary domain (supported by Yandex to avoid duplicate indexing of IP domains).

Common mistakes and how to avoid them

  • Block entire site — used alone Disallow: / This will make the entire site unindexable, which is the most impactful error. Be sure to check if it was intentional
  • Path format error — Missing leading slash (Disallow: admin/) or use a URL instead of a path (Disallow: https://…) are common mistakes
  • Mixed case — Instruction names must be lowercase (user-agent、disallow), the value is case-sensitive
  • Repeat and Conflict Rules — Repeated declaration of the same path will cause maintenance difficulties; the latter is invalid when the User-agent group with the same name is repeated
  • Misuse of robots.txt for security protection — It is just a protocol, not a firewall. Sensitive pages should use verification mechanisms

SEO advice

  • Block but remain indexable — Blocking should not appear on functional pages of search results (shopping cart, account, admin) rather than on important content pages
  • Be sure to declare your Sitemap — Let the crawler immediately know the content map every time it visits
  • Avoid blocking CSS/JS — In modern SEO, blocking CSS/JS files prevents Google from understanding the page rendering results
  • Verify after modification — Use Search Console’s robots.txt testing tool to confirm after deployment
  • keep it simple — robots.txt should be concise and clear, overly complex rules are prone to errors

Browser compatibility

This tool is developed based on modern web standards and is compatible with the latest versions of all major browsers: Google Chrome, Mozilla Firefox, Apple Safari, Microsoft Edge and Opera. It is recommended to use the latest version for the best experience.

Privacy protection

ToolHub takes the privacy of your data seriously. Robots.txt Generator adopted 100% pure front-end architecture, all rule generation and verification are completed in your browser, and no data will be transmitted to external servers. We will not store, analyze or share any rule content you enter, please feel free to use it.

Robots.txt Generator - Instructions for using the online robots.txt generator

Free online robots.txt generator. Quickly create robots.txt that complies with search engine specifications and supports multiple sets of User-agent, Allow/Disallow rules, Sitemap, Crawl-delay, common templates and real-time verification. Pure front-end local computing ensures privacy and security.

Recommended operating procedures

  1. Confirm the input format, units and necessary fields first to avoid incorrect data being transmitted all the way to the results.
  2. Adjust tool options and read instant tips to cross-check with representative boundary values.
  3. Double-check the results before copying or downloading; for use in formal processes, complete validation in the target environment.

Quality and privacy

This tool executes natively in the browser and does not rely on backend processing. The results will be affected by input quality, browser support and related technical specifications. Please keep the original version and backup of important data.

Operation successful