Robots.txt generator
TechnicalBuild a clean robots.txt file with blocked paths, allow exceptions, crawler blocks and your sitemap URL, then copy or download it. Free, no signup, and it runs entirely in your browser.
Upload it to the root of your domain so it lives at /robots.txt. Test it with our robots.txt tester before publishing.
How to use it
- Choose the default rule, then list the paths to block and any exceptions to allow, one per line.
- Add the user-agent names of any crawlers to block completely and enter your sitemap URL.
- Copy the output or download robots.txt and upload it to the root of your domain.
What robots.txt actually does
A robots.txt file is a plain text file at the root of your domain that tells crawlers which parts of the site they may request. Search engines read it before crawling, and reputable bots follow it. It is a crawl instruction, not a security feature: anyone can read the file, and badly behaved bots can ignore it. Anything genuinely private needs a login or server-level protection.
It is also important to know what robots.txt does not do. Blocking a URL stops crawling, but it does not guarantee the URL stays out of search results. If other pages link to a blocked URL, Google can still index the address without its content. To keep a page out of the index, let it be crawled and use a noindex meta tag instead.
What to block and what to leave alone
Good candidates for blocking are pages with no search value that waste crawl activity: admin areas, cart and checkout steps, internal search results, and filter or sort parameters that create endless near-duplicate URLs. The defaults in this tool reflect a typical WordPress or ecommerce setup.
Be careful not to block things search engines need to render your pages. Blocking CSS, JavaScript or image folders can stop Google from seeing your layout the way visitors do. The classic WordPress example is admin-ajax.php, which sits inside /wp-admin/ but is used by front-end features, so it is included as an allow exception.
How the file is structured
Rules are grouped by user-agent. A group that starts with User-agent: * applies to every crawler that does not have its own group. A crawler only follows the most specific group that names it, so if you add a dedicated group for one bot, it ignores the general rules entirely. That is why this tool writes separate full blocks for crawlers you want to shut out completely.
Within a group, Google and Bing apply the longest matching rule, and Allow wins when an Allow and Disallow rule are the same length. This is how an exception like /wp-admin/admin-ajax.php overrides a broader /wp-admin/ block.
The Sitemap line can sit anywhere in the file and must use a full URL. Listing it here helps crawlers find your sitemap even if you have not submitted it in Search Console.
Before you publish
Choose "Block everything" only for staging or development sites. Leaving it on after launch is one of the most common reasons a new site never appears in search. Once the file is ready, paste it into our robots.txt tester and check your most important URLs against Googlebot. If you want AI assistants to keep citing you while opting out of model training, the AI crawler robots.txt generator builds those rules separately.
Questions, answered
Where do I put my robots.txt file?
It must live at the root of the host, for example https://example.com/robots.txt. Crawlers do not look for it in subfolders. Each subdomain needs its own file, so blog.example.com and shop.example.com are treated separately. On most platforms you can upload it by FTP, a file manager or an SEO plugin setting.
Will blocking a page in robots.txt remove it from Google?
Not reliably. Robots.txt stops crawling, but Google can still index a blocked URL if other pages link to it, usually showing it without a description. To remove a page from results, allow crawling and add a noindex meta tag or header, or use the removal tool in Search Console for urgent cases.
Should I block my CSS and JavaScript files?
No. Google renders pages much like a browser, so it needs your stylesheets and scripts to understand layout and content. Blocking them can make pages look broken or thin to the crawler. Only block resources that are truly unrelated to how public pages display, such as private admin tools.
Do I need a robots.txt file at all?
A site without one is crawled normally, since a missing file is treated as allow everything. Still, having a file is useful for blocking low-value URLs like internal search and checkout pages, and for pointing crawlers to your sitemap. Make sure the URL returns a 200 status or a clean 404, not a server error.
Find out what is holding your rankings back
Send us your site. Within 2 business days you get a written audit, an AI visibility check and a short fix list ranked by impact. Free, with no contract attached.