robots.txt for WordPress: What to Block and What Never to Block

Short answer: for most WordPress sites the right robots.txt is short. Block /wp-admin/ but allow admin-ajax.php, optionally block internal search results, add your sitemap, and never block /wp-content/, /wp-includes/ or URLs with ?ver=. That last one is the mistake I found on two of my own sites while writing this.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap_index.xml

Tested on WordPress 7.1 with Yoast SEO, November 2026. Every rule below was checked with my free robots.txt tester, which follows Google’s own matching rules.

How WordPress creates robots.txt (you may not have a file at all)

Most WordPress sites have no robots.txt file on the server. When someone requests /robots.txt, WordPress builds one on the fly with the do_robots() function. Three things decide what it shows:

1. A real file wins. If a robots.txt file exists in your site’s root folder, the web server sends it and WordPress never runs. Plugin settings for robots.txt then do nothing, which confuses a lot of people.
2. WordPress core outputs User-agent: *, Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php. Since WordPress 5.5 it also adds a Sitemap: line for wp-sitemap.xml, but only when the site is public.
3. SEO plugins hook into the robots_txt filter and change it. Yoast, for example, replaces the whole thing.

Here is what Yoast SEO outputs on this site right now, with no robots.txt file on the server:

# START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow:

Sitemap: https://danialtv.com/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK

Note the empty Disallow:. It allows everything, including /wp-admin/. That isn’t a problem: admin pages redirect to the login screen, and WordPress marks the login page noindex. But it does mean the core WordPress rules are gone once Yoast is active.

The “Discourage search engines” checkbox doesn’t use robots.txt anymore

Before WordPress 5.3, ticking Settings → Reading → Discourage search engines added Disallow: /. Since 5.3 it adds a noindex robots meta tag to every page instead. That is better: a page blocked by robots.txt can still show up in Google, but a noindex page can’t. If you see Disallow: / on a live site today, it came from a plugin or a forgotten file, not from that checkbox.

What I found on six real WordPress sites

I ran the live robots.txt of six WordPress sites I run (a personal blog, a magazine, three music sites and this one) through the tester and checked 15 typical WordPress URLs on each. Four out of six had at least one problem:

Site	What its robots.txt does	Problem found
This site (Yoast)	Allows everything, adds sitemap	None
Music site 1	Core WordPress rules + sitemap	None
Magazine	Core rules, blocks ?replytocom=, separate Googlebot-Image group	Googlebot-Image ignores all the other rules (see below)
Personal blog	Blocks /wp-admin/ and search, allows uploads	admin-ajax.php blocked; sitemap URL is http:// on an https site
Music site 2	19 rules, including Disallow: /*?*	Plugin CSS and JS blocked for Googlebot
Music site 3	10 rules, including Disallow: /*?*	All theme and plugin CSS and JS blocked for Googlebot

The ?ver= trap: how one rule hides your CSS from Google

A popular tip says: “block URLs with parameters to save crawl budget”, written as Disallow: /*?*. On WordPress, that blocks far more than you think. WordPress loads every theme and plugin file with a version parameter:

/wp-content/themes/your-theme/style.css?ver=6.7
/wp-includes/js/jquery/jquery.min.js?ver=3.7.1
/wp-content/plugins/elementor/assets/css/frontend.min.css?ver=3.2

All three match /*?*, so Googlebot can’t fetch them. Google renders pages before ranking them; without CSS and JavaScript it may see a broken, unstyled page. One of the two sites had added Allow lines for its own theme folder, so its main stylesheet was fine, but every plugin file was still blocked. The other had no exceptions at all.

The fix: block only the parameters you actually want to keep out, instead of all of them:

# Instead of: Disallow: /*?*
Disallow: /*?replytocom=
Disallow: /?s=
Disallow: /*?add-to-cart=
Disallow: /*?orderby=

Paste your robots.txt into the tester and click the jQuery button: if it says Blocked, you have this problem.

The specific-group trap: Googlebot-Image ignores your other rules

Three of the sites had this at the end of their file, usually added to “make sure images get indexed”:

User-agent: Googlebot-Image
Allow: /*

Google’s rule is that a crawler follows only one group: the one with the most specific matching User-agent. So Googlebot-Image now ignores everything under User-agent: *, including Disallow: /wp-admin/. Nothing breaks here, because images were never blocked in the first place, so the block is simply unnecessary. But the same pattern with User-agent: Googlebot would silently cancel all your general rules for Google Search. If you give a crawler its own group, repeat every rule it should still follow.

WordPress paths: block, allow, or leave alone

Path	Recommendation	Why
/wp-admin/	Block (optional)	Nothing to index there. WordPress core blocks it by default.
/wp-admin/admin-ajax.php	Allow	Themes and plugins load front-end content through it.
/wp-content/uploads/	Never block	Your images. Blocking them removes you from Google Images.
/wp-content/themes/, /wp-content/plugins/	Never block	CSS and JS Google needs to render pages.
/wp-includes/	Never block	Contains jQuery and core scripts used on the front end.
/?s= and /search/	Block	Internal search results are thin, endless pages.
?replytocom=	Block	Duplicate of every post with comments, one per reply link.
/feed/	Leave alone	Google rarely indexes feeds, and they help discovery.
/tag/, /author/, /page/2/	Don’t use robots.txt	If you don’t want them in Google, use noindex in your SEO plugin. Blocked pages can still be indexed.
/wp-json/	Leave alone	Some themes and blocks fetch content from the REST API.
/wp-login.php	Optional	Already noindex; blocking it doesn’t stop bots that attack it.

WooCommerce additions

Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?add-to-cart=
Disallow: /*?orderby=
Disallow: /*?filter_
Disallow: /*?min_price=

Filter and sort parameters can create thousands of near-identical URLs on a store. Block the parameter names you actually use; check them in your shop’s URLs first.

AI crawlers: training versus search

Many site owners now block AI crawlers. The important part is to block the training crawlers without blocking the ones that send you visitors:

User-agent	What it does	If you block it
Google-Extended	Controls use of your content for Gemini training	No effect on Google Search or rankings (Google’s own statement)
GPTBot	OpenAI model training	Not used for training; ChatGPT search unaffected
OAI-SearchBot	ChatGPT search results	You disappear from ChatGPT search answers
ClaudeBot	Anthropic model training	Not used for training
CCBot	Common Crawl public dataset	Left out of a dataset many AI models use
Applebot-Extended	Apple AI training	Apple search (Applebot) unaffected

The tester has a ready template for this (“Allow search, block AI training”) and shows each of these crawlers side by side for any URL.

How to edit robots.txt in WordPress

• Yoast SEO: Yoast SEO → Tools → File editor.
• Rank Math: Rank Math → General Settings → Edit robots.txt.
• A real file: create robots.txt in the site’s root folder with File Manager or FTP. Remember it overrides every plugin setting.
• Code: use the robots_txt filter in a small plugin or your child theme:

add_filter('robots_txt', function ($output, $public) {
    if ($public) {
        $output .= "Disallow: /?s=\nDisallow: /*?replytocom=\n";
    }
    return $output;
}, 20, 2);

After any change, open yourdomain.com/robots.txt to see the real result, test a few URLs in the tester, and check the robots.txt report in Search Console (Settings → robots.txt). Google caches the file for up to 24 hours.

Checklist

• No Disallow: / on the live site.
• The jQuery and image test URLs are Allowed for Googlebot.
• No Disallow: /*?*; block specific parameters instead.
• admin-ajax.php is allowed if /wp-admin/ is blocked.
• The Sitemap line is a full https:// URL.
• Any crawler with its own group repeats the rules it should follow.
• No Noindex or Crawl-delay lines you rely on for Google.
• Pages you want out of Google use noindex, not robots.txt.

FAQ

Where is robots.txt in WordPress?

Usually nowhere: WordPress generates it when /robots.txt is requested. If you can see the file in your root folder with File Manager, that file is what visitors and crawlers get.

Why are my robots.txt changes in Yoast not showing?

Most often because a real robots.txt file exists in the root folder, which the server sends before WordPress runs. The other cause is a caching plugin or CDN serving an old copy; purge the cache.

Should I block /wp-content/plugins/?

No. Plugins load CSS and JavaScript from there, and Google needs those files to render your pages.

What happens if robots.txt returns an error?

A 404 is treated as “no restrictions”. A 5xx server error is worse: Google stops crawling the site for the first 12 hours, then falls back to its last cached copy.

Sources: Google: How Google interprets the robots.txt specification, Google: common crawlers, WordPress: do_robots(), WordPress: WP_Sitemaps::add_robots().