YouTube Telegram X
Tested WordPress, SEO & site-speed guides
DanialTV – Danial Taherifar
Free tools

robots.txt Tester & WordPress Templates

Paste your robots.txt and test any URL against Google’s real matching rules, for Googlebot, Bing and AI crawlers. With ready-made WordPress and WooCommerce templates.

By Danial Taherifar · Updated

Checks
    Try:
    Same URL, every crawler
    CrawlerResultMatching rule

    Runs entirely in your browser and follows Google’s documented matching rules (RFC 9309). Nothing you paste is sent to DanialTV.

    How to use it

    1. Enter your domain and click Open my live robots.txt. Copy what you see and paste it into the box, or load one of the WordPress templates.
    2. Type any URL or path from your site, or use the quick buttons (image, jQuery, search page…).
    3. Read the result: Allowed or Blocked, the exact line that decided it, and which User-agent group was used.
    4. Check the table to see the same URL for Google, Bing, ChatGPT search and the AI training crawlers.
    5. Fix anything under Checks, then copy the result back into Yoast, Rank Math or your robots.txt file.

    How the tester decides

    It follows Google’s documented rules (RFC 9309), and I checked it against every example in Google’s own documentation before publishing:

    • One group per crawler. A crawler uses the group with the most specific User-agent that matches it and ignores all others. If User-agent: Googlebot-Image exists, Googlebot-Image ignores everything under User-agent: *.
    • Longest rule wins. When an Allow and a Disallow both match, the one with the longer path wins. On a tie, Allow wins.
    • * and $. * matches any characters; $ means “the URL ends here”.
    • Paths are case-sensitive. /Wp-Admin/ is not /wp-admin/.
    • Ignored lines. Google ignores Crawl-delay and Noindex in robots.txt; the tester tells you when you rely on them.

    Want the full WordPress guide, including what I found when I ran six real WordPress sites through this tester? Read robots.txt for WordPress: what to block and what never to block.

    FAQ

    Can this tester read my robots.txt automatically?

    Browsers don’t allow one site to read another site’s files directly, so the tool opens yours in a new tab and you paste it in. That also means nothing about your site is stored here.

    Does blocking a page in robots.txt remove it from Google?

    No. A blocked URL can still be indexed if other pages link to it; Google just can’t read it. To remove a page, allow crawling and add a noindex meta tag instead.

    Does blocking Google-Extended hurt my rankings?

    No. Google says Google-Extended doesn’t affect inclusion in Search or ranking. It only controls whether your content is used to train Gemini. Search crawling is still done by Googlebot.

    Sources: Google: How Google interprets robots.txt, Google: common crawlers and Google-Extended.