Blocking AI scrapers from your content

Leaky Paywall can keep AI crawlers from taking your articles, but only if the paywall removes the content on the server instead of hiding it with JavaScript. There are three settings to check inside Leaky Paywall and WordPress, plus your robots.txt file.

Watch: Should publishers block AI bots? (from Who Owns Your Readers: You or AI?)

1. Set your restrictions to zero

Go to Leaky Paywall > Settings > Restrictions. In the Restrictions table, set Number Allowed to 0. Add a row for every post type that holds content you want protected, not just Posts.

With 0 allowed, nobody reads the content until they log in.

2. Check your Restriction Method

This is the setting that matters most for scraping. On the same Restrictions tab, look at Restriction Method:

  • Automatic (recommended, and the default for new installs): posts no logged-out visitor can read, such as those with Number Allowed set to 0, are restricted on the server. Metered posts are checked in the browser.
  • Server: every restricted post is restricted on the server, metered posts included.
  • JavaScript: every restricted post is checked in the browser. WordPress sends the full article and JavaScript hides it afterwards. A scraper reading the raw HTML never runs that JavaScript, so it gets the whole article.

With Automatic or Server, Leaky Paywall swaps a protected article for its teaser and your subscribe message before the page ever leaves the server, so the full text is not in the HTML at all. That applies to article pages and to the WordPress REST API. So with Number Allowed at 0 (step 1), Automatic protects your content. Choose Server if you also want metered posts kept out of the page source.

If you're on an older version, this was a single Alternative Restriction Handling checkbox. Checked is the same as JavaScript, unchecked is the same as Server. See Choosing a restriction method for more.

Server doesn't work with page caching. If you choose it and your paywall message stops showing correctly, that is a caching problem: see Caching with Leaky Paywall.

3. Decide about Search Engine Bot Bypass

Also on the Restrictions tab. When Search Engine Bot Bypass is checked, Googlebot, Bingbot, and a short list of other search crawlers get the full article so your content stays indexed. The dedicated AI crawlers such as GPTBot and ClaudeBot are not on that list and never bypass the paywall.

This is the one real decision in the setup, because letting Googlebot in also lets Google's AI in. AI Overviews and AI Mode are built from the same Search index that Googlebot fills, so if Googlebot can read your full articles, Google can summarize them directly in search results. Bing works the same way: Bingbot feeds Copilot.

Blocking Google-Extended in robots.txt does not change that. Google-Extended covers Gemini training and grounding only. Google states that it does not affect whether your content appears in AI Overviews.

So there are two positions and nothing in between:

  • Unchecked: everything is blocked, including Google. Google indexes only your headline and the teaser of each article (set by Lead In Elements), so you rank for far less and your search traffic will drop.
  • Checked: Google keeps indexing your full articles, and Google can use them in AI Overviews and AI Mode.

If the goal is keeping your archive out of AI entirely, leave it unchecked.

If you check it, also check Paywall Schema Markup on the same tab. It adds structured data (isAccessibleForFree: false) so Google knows the full text it sees is paywalled content, as Google's guidelines for paywalled content ask.

Note: Leaky Paywall recognizes search engine bots by their user agent, which anyone can fake. With the bypass on, a scraper pretending to be Googlebot would get the full text. To guard against that, block requests that claim to be Googlebot but don't come from Google at your firewall or CDN. Cloudflare can do this with its verified bots list.

4. Stop your RSS feed from publishing full articles

The paywall does not filter RSS feeds. If WordPress is set to publish full text in the feed, anything reading yoursite.com/feed/ gets complete articles no matter how the paywall is configured. This is the most common gap on sites that otherwise look locked down.

In WordPress go to Settings > Reading and set For each post in a feed, include to Excerpt.

To check, open yoursite.com/feed/ in a browser and read one item. If the whole article is there, the setting is still on Full text.

5. Update robots.txt

robots.txt is a request, not a block. The AI companies that honor it will stay away, the rest will ignore it, which is why the paywall settings above are what actually protect you. It is still worth setting, because it is the public record of what you asked for if you ever negotiate a licensing deal.

A starting robots.txt that blocks the main AI crawlers:

# AI training and AI answer crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: FacebookBot
User-agent: Amazonbot
User-agent: Bytespider
User-agent: CCBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: ImagesiftBot
User-agent: Omgilibot
User-agent: YouBot
Disallow: /

# Everyone else, including search engines
User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Two of those entries are easy to get wrong:

  • Google-Extended opts you out of Gemini training and grounding. It does not affect Google Search indexing, and it does not keep you out of AI Overviews. Both of those follow Googlebot, so they are controlled by the setting in step 3.
  • Applebot-Extended opts you out of Apple AI training. Do not block plain Applebot unless you also want to disappear from Siri and Spotlight.

Replace the Sitemap line with your own domain. If your site was set up from a template, check that line: pointing at somebody else's sitemap is a common copy and paste error.

On WordPress you can edit robots.txt through Yoast SEO ( Yoast SEO > Tools > File Editor), Rank Math (Rank Math > General Settings > Edit robots.txt), or by uploading a robots.txt file to the root folder of your site.

New AI crawlers appear regularly, so this list needs a review every few months.

How to check that it worked

Open one of your articles in a private browsing window so you are logged out. View the page source (Ctrl+U on Windows, Cmd+Option+U on a Mac) and search it for a sentence from the middle of the article.

If the sentence is not there, the article is not in the HTML and a scraper cannot read it. If it is there, go back to step 2.

Then open yoursite.com/feed/ and do the same check on the feed.

Related

What happens if I use Leaky Paywall as a hard paywall?

More on AI and your paywall

The Truth About Paywall Evasion, AI, Google and Passwords

Cloudflare Pay-Per-Crawl: sustainable income or just spare change?

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.

Still need help? Contact Us Contact Us