Blocking AI scrapers from your content
Leaky Paywall can keep AI crawlers from taking your articles, but only if the paywall removes the content on the server instead of hiding it with JavaScript. There are three settings to check inside Leaky Paywall and WordPress, plus your robots.txt file.
1. Set your restrictions to zero
Go to Leaky Paywall > Settings > Restrictions. In the Restrictions table, set Number Allowed to 0. Add a row for every post type that holds content you want protected, not just Posts.
With 0 allowed, nobody reads the content until they log in.
2. Turn off Alternative Restriction Handling
This is the setting that matters most for scraping, and it is on by default.
On the same Restrictions tab, uncheck Alternative Restriction Handling.
When it is checked, WordPress sends the full article to the browser and JavaScript hides it afterwards. A scraper reading the raw HTML never runs that JavaScript, so it gets the whole article. When it is unchecked, Leaky Paywall swaps the article for a short excerpt and your subscribe message before the page ever leaves the server, so the full text is not in the HTML at all. That applies to article pages and to the WordPress REST API.
Alternative Restriction Handling exists for sites with heavy page caching. If you uncheck it and your paywall message stops showing correctly, that is a caching problem: see Caching with Leaky Paywall.
3. Decide about Search Engine Bot Bypass
Also on the Restrictions tab. When Search Engine Bot Bypass is checked, Googlebot, Bingbot, and a short list of other search crawlers get the full article so your content stays indexed. The dedicated AI crawlers such as GPTBot and ClaudeBot are not on that list and never bypass the paywall.
This is the one real decision in the setup, because letting Googlebot in also lets Google's AI in. AI Overviews and AI Mode are built from the same Search index that Googlebot fills, so if Googlebot can read your full articles, Google can summarize them directly in search results. Bing works the same way: Bingbot feeds Copilot.
Blocking Google-Extended in robots.txt does not change that. Google-Extended covers Gemini training and grounding only. Google states that it does not affect whether your content appears in AI Overviews.
So there are two positions and nothing in between:
- Unchecked: everything is blocked, including Google. Google indexes only your headline and the first 100 characters of each article, so you rank for far less and your search traffic will drop.
- Checked: Google keeps indexing your full articles, and Google can use them in AI Overviews and AI Mode.
If the goal is keeping your archive out of AI entirely, leave it unchecked.
4. Stop your RSS feed from publishing full articles
The paywall does not filter RSS feeds. If WordPress is set to publish full text in the feed, anything reading yoursite.com/feed/ gets complete articles no matter how the paywall is configured. This is the most common gap on sites that otherwise look locked down.
In WordPress go to Settings > Reading and set For each post in a feed, include to Excerpt.
To check, open yoursite.com/feed/ in a browser and read one item. If the whole article is there, the setting is still on Full text.
5. Update robots.txt
robots.txt is a request, not a block. The AI companies that honor it will stay away, the rest will ignore it, which is why the paywall settings above are what actually protect you. It is still worth setting, because it is the public record of what you asked for if you ever negotiate a licensing deal.
A starting robots.txt that blocks the main AI crawlers:
# AI training and AI answer crawlers User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-Web User-agent: anthropic-ai User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: FacebookBot User-agent: Amazonbot User-agent: Bytespider User-agent: CCBot User-agent: cohere-ai User-agent: Diffbot User-agent: ImagesiftBot User-agent: Omgilibot User-agent: YouBot Disallow: / # Everyone else, including search engines User-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml
Two of those entries are easy to get wrong:
- Google-Extended opts you out of Gemini training and grounding. It does not affect Google Search indexing, and it does not keep you out of AI Overviews. Both of those follow Googlebot, so they are controlled by the setting in step 3.
- Applebot-Extended opts you out of Apple AI training. Do not block plain Applebot unless you also want to disappear from Siri and Spotlight.
Replace the Sitemap line with your own domain. If your site was set up from a template, check that line: pointing at somebody else's sitemap is a common copy and paste error.
On WordPress you can edit robots.txt through Yoast SEO (Yoast SEO > Tools > File Editor), Rank Math (Rank Math > General Settings > Edit robots.txt), or by uploading a robots.txt file to the root folder of your site.
New AI crawlers appear regularly, so this list needs a review every few months.
How to check that it worked
Open one of your articles in a private browsing window so you are logged out. View the page source (Ctrl+U on Windows, Cmd+Option+U on a Mac) and search it for a sentence from the middle of the article.
If the sentence is not there, the article is not in the HTML and a scraper cannot read it. If it is there, go back to step 2.
Then open yoursite.com/feed/ and do the same check on the feed.