How To Block AI Crawlers Like GPTBot and ClaudeBot in ASP.NET Core and IIS

AI companies run crawlers that collect web pages to train models and to answer questions in AI search and chat products. Some site owners welcome that; others want to opt out, or are simply tired of the extra load. This guide shows three layers of control for .NET sites: robots.txt, ASP.NET Core middleware, and an IIS rewrite rule, all without touching Googlebot or Bingbot.

Quick Answer

1. Add the AI crawler user agents to robots.txt with Disallow: /. Well-behaved crawlers stop crawling.
2. For crawlers that ignore it, or user-initiated fetchers that don't check robots.txt, return 403 from middleware when the User-Agent contains one of their tokens.
3. Leave search engine crawlers alone, or your pages disappear from search results.

The Main AI Crawlers and Their User Agents

Companies add and rename crawlers from time to time, so check each vendor's crawler documentation when you set this up. A typical user agent looks like Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot); match on the token, not the full string.

Step 1: robots.txt

Put robots.txt in wwwroot (ASP.NET Core) or the site root (IIS). Several User-agent lines followed by one Disallow apply the rule to all of them:

# Block AI training and AI search crawlers, allow everything else
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Block training, but stay visible in AI search?

AI search products (ChatGPT search, Claude, Perplexity) can send readers to your site with a cited link. If you want to opt out of model training but still appear in those answers, block only the training crawlers:

# Opt out of AI *training* only; still allow AI search/answer engines to cite you
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

robots.txt is a request, not an enforcement mechanism. The major AI companies say their crawlers honor it, but user-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) act on a person's behalf and may not check it. If you need a hard block, add Step 2.

Step 2: Block AI Crawlers in ASP.NET Core Middleware

This middleware returns 403 Forbidden to any request whose user agent contains one of the tokens, except for /robots.txt so crawlers can still read your rules:

// Returns 403 to AI training/answer crawlers that identify themselves.
// Search engine crawlers (Googlebot, Bingbot) are NOT in this list.
public static class AiCrawlerBlocker
{
    private static readonly string[] BlockedTokens =
    [
        "GPTBot", "OAI-SearchBot", "ChatGPT-User",       // OpenAI
        "ClaudeBot", "Claude-SearchBot", "Claude-User",  // Anthropic
        "PerplexityBot", "Perplexity-User",              // Perplexity
        "CCBot",                                         // Common Crawl
        "Bytespider",                                    // ByteDance
        "meta-externalagent",                            // Meta
        "Amazonbot",                                     // Amazon
        "Applebot-Extended", "Google-Extended"           // robots.txt tokens; harmless here
    ];

    public static bool IsAiCrawler(string? userAgent) =>
        !string.IsNullOrEmpty(userAgent) &&
        BlockedTokens.Any(token => userAgent.Contains(token, StringComparison.OrdinalIgnoreCase));

    public static IApplicationBuilder UseAiCrawlerBlocker(this IApplicationBuilder app) =>
        app.Use(async (context, next) =>
        {
            // Always let crawlers read robots.txt so they can see your rules.
            if (!context.Request.Path.StartsWithSegments("/robots.txt") &&
                IsAiCrawler(context.Request.Headers.UserAgent.ToString()))
            {
                context.Response.StatusCode = StatusCodes.Status403Forbidden;
                return;
            }
            await next();
        });
}

Register it early in the pipeline:

// Program.cs
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.UseAiCrawlerBlocker(); // early, before static files and routing
app.UseStaticFiles();

app.MapGet("/", () => "Hello, humans and search engines");
app.Run();

Tested on .NET 10: a request with a GPTBot user agent gets 403, the same bot can still fetch /robots.txt, and a normal Chrome user agent gets 200. Googlebot and Bingbot don't contain any of these tokens, so they are unaffected.

Step 3: Block AI Crawlers in IIS (No Code)

For any site on IIS, including classic ASP.NET and static sites, the URL Rewrite module can do the same thing in web.config:

<!-- IIS: block AI crawlers by user agent with the URL Rewrite module -->
<configuration>
  <system.webServer>
    <rewrite>
      <rules>
        <rule name="Block AI crawlers" stopProcessing="true">
          <match url="^robots\.txt$" negate="true" />
          <conditions>
            <add input="{HTTP_USER_AGENT}" pattern="GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|meta-externalagent|Amazonbot" />
          </conditions>
          <action type="CustomResponse" statusCode="403" statusReason="Forbidden" statusDescription="AI crawlers are not allowed" />
        </rule>
      </rules>
    </rewrite>
  </system.webServer>
</configuration>

What User-Agent Blocking Can't Do

• Scrapers that pretend to be browsers won't be caught. Rate limiting (ASP.NET Core has built-in rate-limiting middleware) and CDN bot protection handle those better.
• Content already collected isn't removed by blocking now. Blocking only affects future crawls.
• Datasets built by others (such as Common Crawl archives) may already include your pages; blocking CCBot keeps you out of future crawls.

Check That It Works

curl -I -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://example.com/
curl -I -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/

The first should return 403; the second should return 200. Your server or CDN logs will show whether blocked crawlers keep coming back.

Frequently Asked Questions

Does blocking GPTBot affect my Google rankings?

No. GPTBot is OpenAI's crawler and has nothing to do with Google Search. Even Google-Extended only controls whether Google may use your content for Gemini training; it doesn't affect Googlebot or rankings.

How do I block ChatGPT from reading my website?

Disallow GPTBot, OAI-SearchBot and ChatGPT-User in robots.txt, and add server-side blocking of those tokens for a hard stop, as shown above.

What is the ClaudeBot user agent?

Anthropic's training crawler includes the token ClaudeBot in its user agent. Anthropic also uses Claude-SearchBot for search indexing and Claude-User for pages fetched at a user's request.

Should I block AI crawlers?

It depends on your goals. Blocking protects your content and server resources; allowing AI search crawlers can bring cited traffic. Many sites block training crawlers and allow search crawlers.

Related: excluding bot traffic from your logs and analytics in C#.

Comments

Name

Email

Website (optional)

Comment

Post Comment