How To Ignore Bot Traffic by User Agent in ASP.NET with C#

Updated October 2026: code is now inline, and there is a new, simpler approach that matches the crawler list's regular-expression patterns correctly with one compiled regex. AI crawlers such as GPTBot and ClaudeBot are now a large share of bot traffic; to block them rather than just exclude them from stats, see how to block AI crawlers in ASP.NET Core.

Quick Answer

Download the open-source crawler-user-agents.json list (over 1,500 bot patterns), combine its patterns into one compiled Regex, and check Request.Headers.UserAgent against it in middleware. Skip logging or analytics for requests that match. The code is below.

What Are Bots?

User agent "bots", often called "web crawlers", "spiders", or "robots", refer specifically to software agents that traverse the web, reading pages and following links, typically for the purpose of indexing content. They send their own unique user agent strings so that servers can identify them and treat their requests differently from those of standard web browsers if needed.

User agent strings are a part of the headers sent by web browsers, crawlers, and other software that interact with web servers. These strings provide the server with information about the software making the request, such as its type, name, version, and often some information about the device or system it's running on.

Here are some reasons and purposes behind user agent bots:

1. Search Engines: The most common bots you'll encounter are search engine crawlers like Googlebot, Bingbot, etc. They index the web so that search engines can return relevant results.

2. SEO & Web Analytics: There are bots designed to analyze websites for search engine optimization purposes.

3. Archiving: Bots like the Internet Archive's Wayback Machine crawl and archive websites for historical purposes.

4. Data Scraping: Some bots are designed to extract information from websites, either for legitimate reasons or for unauthorized data harvesting.

5. Website Monitoring: Bots can be used to monitor website uptime and performance.

6. Research: Academic researchers may use bots to study the structure and content of the web.

However, not all bots are benign. Some bots are designed with malicious intent. These might:

1. Scrape Content: Stealing content to be reposted elsewhere.
2. Brute Force Attacks: Attempting to guess login credentials.
3. Exploit Vulnerabilities: Searching websites for security weaknesses to exploit.

Because of the range of purposes and behaviors, many websites use a file called robots.txt to give hints or directives to bots about what parts of their website should or shouldn't be accessed. Respectful bots adhere to these rules, but not all bots do.

Being able to recognize the user agent of a bot, especially a malicious one, can be helpful for web administrators in managing access and optimizing their site's relationship with search engines and other online services.

Why Exclude Their Traffic?

Excluding bot traffic from your logs can be beneficial for several reasons:

1. Relevance: By removing bot traffic, your logs will better represent actual human users. This makes the analysis more relevant when making decisions based on user behavior, needs, and problems.

2. Cost Efficiency: Storing, transmitting, and analyzing data costs money. If you're paying for storage or logging services based on the amount of data or number of requests, excluding bots can save costs.

3. Performance Metrics: Bot traffic can skew performance metrics such as page load times, bounce rates, and session lengths. For a more accurate view of site performance and user experience, it's helpful to focus on human users.

4. Resource Optimization: Bots can generate a lot of traffic in a short amount of time, causing unnecessary load on servers. By reducing or managing bot traffic, server resources can be used more efficiently for actual human users.

5. Security: Some bots are malicious, looking for vulnerabilities on websites. By identifying and blocking such traffic, you can enhance your website's security. Moreover, by regularly monitoring and analyzing logs for bot patterns, you can detect and thwart potential threats.

6. Accurate Analytics: Bot traffic can inflate page views, sessions, and other analytics metrics. By filtering out this traffic, marketers and business owners get a clearer view of actual engagement and effectiveness of marketing campaigns.

7. Ad Fraud Detection: If you're running paid ads, bots can generate fake clicks that cost you money. By identifying bot traffic, you can ensure your ad budget is being spent effectively and detect potential fraud.

How to Exclude Bot Traffic:

1. robots.txt: This file allows website administrators to request that certain bots (or all bots) not crawl certain parts of their website. However, not all bots respect robots.txt, especially malicious ones.

2. User Agent Filtering: Many bots identify themselves with a unique user agent string. Server logs or analytics platforms can be configured to exclude or tag traffic from known bot user agents.

3. CAPTCHAs: For parts of a site that are especially sensitive or resource-intensive, consider implementing a CAPTCHA to ensure the user is human.

4. Advanced Detection Tools: There are tools and services designed to detect and block bots in real-time, often using a combination of user agent strings, behavior patterns, and known IP addresses.

5. Log Analysis: Regularly review server logs to identify unexpected or suspicious traffic patterns, which might suggest bot activity.

6. Rate Limiting: Implementing rate limiting on your server can prevent any user (or bot) from making too many requests in a short time frame.

Remember that not all bots are bad. For instance, search engine bots are essential for indexing your site. It's crucial to differentiate between legitimate and unwanted bot traffic and take measures accordingly.

How To Exclude Bots By User Agent in C#

There are various lists online of bot user agents. A good one is crawler-user-agents.json, which contains over 1,500 entries and is updated regularly. Each entry's pattern is a regular expression (for example PerplexityBot\/), so match it as a regex, not with string.Contains.

The simple approach: one compiled regex (.NET 8+)

Load the file once at startup, join every pattern into a single compiled regex, and register the detector as a singleton:

using System.Text.Json;
using System.Text.RegularExpressions;

// Loads crawler-user-agents.json once and checks user agents against all patterns
// with a single compiled regex. Register as a singleton.
public sealed class BotDetector
{
    private readonly Regex _botRegex;

    public BotDetector(string jsonPath)
    {
        using FileStream stream = File.OpenRead(jsonPath);
        List<CrawlerEntry> entries = JsonSerializer.Deserialize<List<CrawlerEntry>>(stream) ?? [];

        // The "pattern" values are regular expressions, not plain substrings.
        string combined = string.Join("|", entries
            .Select(e => e.pattern)
            .Where(p => !string.IsNullOrWhiteSpace(p))
            .Select(p => $"(?:{p})"));

        _botRegex = new Regex(combined,
            RegexOptions.Compiled | RegexOptions.CultureInvariant,
            matchTimeout: TimeSpan.FromMilliseconds(100));
    }

    public bool IsBot(string? userAgent)
    {
        if (string.IsNullOrWhiteSpace(userAgent))
        {
            return true; // real browsers always send a user agent
        }

        try
        {
            return _botRegex.IsMatch(userAgent);
        }
        catch (RegexMatchTimeoutException)
        {
            return false;
        }
    }

    private sealed record CrawlerEntry(string pattern);
}

Then flag each request once in middleware and skip bots wherever you record page views:

// Program.cs: flag bot requests once, then skip them when recording page views
var builder = WebApplication.CreateBuilder(args);

// Download https://raw.githubusercontent.com/monperrus/crawler-user-agents/master/crawler-user-agents.json
// into your project (and refresh it now and then).
builder.Services.AddSingleton(new BotDetector(
    Path.Combine(builder.Environment.ContentRootPath, "crawler-user-agents.json")));

var app = builder.Build();

app.Use(async (context, next) =>
{
    var detector = context.RequestServices.GetRequiredService<BotDetector>();
    context.Items["IsBot"] = detector.IsBot(context.Request.Headers.UserAgent.ToString());
    await next();
});

app.MapGet("/", (HttpContext context, ILogger<Program> logger) =>
{
    if (context.Items["IsBot"] is not true)
    {
        logger.LogInformation("Page view: {Path}", context.Request.Path); // or write to your stats table
    }
    return "Hello";
});

app.Run();

Tested against real user agents: Chrome and Safari are not flagged; Googlebot, GPTBot, ClaudeBot and empty user agents are.

The database-backed approach

If you want to add or remove patterns without redeploying, store them in a table and cache them. This was the original approach in this post: load the JSON into a model, persist each pattern to the database, and use a cache service in a base controller that logs traffic.

Model for loading the JSON

using Newtonsoft.Json;

namespace Example.Models
{
    public class UserAgentModel
    {
        [JsonProperty("pattern")]
        public string? Pattern { get; set; }

        [JsonProperty("addition_date")]
        public string? AdditionDate { get; set; }

        [JsonProperty("url")]
        public string? Url { get; set; }

        [JsonProperty("instances")]
        public List<string>? Instances { get; set; }
    }
}

Model for the database

using System.ComponentModel.DataAnnotations;
using System.ComponentModel.DataAnnotations.Schema;

namespace Example.Data.Models
{
    public class ExcludeUserAgent
    {
        [Key]
        public int ExcludeUserAgentId { get; set; }
        
        required public string UserAgent { get; set; }
        
        [Column(TypeName = "datetime2")]
        public DateTime CreateDate { get; set; }
    }
}

Repository for excluded user agents

using Example.Data.DbContextInfo;
using Example.Data.Models;
using Example.Data.Repositories.Interfaces;

namespace Example.Data.Repositories.Implementations
{
    public class ExcludeUserAgentRepository : IExcludeUserAgentRepository
    {
        private readonly IApplicationDbContext context;

        public ExcludeUserAgentRepository(IApplicationDbContext context)
        {
            this.context = context;
        }

        public IEnumerable<ExcludeUserAgent> GetAll()
        {
            return this.context.ExcludeUserAgents.ToList();
        }

        public void Create(ExcludeUserAgent excludeUserAgent)
        {
            this.context.ExcludeUserAgents.Add(excludeUserAgent);
            this.context.SaveChanges();
        }

        public bool Exists(string userAgent)
        {
            return this.context.ExcludeUserAgents.Any(e => e.UserAgent == userAgent);
        }
    }
}

Cache service

using Example.Data.Repositories.Interfaces;
using Microsoft.Extensions.Caching.Memory;

namespace Example.Web.Services
{
    public class UserAgentCacheService
    {
        private const string CACHEKEY = "ExcludedUserAgents";
        private readonly IMemoryCache memoryCache;
        private readonly IServiceProvider serviceProvider;

        public UserAgentCacheService(IServiceProvider serviceProvider, IMemoryCache memoryCache)
        {
            this.serviceProvider = serviceProvider;
            this.memoryCache = memoryCache;
            this.LoadUserAgentsToCache();
        }

        public bool IsUserAgentExcluded(string userAgent)
        {
            this.memoryCache.TryGetValue<List<string>>(CACHEKEY, out var list);

            if (list == null)
            {
                return false;
            }

            return list.Any(e => userAgent.Contains(e));
        }

        private void LoadUserAgentsToCache()
        {
            using var scope = this.serviceProvider.CreateScope();
            var excludeUserAgentRepo = scope.ServiceProvider.GetRequiredService<IExcludeUserAgentRepository>();
            var allExcludedUserAgents = excludeUserAgentRepo.GetAll();
            var excludedUserAgents = allExcludedUserAgents.Select(e => e.UserAgent).ToList();

            this.memoryCache.Set(CACHEKEY, excludedUserAgents);
        }
    }
}

If you load the crawler list into this table, change IsUserAgentExcluded to use Regex.IsMatch (or the combined regex from the first approach), because the patterns are regular expressions.

Limits of User-Agent Filtering

• Bad bots lie. Scrapers often send a normal browser user agent. User-agent filtering catches honest bots; for the rest, use rate limiting, behavior-based detection or a service such as Cloudflare's bot management.
• Verify the important ones. Anyone can claim to be Googlebot. Google and Bing publish IP ranges and support reverse DNS checks if you need to be sure.
• Keep the list fresh. New crawlers appear constantly, especially AI crawlers. Refresh the JSON file every few months.

Frequently Asked Questions

How do I detect a bot by user agent in C#?

Compare Request.Headers.UserAgent against a list of known crawler patterns, ideally the regex patterns from crawler-user-agents.json combined into one compiled Regex. Treat a missing user agent as a bot as well.

What is the GPTBot user agent?

OpenAI's training crawler identifies itself with a user agent containing GPTBot, for example Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot). Match on the GPTBot token rather than the full string, since the version changes.

Should I block bots or just exclude them from analytics?

Exclude search engine crawlers from analytics but never block them, or your pages drop out of search results. Block only bots you don't want, such as aggressive scrapers or AI crawlers you haven't agreed to.

Comments

Name

Email

Website (optional)

Comment

Post Comment