How To Ignore Bot Traffic by User Agent in ASP.NET with C#

What Are Bots?

User agent "bots", often called "web crawlers", "spiders", or "robots", refer specifically to software agents that traverse the web, reading pages and following links, typically for the purpose of indexing content. They send their own unique user agent strings so that servers can identify them and treat their requests differently from those of standard web browsers if needed.

User agent strings are a part of the headers sent by web browsers, crawlers, and other software that interact with web servers. These strings provide the server with information about the software making the request, such as its type, name, version, and often some information about the device or system it's running on.

Here are some reasons and purposes behind user agent bots:

1. Search Engines: The most common bots you'll encounter are search engine crawlers like Googlebot, Bingbot, etc. They index the web so that search engines can return relevant results.

2. SEO & Web Analytics: There are bots designed to analyze websites for search engine optimization purposes.

3. Archiving: Bots like the Internet Archive's Wayback Machine crawl and archive websites for historical purposes.

4. Data Scraping: Some bots are designed to extract information from websites, either for legitimate reasons or for unauthorized data harvesting.

5. Website Monitoring: Bots can be used to monitor website uptime and performance.

6. Research: Academic researchers may use bots to study the structure and content of the web.

However, not all bots are benign. Some bots are designed with malicious intent. These might:

1. Scrape Content: Stealing content to be reposted elsewhere.
2. Brute Force Attacks: Attempting to guess login credentials.
3. Exploit Vulnerabilities: Searching websites for security weaknesses to exploit.

Because of the range of purposes and behaviors, many websites use a file called robots.txt to give hints or directives to bots about what parts of their website should or shouldn't be accessed. Respectful bots adhere to these rules, but not all bots do.

Being able to recognize the user agent of a bot, especially a malicious one, can be helpful for web administrators in managing access and optimizing their site's relationship with search engines and other online services.

Why Exclude Their Traffic?

Excluding bot traffic from your logs can be beneficial for several reasons:

1. Relevance: By removing bot traffic, your logs will better represent actual human users. This makes the analysis more relevant when making decisions based on user behavior, needs, and problems.

2. Cost Efficiency: Storing, transmitting, and analyzing data costs money. If you're paying for storage or logging services based on the amount of data or number of requests, excluding bots can save costs.

3. Performance Metrics: Bot traffic can skew performance metrics such as page load times, bounce rates, and session lengths. For a more accurate view of site performance and user experience, it's helpful to focus on human users.

4. Resource Optimization: Bots can generate a lot of traffic in a short amount of time, causing unnecessary load on servers. By reducing or managing bot traffic, server resources can be used more efficiently for actual human users.

5. Security: Some bots are malicious, looking for vulnerabilities on websites. By identifying and blocking such traffic, you can enhance your website's security. Moreover, by regularly monitoring and analyzing logs for bot patterns, you can detect and thwart potential threats.

6. Accurate Analytics: Bot traffic can inflate page views, sessions, and other analytics metrics. By filtering out this traffic, marketers and business owners get a clearer view of actual engagement and effectiveness of marketing campaigns.

7. Ad Fraud Detection: If you're running paid ads, bots can generate fake clicks that cost you money. By identifying bot traffic, you can ensure your ad budget is being spent effectively and detect potential fraud.

How to Exclude Bot Traffic:

1. robots.txt: This file allows website administrators to request that certain bots (or all bots) not crawl certain parts of their website. However, not all bots respect robots.txt, especially malicious ones.

2. User Agent Filtering: Many bots identify themselves with a unique user agent string. Server logs or analytics platforms can be configured to exclude or tag traffic from known bot user agents.

3. CAPTCHAs: For parts of a site that are especially sensitive or resource-intensive, consider implementing a CAPTCHA to ensure the user is human.

4. Advanced Detection Tools: There are tools and services designed to detect and block bots in real-time, often using a combination of user agent strings, behavior patterns, and known IP addresses.

5. Log Analysis: Regularly review server logs to identify unexpected or suspicious traffic patterns, which might suggest bot activity.

6. Rate Limiting: Implementing rate limiting on your server can prevent any user (or bot) from making too many requests in a short time frame.

Remember that not all bots are bad. For instance, search engine bots are essential for indexing your site. It's crucial to differentiate between legitimate and unwanted bot traffic and take measures accordingly.

How To Exclude Bots By User Agent in C#

There are various lists online which have bot user agents. One is this: crawler-user-agents.json which contains over 1,000 user agent strings.

By using the JSON model, it is possible to load all of these user agents into a list which can then be persisted to your database. Once in the database, you can make a service which is used by your web server to load and cache all the user agents to block.

When you use service, you can have a base class on your controller, which logs traffic, exclude bots. This way you can always append to the list of bots and the cache will take effect when the application is reloaded.

Model For JSON Load

Model For Database

Repository For Excluded User Agents

Cache Service

Comments