Understand and configure bot rules
Bot Rules let you remove traffic from your report suite that is generated by known spiders and bots. Removing bot traffic can provide a more accurate measurement of user activity on your website.
After bot rules are defined, all incoming traffic is compared against the defined rules. Traffic that matches any of these rules is not collected in the report suite and is not included in traffic metrics.
Removing bot traffic typically reduces the volume of traffic and conversion metrics. Many customers find that removing bot traffic results in increased conversion rates and increases in other usability metrics.
Bot traffic data is stored in a separate repository for display in the Bots and Bot Pages reports.
Update or upload bot rules
The following video demonstrates how to configure bot rules:
To update or upload bot rules:
-
Go to Analytics > Admin > Report Suites.
-
Select the report suite where you want to update bot rules, then select Edit Settings > General > Bot Rules.
-
Use any of the following options to update or upload bot rules for the report suite:
-
Select Enable IAB Bot Filtering Rules to remove bots in IAB’s (International Advertising Bureau’s) International Spiders & Bots List to remove bot traffic.
We recommend that you select this option at a minimum.
For more information, see the section below, Standard IAB bot rules.
-
Select Add Rule to define and add custom bot rules based on user agents, IP addresses or IP ranges.
For more information, see the section below, Custom bot rules.
-
Next to the Select CSV Bot file to import area, select Choose File, then select the CSV file that defines the bot rules.
For more information, see the section below, Upload bot rules.
-
-
Select Save.
Standard IAB bot rules
Standard IAB bot rules can be turned on by checking the Enable IAB Bot Filtering Rules checkbox. This selection will remove bots in IAB’s (International Advertising Bureau’s) International Spiders & Bots List to remove bot traffic. Adobe updates this list from the IAB on a monthly basis.
Adobe is unable to provide the detailed IAB bot list to customers, though you can use the Bots Report to view a list of bots that have accessed your site. To submit a bot to the IAB list, visit IAB.
For information about how to enable standard IAB bot rules in a report suite, see Update or upload bot rules.
Custom bot rules
Custom bot rules let you filter traffic based conditions you define. To begin the process of enabling custom bot rules in a report suite, see Update or upload bot rules.
Custom bot rules are defined using the following condition types:
- User Agent
- IP Address
- IP Range
Multiple conditions can be defined for a single rule. Multiple conditions are matched using “or”. For example, if you provide a value for User Agent and IP Address, the traffic is considered bot traffic if either condition is met.
User Agent
A User Agent condition checks the user agent value to see if it starts with or contains the specified string. If contains is selected, the substring is matched if it occurs anywhere in the user agent.
Optional values can be included in the does not contain list to define values that the user agent must not contain for a successful match. Multiple values can be specified by including one value per line. If the user agent meets the criteria specified in the match string, but also contains a string on the does not contain list, it is not considered a match.
The contains field is limited to 100 characters. The does not contain list is limited to 255 characters minus a separator character for each new line. (This is equal to the number of strings - 1. If you specify 4 does not contain strings, 3 separator characters are required.) All string matches are case-insensitive.
IP Address (including wildcard matches)
Matches an IP address or multiple addresses in the same block using wildcards (*). Provide the numeric values of the IP address you want to match. Substitute * for any values you want to match using a wildcard. The following list contains examples of IP address match string:
10.10.10.1
10.10.10.*
IP Address Range
Provide the start and end ranges of the IP addresses to match. Substitute * for any values you want to match using a wildcard.
Define a custom bot rule
- Go to Analytics > Admin, select one or more report suites and click General > Bot Rules.
- Click Add Rule and define one or more match conditions.
- Click Save. The change should take effect within 30 minutes.
Upload bot rules
To bulk import bot rules, you can upload a CSV file that defines the rules.
-
To begin the process of uploading bot rules to a report suite, see Update or upload bot rules.
-
Create a CSV file with the following columns, on row 1 of the spreadsheet and in the order presented:
table 0-row-6 1-row-6 Column 1, Row 1 Column 2, Row 1 Column 3, Row 1 Column 4, Row 1 Column 5, Row 1 Column 6, Row 1 Bot Name IP Begin IP End Rule
(contains or starts with)User Agent Include User Agent Exclude
(255-character limit)You can define three types of bot rules:
- User agent contains or starts with
- Single IP address or wildcard match
- IP range match
Each row in the import file can contain only one of the following bot definitions:
note note NOTE To match a bot using a combination of rules joined with an OR (for example, user agent or IP address), provide an identical name for all rules that you want to combine in the bot name field. AND matches are not supported. -
User agent contains or starts with: Provide a single user agent string to match in the Agent Include column. Specify the type of match you want performed by placing contains or starts with in the Agent Match Rule field. An optional value can be included in the Agent Exclude column that defines one or more pipe-delimited (
|) strings that the Agent does not contain. String matches are case-insensitive. Both the IP Start and IP End columns must be empty. -
Single IP address or wildcard match: To match a single IP address (
10.10.10.1) or wildcard IP address (10.10.*.*), place the same value in both the IP Start and IP End columns. Match Rule, Agent Include, and Agent Exclude must be empty. -
IP range match: Define a range of IP addresses using the IP Start and IP End columns. Wildcards can be used to match IP ranges, for example
10.10.10.*to10.10.20.*. Match Rule, Agent Include, and Agent Exclude must be empty.
-
On the Bot Rules page in Report Suite Manager, next to the Select CSV Bot file to import area, select Choose File, then select the CSV file that defines the bot rules that you want to import.
-
(Optional) Select the Overwrite existing rules checkbox to delete all existing rules and replace them with the rules defined in the upload file.
-
Select Import File.
-
In the Rule Sets area, review the rules that were imported.
-
Select Save.
Export bot rules
To export all rules defined in the UI in a CSV format:
-
Go to Analytics > Admin > Report Suites.
-
Select the report suite that contains the bot rules that you want to export, then select Edit Settings > General > Bot Rules.
-
Select Export Bot Rules, then save the CSV file to your file system.
Impact of bot rules on data collection section_F01A3130E7A04A9993371CF26F6586F2
Bot Rules are applied to all analytics data. Data removed by Bot Rules is visible only in the Bots and Bot Pages Reports.
VISTA rules are applied after Bot Rules. See Processing order in the Technotes user guide.
High-Hit Visit Processing: If more than 100 hits occur in a visit, reporting determines if the time of the visit in seconds is less than or equal to the number of hits in the visit. In this situation, due to the cost of processing long, intense visits, reporting starts over with a new visit. High-hit visits are typically caused by bot attacks and are not considered normal visitor browsing.
bots are billed as server calls.Impact of IP Obfuscation on bot filtering section_92E60B95BE8940D983F28C79E0CD6B12
The IAB bot list is based solely on user agent, so filtering based on that list is not impacted by IP obfuscation settings. For non-IAB bot filtering (custom rules), IP may be part of the filtering criteria. If filtering bots using IP, bot filtering happens after the last octet has been removed if that setting is enabled, but before the other IP obfuscation options, such as deleting the entire IP or replacing it with some unique ID.
If IP obfuscation is enabled, IP exclusion happens before the IP address is obfuscated, so customers do not need to change anything when they enable IP obfuscation.
If the last octet is removed, that is done before IP filtering. As such, the last octet is replaced with a 0, and IP exclusion rules should be updated to match IP addresses with a zero on the end. Matching * should match 0.