AskF5 Knowledge Base
- Home
- Supported Products
- BIG-IP ASM / VE
- Configuration Guide for BIG-IP Application Security Manager
- Configuring Anomaly Detection
Applies To:
Show Versions
Manual Chapter: Configuring Anomaly Detection
Anomaly detection is a way of detecting patterns in traffic that do not conform with normal behavior, such as an increase in latency or the transactions rate. Application Security ManagerTM provides ways for you to configure the system to detect and mitigate anomalies, including:
| Denial-of-service (DoS) attacks: Detects the spikes and anomalies in Layer 7 (application layer) traffic. |
| Brute force attacks: Protects against hackers forceful attempts to gain access to a web site. |
| Increased violations from certain IP addresses: Prevents attacks originating from specific IP addresses. |
| Bot detection and web scraping: Prevents automated extraction of data from web sites. |
A denial-of-service attack (DoS attack) is an attempt to make a computer resource unavailable to its intended users. The DoS attack generally consists of the concerted, malevolent efforts to prevent an Internet site or service from functioning efficiently or at all, temporarily or indefinitely. Perpetrators of DoS attacks typically target sites or services, such as banks, credit card payment gateways, and e-commerce web sites.
One common method of attack involves saturating the target (victim) machine with external communications requests, so that the target system cannot respond to legitimate traffic, or responds so slowly as to be rendered effectively unavailable. In general terms, DoS attacks are implemented by forcing the targeted computer to reset, or by consuming its resources so that it can no longer provide its intended service, or by obstructing the communication media between the intended users and the victim so that they can no longer communicate adequately.
Denial-of-service attacks are also known as HTTP-GET attacks or page flood attacks. The attacks can either be initiated from a single user (single IP address) or from thousands of computers (distributed DoS attack), which overwhelms the target system. A page flood attack (or HTTP flood attack) may resemble the patterns of normal Web surfing, making it harder for automated tools to differentiate between legitimate Web traffic and an attempted attack.
Application Security Manager considers traffic to be a DoS attack based on calculations for transaction rate (TPS-based) or latency (latency-based). You can configure which calculations you want the system to use.
If you choose TPS-based, the system detects DoS attacks based on the following calculations:
 | Transaction rate during detection interval The average number of requests per second sent for a specific URL, or by a specific IP address. The system calculates this number every minute. |
| | Transaction rate during history interval The average number of requests per second sent for a specific URL, or by a specific IP address. The system calculates this number every hour. |
If the ratio of the transaction rate during the detection interval to the transaction rate during the history interval is greater than the specific percentage you configure on the DoS Attack Prevention screen (the TPS increased by percentage), the system considers the URL to be under attack, or the IP address to be suspicious. To prevent further attacks, the system drops requests for this URL, and drops requests from the suspicious IP address.
If you choose latency-based, DoS attacks are detected based on the following calculations:
| | Latency during detection interval The average time it takes for the system to respond to a request for a specific URL, for each web application, over the last minute. This average is updated every second. |
| | Latency during history interval The average time it takes for the system to respond to a request for a specific URL, for each web application, over the last hour. This average is updated every minute. |
If the ratio of the latency during the detection interval to the latency during the history interval is greater than the percentage you configure on the DoS Attack Prevention screen (the Latency increased by percentage), the system detects that this URL is under attack.
You can configure the Application Security Manager to monitor latency of Layer 7 (application layer) traffic and transactions per second to detect the spikes and anomalies in the typical traffic pattern. The spikes and anomalies may indicate an attempted attack. The system rejects suspicious traffic based on absolute IP addresses and your configuration.
| 1. | In the navigation pane, expand Application Security and click Anomaly Detection. The DoS Attack Prevention screen opens. |
| 2. | In the editing context area, verify that the edited security policy is the one you want to update. |
| 3. | For Operation Mode, select how you want the system to react to DoS attacks: |
| Off Does not check for DoS attacks. This is the default setting. |
| Transparent Displays information about DoS attacks on the DoS Attacks reporting screen. |
| Blocking Displays information about DoS attacks on the DoS Attacks reporting screen, and drops connections coming from an attacking IP address or going to a URL being attacked. |
| 4. | For the Detection Mode, select the way you want the system to look for DoS attacks: |
| TPS-based Determines DoS attacks from the client side based on the number of requests per second sent to a specific URL, or the number of transactions per second coming from a specific IP address. This is the default setting. |
| Latency-based Determines DoS attacks from the server side based on the average time it takes for the system to respond to a request for a specific URL. |
| 5. |
| Latency increased by: Specifies that the system considers traffic to be an attack if the latency has increased by this percentage. The default value is 500%. |
| Latency reached: Specifies that the system considers traffic to be an attack if the latency is equal to or greater than this value. This setting provides an absolute value, so, for example, if an attack increases latency gradually, the increase might not exceed the Latency Increased by threshold and would not be detected. If server latency reaches the Latency reached value, the system considers traffic to be an attack even if it did not meet the Latency increased by criterion. The default value is 10000 ms. |
| Minimum Latency Threshold for detection: Specifies that the system considers traffic to be an attack if the detection interval for a specific URL equals, or is greater than, this number, and at least one of the Latency increased by number was reached. If the detection interval is lower than this number, the system does not consider this traffic to be an attack even if the Latency increased by number was reached. The default setting is 200 ms. |
| 6. | For the Prevention Policy setting, select one or more options to determine how you want the system to handle a DoS attack: |
| Source IP-Based Client-Side Integrity Defense Checks whether a client is a legal browser or an illegal script by injecting JavaScript into responses when suspicious IP addresses are requested. Legal browsers can process JavaScript and respond properly, whereas illegal scripts cannot. The default is disabled. |
| URL-Based Client-Side Integrity Defense Checks whether a client is a legal browser or an illegal script by injecting JavaScript into responses when suspicious URLs are requested. Legal browsers can process JavaScript and respond properly, whereas illegal scripts cannot. This setting enforces strong protection and prevents distributed DoS attacks but affects more clients. The default is disabled. |
| Source IP-Based Rate Limiting Check to drop requests from suspicious IP addresses. Application Security Manager drops connections to limit the rate of requests to the average rate prior to the attack, or lower than the absolute threshold specified by the IP detection TPS reached setting. The default is enabled. |
| URL-Based Rate Limiting Check to indicate that when the system detects a URL under attack, Application Security Manager drops connections to limit the rate of requests to the URL to the average rate prior to the attack. |
| 7. | For IP Detection Criteria, type the threshold values: |
Note: This setting appears only if Prevention Policy is set to Source IP-Based Client Side Integrity Defense and/or Source IP-Based Rate Limiting.
| TPS increased by: Specifies that the system considers an IP address to be that of an attacker, if the transactions (requests) sent per second have increased by this percentage. The default value is 500%. |
| TPS reached: Specifies that the system considers an IP address to be suspicious if the number of transactions (requests) sent per second from an IP address is equal to or greater than this value. This setting provides an absolute value, so, for example, if an attack increases the number of transactions gradually, the increase might not exceed the TPS increased by threshold and would not be detected. If the TPS reaches the TPS reached value, the system considers traffic to be an attack even if it did not meet the TPS increased by criterion. The default value is 200 TPS. |
If either of these criteria is met, the system handles the attack according to the Prevention Policy settings.
| 8. | For URL Detection Criteria, type the threshold values: |
Note: This setting appears only if Prevention Policy is set to URL-Based Client Side Integrity Defense and/or URL-Based Rate Limiting.
| TPS increased by: Specifies that the system considers a URL to be an attack if the number of transactions (requests) sent per second to the URL have increased by this percentage. The default value is 500%. |
| TPS reached: Specifies that the system considers a URL to be suspicious if the number of transactions (requests) sent per second to the URL is equal to or greater than this value. This setting provides an absolute value, so, for example, if an attack increases the number of transactions gradually, the increase might not exceed the TPS Increased by threshold and would not be detected. If the TPS reaches the TPS reached value, the system considers traffic to be an attack even if it did not meet the TPS increased by criterion. The default value is 1000 TPS. |
If either of these criteria is met, the system handles the attack according to the Prevention Policy settings.
| 9. | For the Prevention Duration setting, specify the length of time for which the system mitigates DoS attacks: |
| Unlimited: Select if you want the system to perform attack prevention until it detects the end of the attack. |
| Maximum: Select and type a value, in seconds. The system prevents detected DoS attacks for the time configured here (even if the attack is still occurring), or until the system detects the end of the attack, whichever is sooner. |
| 10. | In IP Address Whitelist, type the IP addresses and subnets that do not need to be checked for DoS attacks, and click Add. |
| 11. | Click Save to save the detection and prevention criteria. |
| 12. | To put the security policy changes into effect immediately, click the Apply Policy button in the editing context area. |
You can view details about DoS attacks that the system detected and logged. For information about the DoS Attacks reports, refer to Viewing DoS Attacks reports. You can also configure remote logging support for DoS attacks when creating a logging profile. For information about creating remote logging profiles, refer to Configuring a logging profile for remote storage.
You can configure the Application Security Manager to detect, report on, and prevent Layer 7 brute force attack attempts. Brute force attacks are attempts to break in to secured areas of a web application by trying exhaustive, systematic user name/password combinations to discover legitimate authentication credentials. Malicious users send high volumes of these combinations, often scripted, to avoid security mechanisms. In this automated scenario, one malicious user can make thousands of login attempts per minute.
In Application Security Manager, you can configure the login URL of a web application to protect, the mitigation methods, and the access validation criteria for logon responses. With brute force protection, the system can monitor traffic to detect excessive failures to authenticate, monitor suspicious IP addresses (generating a high volume of login attempts), and detect other anomalies in the typical traffic pattern for the login URL.
Application Security Manager tracks the number of failed login attempts for each web application URL in two intervals:
| History interval: Failed login attempts for the past hour (updated every minute). |
| Detection interval: Failed login attempts for the past minute (updated every second). |
The system considers it to be a brute force attack if the failed login rate during the detection interval exceeds the failed login rate during the history interval.
| 1. | In the navigation pane, expand Application Security, point to Anomaly Detection, and click Brute Force Attack Prevention. The Brute Force Attack Prevention screen opens. |
| 2. | In the editing context area, verify that the current edited security policy is the one you want to update. |
| 3. | Click the Create button. The New Brute Force Protection Configuration screen opens. Next, you configure the login information. |
Continue configuring the settings in the Brute Force Protection Configuration area of the New Brute Force Protection Configuration screen.
| 1. | For Login URL: |
| Select Explicit (to specify one URL) or Wildcard (to specify a pattern that will match multiple URLs). |
| Type either a specific URL or a wildcard expression (for example, Login*.php). |
The system automatically adds any login URL (explicit or wildcard) that you specify on this screen to the security policys list of allowed URLs.
| 2. | For Authentication Type, select the method by which the application server validates user logon credentials: |
| HTML Form Specifies that the code of the login URL is in HTML format. This is the default setting. |
| HTTP Basic Authentication Specifies that the web server uses HTTP basic authentication to authenticate users. |
| HTTP Digest Authentication Specifies that the web server uses HTTP digest authentication to authenticate users. |
| 3. | For Username Parameter Name, type the user name parameter included in the code of the HTML form. When the system detects this parameter with the password parameter, the system recognizes the request as a login attempt. (Applies only to the HTML Form authentication type.) |
| 4. | For the Password Parameter Name setting, type the password parameter written in the code of the HTML form. When the system detects this parameter with the username parameter, the system recognizes that request as a login attempt. (Applies only to the HTML Form authentication type.) Next, you can configure session-based mitigation. |
Session-based mitigation counts the number of failed login attempts that occur during one session for an IP address. When the system detects a brute force attack, it triggers the Maximum login attempts are exceeded violation, and applies the blocking policy.
To configure session-based mitigation, continue configuring the settings in the Session-based Brute Force Protection area of the New Brute Force Protection Configuration screen.
| 1. | Above the Session-based Brute Force Protection area, click the Blocking Settings link to open the Blocking Policy screen where you can configure the blocking actions that the system takes when the Maximum login attempts are exceeded violation occurs. |
| 2. | For Login Attempts From The Same Client, type the number of times a user can attempt to log on before the system blocks the request. The default value is 5. |
| 3. | For Re-enable Login After, type the number of seconds the user must wait to log on after they have been blocked. Next, you can optionally configure dynamic brute force protection. |
Note that it is not required to configure both dynamic brute force protection and session-based brute force protection.
Dynamic mitigation detects and mitigates brute force attacks based on statistical analysis of the traffic. You configure dynamic mitigation to determine when the system should consider the login URL to be under attack, and how to react to an attack. The system mitigates attacks when the volume of unsuccessful login attempts is significantly greater than the typical number of failed logins.
To configure dynamic brute force protection, use the settings in the Dynamic Brute Force Protection area of the New Brute Force Protection Configuration screen.
| 1. | For Operation Mode, select how the system handles brute force attacks: |
| Off Does not monitor traffic to detect brute force attacks. |
| Transparent Issues reporting data only on attacks. Do not drop illegal requests. This is the default setting. |
| Blocking Drops illegal requests and log reporting data. |
| 2. | For the Detection Criteria setting, specify when to consider login attempts to be an attack. |
| Failed Logins Attempts increased by The system considers logon attempts to be an attack if, for all IP addresses tracked, the ratio between the detection interval and the history interval is greater than this number. The default setting is 500 percent. |
| Failed Login Attempts Rate reached The system considers logon attempts to be an attack if, for all IP addresses tracked, the logon rate reaches this number. The default setting is 1000 logon attempts per second. |
| Minimum Failed Login Attempts The system considers logon attempts to be an attack if, for all IP addresses tracked, the number of logon attempts is equal to, or greater than, this number. This setting prevents false positive attack detection. The default setting is 100 logon attempts per second. |
| 3. | For the Prevention Policy setting, select the methods you want the system to use to mitigate an attack (the methods are applied in the order listed). |
| Source IP-Based Client-Side Integrity Defense Check to determine whether the client is a legal browser or an illegal script by injecting JavaScript into responses when suspicious IP addresses are requested. Legal browsers can process JavaScript and respond properly, whereas illegal scripts cannot. The default is disabled. |
| URL-Based Client-Side Integrity Defense Check to determine whether the client is a legal browser or an illegal script by injecting JavaScript into responses when suspicious URLs are requested. Legal browsers can process JavaScript and respond properly, whereas illegal scripts cannot. The default is disabled. |
| Source IP-Based Rate Limiting Check to drop requests from suspicious IP addresses. Application Security Manager drops connections to limit the rate of login attempts to the average rate prior to the attack. The default is enabled. |
| URL-Based Rate Limiting Check to indicate that when the system detects a URL under attack, Application Security Manager performs rate limiting and limits the rate of all logon requests to the normal level. The default is enabled. |
| 4. | For Suspicious Criteria (per IP address), specify how to identify traffic that may be an attack. If at least one of the criteria is met, the system treats the IP address as an attacker, and prevents the attacker from trying to guess the password. The system also limits the number of login attempts to the normal level. |
| Failed Login Attempts increased by Type a number. Login attempts from an individual IP address are considered an attack if the number of failed login attempts has increased by this percentage over the normal number of failed logins. The default setting is 500 percent. |
| Failed Login Attempts Rate reached Type a number. An individual IP address is suspicious if the number of login attempts per second from an IP address is equal to or greater than this number. The default setting is 1000 login attempts per second. |
| 5. | For the Prevention Duration setting, specify the length of time for which the system mitigates brute force attacks. |
| Unlimited Specifies that after the system detects and mitigates a brute force attack, it performs attack prevention until it detects the end of the attack. |
| Maximum Specifies that after the system detects and mitigates a brute force attack, it performs attack prevention either for the time configured here (even if the system detects that the attack continues), or until the system detects the end of the attack, whichever is earlier. Type a value, in seconds, in the box. |
| 6. | In IP Address Whitelist, add the IP addresses or subnets that do not need to be checked for brute force attacks. Next, you can define validation criteria to apply to the response of the login URL. |
Continue configuring the settings in the Access Validation area of the New Brute Force Protection Configuration screen.
For Access Validation, define at least one validation criterion for responses of the login URL:
| A string that should appear in the response Type a string that must appear in the response for the system to detect a successful login attempt; for example, Successful Login. |
| A string that should NOT appear in the response Type a string that indicates a failed login attempt; for example, Authentication failed. |
| Expected HTTP response status code Type an HTTP response code that is sent when the user successfully logs in; for example, 200. |
| Expected validation header name and value (for example, Location header) Type a header name and value that is sent when the user successfully logs in. |
| Expected validation domain cookie name Type a defined domain cookie name that is sent when the user successfully logs in. |
| Expected parameter name (added to URI links in the response) Type a parameter that is sent when the user successfully logs in. Next, you can finish brute force protection configuration. |
| 1. | At the bottom of the New Brute Force Protection Configuration screen, click Create. The screen refreshes, and you see your protected login URL in the list. |
| 2. | To put the security policy changes into effect immediately, click the Apply Policy button in the editing context area. |
For how you can view details about brute force attacks that the system detected and logged, refer to the section, Viewing Brute Force Attack reports.
You can configure the IP Enforcer to perform enforcement based on IP address. When the IP Enforcer is in blocking mode, Application Security Manager drops requests for any IP address if more than the maximum number of blocked violations occur in one minute. The system detects an attacker and drops all requests from that IP address for a specified time interval. The IP Enforcer can also operate in transparent mode where it reports IP addresses that exceed the blocked violation threshold but it does not drop the requests. By default, IP address enforcement is off.
Note: IP address enforcement stops traffic only if the security policys enforcement mode is Blocking, and some violations must have the Block flag enabled (on the Blocking Policy screen).
When the IP Enforcer is configured, you can view IP Enforcer statistics to investigate and release the blocked IP addresses, view dropped requests from that IP address, and examine violations that occurred for each IP address. For details, see Viewing IP Enforcer statistics.
| 1. | In the navigation pane, expand Application Security, point to Anomaly Detection and click IP Enforcer. The IP Enforcer Configuration screen opens. |
| 2. | In the editing context area, verify that the edited security policy is the one you want to update. |
| 3. | In the IP Enforcer Configuration area, for the Operation Mode setting, select the mode for IP enforcement: |
| Off Does not perform IP-based enforcement. This is the default setting. |
| Transparent Issues reporting data. In this mode, if the violation threshold is exceeded, the system logs the attack data, but does not prevent access to the application by the offending client. |
| Blocking Prohibits access by this IP address if more than the maximum number of blocked violations occur in one minute. |
| 4. | For the Violation Threshold setting, type the maximum number of blocked violations that you want to allow per minute from each IP address. The default value is 10 blocked violations per minute. |
| 5. | For the Prevention Duration setting, type the number of seconds you want to block (or report) requests. The default value is 60. |
| 6. | Click Save to save the IP enforcement configuration. |
| 7. | To put the security policy changes into effect immediately, click the Apply Policy button in the editing context area. |
You can configure Application Security Manager to detect and prevent various bot activities and web scraping on web sites that it is protecting. Web scraping is a technique for extracting information from web sites typically using automated programs, or bots (short for web robots).
The Application Security Manager attempts to determine whether a web client source is human. This is how it works:
| | Dropped request If the system cannot check requests for human activity, the request is dropped with no further checking. This action occurs only if the Block flag is set for the Web scraping detected violation. The system does not drop requests if the security policy is running in transparent mode, or if only the Learn or Alarm flags are set for the violation. |
| | Grace interval The grace interval is how many requests the system reviews while trying to detect whether the client is human. During the grace interval, requests are not blocked or reported. What occurs next depends on whether the system detects human activity: |
| If the system detects human activity The grace interval ends and the system handles the number of requests specified in the Safe Interval, then restarts the grace interval and starts checking again. |
| If the system does not detect human activity The system issues the Web Scraping Detected violation until it reaches the number of requests in the Unsafe Interval. If the system is configured to block traffic if that violation occurs, the system blocks requests during this time. In transparent mode or if the violation is set to Alarm only, the violation is logged and requests are permitted. After reaching the Unsafe Interval, the system restarts the grace interval and starts checking again. |
The system can accurately detect human users only when all these conditions exist:
| The Block setting for the Web Scraping Detected violation is enabled on the Blocking Policy screen. |
If your environment uses legitimate automated tests, you can create a white list of IP addresses or address ranges from which to allow access. The system does not perform web scraping detection on these addresses. In addition, the system allows requests from well-known search engines and legal web robots including the following:
| Google (googlebot.com) |
| Yahoo (crawl.yahoo.net) |
| MSN (search.msn.com) |
| ask.com (ask.com) |
| AOL (using googlebot.com) |
| 1. | In the navigation pane, expand Application Security, point to Anomaly Detection, then click Web Scraping. The Web Scraping screen opens. |
| 2. | In the editing context area, verify that the edited security policy is the one you want to update. |
| 3. | For Grace Interval, type the number of requests to allow while determining whether the client is human. The default value is 100. |
| 4. | For Unsafe Interval, type the number of requests that cause the Web Scraping Detected violation if no human activity is detected during the grace period. When the number is reached, reactivate the grace period. The default value is 100. |
| 5. | For Safe Interval, type the number of requests to allow after human activity is detected and before reactivating the grace threshold to check again for non-human clients. The default value is 2000. |
| 6. | In IP Address Whitelist, add the IP addresses or subnets that do not need to be checked for web scraping. |
| 7. | Click Save. |
You can view details about web scraping attacks that the system detected and logged, as described in Viewing web scraping statistics.
