Applies To:

Show Versions Show Versions

Manual Chapter: Implementing Anomaly Detection
Manual Chapter
Table of Contents   |   << Previous Chapter   |   Next Chapter >>

Anomaly detection is a way of detecting patterns in traffic that do not conform with normal behavior, such as an increase in latency or the transactions rate. Application Security Manager provides ways for you to configure the system to detect and mitigate anomalies, including:
Denial-of-service (DoS) attacks: Detects the spikes and anomalies in Layer 7 (application layer) traffic.
Brute force attacks: Protects against hackers forceful attempts to gain access to a web site.
Increased violations from certain IP addresses: Prevents attacks originating from specific IP addresses.
Bot detection and web scraping: Prevents automated extraction of data from web sites.
The remote logging formats for anomalies are predefined, and each remote logging type has a specific format in which it stores information about the anomaly. For details about the predefined remote logging formats for anomalies, refer to Appendix G, Remote Logging Formats for Anomalies.
A denial-of-service attack (DoS attack) is an attempt to make a computer resource unavailable to its intended users. The DoS attack generally consists of the concerted, malevolent efforts to prevent an Internet site or service from functioning efficiently or at all, temporarily or indefinitely. Perpetrators of DoS attacks typically target sites or services, such as banks, credit card payment gateways, and e-commerce web sites.
One common method of attack involves saturating the target (victim) machine with external communications requests, so that the target system cannot respond to legitimate traffic, or responds so slowly as to be rendered effectively unavailable. In general terms, DoS attacks are implemented by forcing the targeted computer to reset, or by consuming its resources so that it can no longer provide its intended service, or by obstructing the communication media between the intended users and the victim so that they can no longer communicate adequately.
Denial-of-service attacks are also known as HTTP-GET attacks or page flood attacks. The attacks can either be initiated from a single user (single IP address) or from thousands of computers (distributed DoS attack), which overwhelms the target system. A page flood attack (or HTTP flood attack) may resemble the patterns of normal Web surfing, making it harder for automated tools to differentiate between legitimate Web traffic and an attempted attack.
Application Security Manager considers traffic to be a DoS attack based on calculations for transaction rates on the client side (TPS-based) or latency on the server side (latency-based). You can specify the calculations that you want the system to use.
You can view details about DoS attacks that the system detected and logged. For information about the DoS Attacks reports, refer to Viewing DoS Attacks reports. You can also configure remote logging support for DoS attacks when creating a logging profile. For information about creating remote logging profiles, refer to Creating logging profiles.
You can configure Application Security Manager to mitigate DoS attacks based on transaction rates using TPS-based DoS protection. If you choose TPS-based DoS protection, the system detects DoS attacks from the client side using the following calculations:
Transaction rate during detection interval
The average number of requests per second sent for a specific URL, or by a specific IP address. The system calculates this number every minute.
Transaction rate during history interval
The average number of requests per second sent for a specific URL, or by a specific IP address. The system calculates this number every hour.
If the ratio of the transaction rate during the detection interval to the transaction rate during the history interval is greater than the specific percentage you configure on the DoS Attack Prevention screen (the TPS increased by percentage), the system considers the URL to be under attack, or the IP address to be suspicious. To prevent further attacks, the system drops requests for this URL, and drops requests from the suspicious IP address.
1.
On the Main tab, expand Application Security and click Anomaly Detection.
The DoS Attack Prevention screen opens.
2.
In the editing context area, ensure that the Current edited policy is the one that you want to update.
3.
For Operation Mode, select the way you want the system to react to DoS attacks:
Transparent
Displays information about DoS attacks on the DoS Attacks reporting screen.
Blocking
Displays information about DoS attacks on the DoS Attacks reporting screen, and drops connections coming from an attacking IP address or going to a URL being attacked.
4.
For the Detection Mode, select TPS-based if not already set.
5.
For the Prevention Policy setting, select one or more options to determine how you want the system to handle a DoS attack.
Note: If you enable more than one option, the system uses the options in the order in which they are listed.
Source IP-Based Client-Side Integrity Defense
Determines whether a client is a legal browser or an illegal script by injecting JavaScript into responses when suspicious IP addresses are requested. Legal browsers can process JavaScript and respond properly, whereas illegal scripts cannot. The default is disabled.
URL-Based Client-Side Integrity Defense
Determines whether a client is a legal browser or an illegal script by injecting JavaScript into responses when suspicious URLs are requested. Legal browsers can process JavaScript and respond properly, whereas illegal scripts cannot. This setting enforces strong protection and prevents distributed DoS attacks but affects more clients. The default is disabled.
Source IP-Based Rate Limiting
Drops requests from suspicious IP addresses. The system limits the rate of requests to the average rate prior to the attack, or lower than the absolute threshold specified by the IP detection TPS reached setting. The default is enabled.
URL-Based Rate Limiting
Indicates that when the system detects a URL under attack, Application Security Manager drops connections to limit the rate of requests to the URL to the average rate prior to the attack.
6.
For IP Detection Criteria, type the threshold values.
Note: This setting appears only if Prevention Policy is set to Source IP-Based Client Side Integrity Defense and/or Source IP-Based Rate Limiting.
TPS increased by: Specifies that the system considers an IP address to be that of an attacker, if the transactions sent per second have increased by this percentage. The default value is 500%.
TPS reached: Specifies that the system considers an IP address to be suspicious if the number of transactions sent per second from an IP address equals, or is greater than, this value. This setting provides an absolute value, so, for example, if an attack increases the number of transactions gradually, the increase might not exceed the TPS increased by threshold and would not be detected. If the TPS reaches the TPS reached value, the system considers traffic to be an attack even if it did not meet the TPS increased by criterion. The default value is 200 TPS.
Minimum TPS Threshold for detection: Specifies that the system considers an IP address to be an attacker if the detected TPS for a specific IP address equals, or is greater than, this number, and the TPS increased by number was reached. The default setting is 40 transactions per second.
If any of these criteria is met, the system handles the attack according to the Prevention Policy settings.
7.
For URL Detection Criteria, type the threshold values.
Note: This setting appears only if Prevention Policy is set to URL-Based Client Side Integrity Defense and/or URL-Based Rate Limiting.
TPS increased by: Specifies that the system considers a URL to be an attacker if the number of transactions (requests) sent per second to the URL have increased by this percentage. The default value is 500%.
TPS reached: Specifies that the system considers a URL to be suspicious if the number of transactions (requests) sent per second to the URL is equal to or greater than this value. This setting provides an absolute value, so, for example, if an attack increases the number of transactions gradually, the increase might not exceed the TPS Increased by threshold and would not be detected. If the TPS reaches the TPS reached value, the system considers traffic to be an attack even if it did not meet the TPS increased by criterion. The default value is 1000 TPS.
Minimum TPS Threshold for detection: Specifies that the system considers a URL to be an attacker if the detected TPS for a specific URL equals, or is greater than, this number, and the TPS increased by number was reached. The default setting is 200 transactions per second.
If any of these criteria is met, the system handles the attack according to the Prevention Policy settings.
8.
For the Prevention Duration setting, specify the length of time for which the system mitigates DoS attacks:
Unlimited: Performs attack prevention until the system detects the end of the attack.
Maximum: Select and type a value, in seconds.
Prevents detected DoS attacks for the time configured here (even if the attack is still occurring), or until the system detects the end of the attack, whichever is sooner.
9.
In IP Address Whitelist, type the IP addresses and subnets that do not need to be examined for DoS attacks, and click Add.
You can add up to 20 IP addresses. The IP addresses in the whitelist are also added to the centralized IP address exceptions list with the Ignore in Anomaly Detection setting enabled.
10.
Click Save to save the detection and prevention criteria.
11.
To put the security policy changes into effect immediately, click the Apply Policy button in the editing context area.
You can configure Application Security Manager to mitigate DoS attacks based on server latency using latency-based DoS protection. If you choose latency-based DoS detection, the system detects DoS attacks on the server side using the following calculations:
Latency during detection interval
The average time it takes for the system to respond to a request for a specific URL, for each web application, over the last minute. This average is updated every second.
Latency during history interval
The average time it takes for the system to respond to a request for a specific URL, for each web application, over the last hour. This average is updated every minute.
If the ratio of the latency during the detection interval to the latency during the history interval is greater than the percentage you configure on the DoS Attack Prevention screen (the Latency increased by percentage), the system detects that this URL is under attack.
1.
On the Main tab, expand Application Security and click Anomaly Detection.
The DoS Attack Prevention screen opens.
2.
In the editing context area, verify that the Current edited policy is the one you want to update.
3.
For Operation Mode, select how you want the system to react to DoS attacks:
Transparent
Displays information about DoS attacks on the DoS Attacks reporting screen.
Blocking
Displays information about DoS attacks on the DoS Attacks reporting screen, and drops connections coming from an attacking IP address or going to a URL being attacked.
4.
For the Detection Mode, select Latency-based.
5.
For Detection Criteria, specify the threshold values:
Latency increased by: Specifies that the system considers traffic to be an attack if the minimum latency threshold was reached and the latency has increased by this percentage. The default value is 500%.
Latency reached: Specifies that the system considers traffic to be an attack if the latency is equal to or greater than this value. This setting provides an absolute value, so, for example, if an attack increases latency gradually, the increase might not exceed the Latency Increased by threshold and would not be detected. If server latency reaches the Latency reached value, the system considers traffic to be an attack even if it did not meet the Latency increased by criterion. The default value is 10000 ms.
Minimum Latency Threshold for detection: Specifies that the system considers traffic to be an attack if the detection interval for a specific URL equals, or is greater than, this number, and at least one of the Latency increased by number was reached. The default setting is 200 ms.
6.
For the Prevention Policy setting, select one or more options to determine how you want the system to handle a DoS attack:
Note: If you enable more than one option, the system enforces the options in the order in which they are listed.
Source IP-Based Client-Side Integrity Defense
Determines whether a client is a legal browser or an illegal script by injecting JavaScript into responses when suspicious IP addresses are requested. Legal browsers can process JavaScript and respond properly, whereas illegal scripts cannot. The default is disabled.
URL-Based Client-Side Integrity Defense
Determines whether a client is a legal browser or an illegal script by injecting JavaScript into responses when suspicious URLs are requested. Legal browsers can process JavaScript and respond properly, whereas illegal scripts cannot. This setting enforces strong protection and prevents distributed DoS attacks but affects more clients. The default is disabled.
Source IP-Based Rate Limiting
Drops requests from suspicious IP addresses. The system limits the rate of requests to the average rate prior to the attack, or lower than the absolute threshold specified by the IP detection TPS reached setting. The default is enabled.
URL-Based Rate Limiting
Indicates that when the system detects a URL under attack, Application Security Manager drops connections to limit the rate of requests to the URL to the average rate prior to the attack.
7.
For Suspicious IP Criteria, type the threshold values:
Note: This setting appears only if Prevention Policy is set to Source IP-Based Client Side Integrity Defense and/or Source IP-Based Rate Limiting.
TPS increased by: Specifies that the system considers an IP address to be that of an attacker, if the transactions (requests) sent per second have increased by this percentage. The default value is 500%.
TPS reached: Specifies that the system considers an IP address to be suspicious if the number of transactions (requests) sent per second from an IP address is equal to or greater than this value. This setting provides an absolute value, so, for example, if an attack increases the number of transactions gradually, the increase might not exceed the TPS increased by threshold and would not be detected. If the TPS reaches the TPS reached value, the system considers traffic to be an attack even if it did not meet the TPS increased by criterion. The default value is 200 requests per second.
Minimum TPS Threshold for detection: Specifies that the system considers an IP address to be suspicious if the detected TPS for a specific IP address equals, or is greater than, this number, and the TPS increased by number was reached. The default setting is 40 transactions per second.
If any of these criteria is met, the system handles the attack according to the Prevention Policy settings.
8.
For Suspicious URL Criteria, type the threshold values:
Note: This setting appears only if Prevention Policy is set to URL-Based Client Side Integrity Defense and/or URL-Based Rate Limiting.
TPS increased by: Specifies that the system considers a URL to be an attack if the number of transactions (requests) sent per second to the URL have increased by this percentage. The default value is 500%.
TPS reached: Specifies that the system considers a URL to be suspicious if the number of transactions (requests) sent per second to the URL is equal to or greater than this value. This setting provides an absolute value, so, for example, if an attack increases the number of transactions gradually, the increase might not exceed the TPS Increased by threshold and would not be detected. If the TPS reaches the TPS reached value, the system considers traffic to be an attack even if it did not meet the TPS increased by criterion. The default value is 1000 TPS.
Minimum TPS Threshold for detection: Specifies that the system considers a URL to be suspicious if the detected TPS for a specific URL equals, or is greater than, this number, and the TPS increased by number was reached. The default setting is 200 transactions per second.
If any of these criteria is met, the system handles the attack according to the Prevention Policy settings.
9.
For the Prevention Duration setting, specify the length of time for which the system mitigates DoS attacks:
Unlimited: Select if you want the system to perform attack prevention until it detects the end of the attack.
Maximum: Select and type a value, in seconds. The system prevents detected DoS attacks for the time configured here (even if the attack is still occurring), or until the system detects the end of the attack, whichever is sooner.
10.
In IP Address Whitelist, type the IP addresses and subnets that do not need to be examined for DoS attacks, and click Add.
You can add up to 20 IP addresses. The IP addresses in the whitelist are also added to the centralized IP address exceptions list. Exceptions from the central list having the Ignore in Anomaly Detection setting enabled, also appear on the list here.
11.
Click Save to save the detection and prevention criteria.
12.
To put the security policy changes into effect immediately, click the Apply Policy button in the editing context area.
You can configure the Application Security Manager to detect, report on, and prevent Layer 7 brute force attack attempts. Brute force attacks are attempts to break in to secured areas of a web application by trying exhaustive, systematic user name/password combinations to discover legitimate authentication credentials. Malicious users send high volumes of these combinations, often scripted, to avoid security mechanisms. In this automated scenario, one malicious user can make thousands of login attempts per minute.
In Application Security Manager, you can configure the login URL of a web application to indicate the mitigation methods and the access validation criteria for logon responses. With brute force protection, the system can monitor traffic to detect excessive failures to authenticate, monitor suspicious IP addresses (generating a high volume of login attempts), and detect other anomalies in the typical traffic pattern for the login URL.
History interval: Failed login attempts for the past hour (updated every minute).
Detection interval: Failed login attempts for the past minute (updated every second).
The system considers it to be a brute force attack if the failed login rate during the detection interval exceeds the failed login rate during the history interval.
1.
On the Main tab, expand Application Security, point to Anomaly Detection, and click Brute Force Attack Prevention.
The Brute Force Attack Prevention screen opens.
2.
In the editing context area, verify that the Current edited policy is the one you want to update.
3.
Click the Create button.
The New Brute Force Protection Configuration screen opens. Next, you configure the login information.
4.
For Login Page, select a predefined security policy login page (with authentication) from the list. This is the URL (typically the login page) that you want to protect against brute force attacks. If you need to create a login page, see Creating login pages.
Session-based mitigation counts the number of failed login attempts that occur during one session for an IP address. When the system detects a brute force attack, it triggers the Brute Force: Maximum login attempts are exceeded violation, and applies the blocking policy.
To configure session-based mitigation, continue configuring the settings in the Session-based Brute Force Protection area of the New Brute Force Protection Configuration screen.
1.
Optionally, above the Session-based Brute Force Protection area, click the Blocking Settings link to open the Policy Blocking Settings screen. Configure the blocking actions that the system takes when the Brute Force: Maximum login attempts are exceeded violation occurs.
Note: See Configuring policy blocking, for more information.
2.
For Login Attempts From The Same Client, type the number of times a user can attempt to log in before the system blocks the request. The default value is 5.
3.
For Re-enable Login After, type the number of seconds the user must wait to log in after they have been blocked. The default value is 600 seconds.
Dynamic mitigation detects and mitigates brute force attacks based on statistical analysis of the traffic. You configure dynamic mitigation to determine when the system should consider the login URL to be under attack, and how to react to an attack. The system mitigates attacks when the volume of unsuccessful login attempts is significantly greater than the typical number of failed logins.
To configure dynamic brute force protection, use the settings in the Dynamic Brute Force Protection area of the New Brute Force Protection Configuration screen.
1.
For Operation Mode, select how the system handles brute force attacks:
Off
Does not monitor traffic to detect brute force attacks.
Transparent
Issues reporting data only on attacks. Do not drop illegal requests. This is the default setting.
Blocking
Drops illegal requests and log reporting data.
2.
For the Detection Criteria setting, specify when to consider login attempts to be an attack.
Failed Logins Attempts increased by
The system considers logon attempts to be an attack if, for all IP addresses tracked, the ratio between the detection interval and the history interval is greater than this number. The default setting is 500 percent.
Failed Login Attempts Rate reached
The system considers logon attempts to be an attack if, for all IP addresses tracked, the logon rate reaches this number. The default setting is 100 logon attempts per second.
Minimum Failed Login Attempts
The system considers logon attempts to be an attack if, for all IP addresses tracked, the number of logon attempts is equal to, or greater than, this number. This setting prevents false positive attack detection. The default setting is 20 logon attempts per second.
3.
For the Prevention Policy setting, select the methods you want the system to use to mitigate an attack (the methods are applied in the order listed).
Source IP-Based Client-Side Integrity Defense
Select to determine whether the client is a legal browser or an illegal script by injecting JavaScript into responses when suspicious IP addresses are requested. Legal browsers can process JavaScript and respond properly, whereas illegal scripts cannot. The default is disabled.
URL-Based Client-Side Integrity Defense
Select to determine whether the client is a legal browser or an illegal script by injecting JavaScript into responses when suspicious URLs are requested. Legal browsers can process JavaScript and respond properly, whereas illegal scripts cannot. The default is disabled.
Source IP-Based Rate Limiting
Select to drop requests from suspicious IP addresses. Application Security Manager drops connections to limit the rate of login attempts to the average rate prior to the attack. The default is enabled.
URL-Based Rate Limiting
Select to indicate that when the system detects a URL under attack, Application Security Manager performs rate limiting and limits the rate of all logon requests to the normal level. The default is enabled.
4.
For Suspicious Criteria (per IP address), specify how to identify traffic that may be an attack. If at least one of the criteria is met, the system treats the IP address as an attacker, and prevents the attacker from trying to guess the password. The system also limits the number of login attempts to the normal level.
Failed Login Attempts increased by
Type a number. Login attempts from an individual IP address are considered an attack if the number of failed login attempts has increased by this percentage over the normal number of failed logins. The default setting is 500 percent.
Failed Login Attempts Rate reached
Type a number. An individual IP address is suspicious if the number of login attempts per second from an IP address is equal to or greater than this number. The default setting is 20 login attempts per second.
5.
For the Prevention Duration setting, specify how long the system should mitigate brute force attacks.
Unlimited
Perform attack prevention until it detects the end of the attack.
Maximum
Perform attack prevention either for the time configured here (even if the system detects that the attack continues), or until the system detects the end of the attack, whichever is earlier. Type a value, in seconds, in the field.
6.
In IP Address Whitelist, add the IP addresses or subnets that do not need to be examined for brute force attacks.
You can add up to 20 IP addresses. The IP addresses in the whitelist are also added to the centralized IP address exceptions list. Exceptions from the central list having the Ignore in Anomaly Detection setting enabled, also appear on the list here.
1.
At the bottom of the New Brute Force Protection Configuration screen, click Create.
The screen refreshes, and you see your protected login URL in the list.
2.
To put the security policy changes into effect immediately, click the Apply Policy button in the editing context area.
You can configure the IP Enforcer to perform enforcement based on IP address. When the IP Enforcer is in blocking mode, Application Security Manager drops requests for any IP address if more than the maximum number of blocked violations occur in one minute. The system detects an attacker and drops all requests from that IP address for a specified time interval. The IP Enforcer can also operate in transparent mode where it reports IP addresses that exceed the blocked violation threshold but it does not drop the requests. By default, IP address enforcement is off.
Note: IP address enforcement stops traffic only if the security policys enforcement mode is Blocking, and some violations must have the Block flag enabled (on the Policy Blocking Settings screen).
When the IP Enforcer is configured, you can view IP Enforcer statistics to investigate and release the blocked IP addresses, view dropped requests from that IP address, and examine violations that occurred for each IP address. For details, see Viewing IP Enforcer statistics.
1.
On the Main tab, expand Application Security, point to Anomaly Detection and click IP Enforcer.
The IP Enforcer Configuration screen opens.
2.
In the editing context area, verify that the Current edited policy is the one you want to update.
3.
In the IP Enforcer Configuration area, for the Operation Mode setting, select the mode for IP enforcement:
Off
Does not perform IP-based enforcement. This is the default setting.
Transparent
Issues reporting data. In this mode, if the violation threshold is exceeded, the system logs the attack data, but does not prevent access to the application by the offending client.
Blocking
Prohibits access by this IP address if more than the maximum number of blocked violations occur in one minute.
4.
For the Violation Threshold setting, type the maximum number of blocked violations that you want to allow per minute from each IP address. The default value is 10 blocked violations per minute.
5.
For the Prevention Duration setting, type the number of seconds you want to block (or report) requests. The default value is 60.
6.
Click Save to save the IP enforcement configuration.
7.
To put the security policy changes into effect immediately, click the Apply Policy button in the editing context area.
You can configure Application Security Manager to detect and prevent various bot activities and web scraping on web sites that it is protecting. Web scraping is a technique for extracting information from web sites typically using automated programs, or bots (short for web robots).
Dropped request
If the system cannot examine a request for human activity, the request is dropped with no further checking. This action occurs only if the Block flag is set for the Web scraping detected violation. The system does not drop requests if the security policy is running in transparent mode, or if only the Learn or Alarm flags are set for the violation.
Grace interval
The grace interval is how many requests the system reviews while trying to detect whether the client is human. During the grace interval, requests are not blocked or reported. What occurs next depends on whether the system detects human activity:
If the system detects human activity
The grace interval ends and the system handles the number of requests specified in the Safe Interval, then restarts the grace interval and starts examining again.
If the system does not detect human activity
The system issues the Web Scraping Detected violation until it reaches the number of requests in the Unsafe Interval. If the system is configured to block traffic if that violation occurs, the system blocks requests during this time. In transparent mode or if the violation is set to Alarm only, the violation is logged and requests are permitted. After reaching the Unsafe Interval, the system restarts the grace interval and starts examining again.
The Block setting for the Web Scraping Detected violation is enabled on the Policy Blocking Settings screen.
You can configure the system to perform web scraping detection. If your environment uses legitimate automated tests, you can create a white list of IP addresses or address ranges from which to allow access. The system does not perform web scraping detection on these addresses or the configured search engines.
1.
On the Main tab, expand Application Security, point to Anomaly Detection, then click Web Scraping.
The Web Scraping screen opens.
2.
In the editing context area, verify that the Current edited policy is the one you want to update.
3.
For Web Scraping Detection, select the Enabled check box.
The web scraping settings are now available.
4.
For Grace Interval, type the number of requests to allow while determining whether the client is human. The default value is 100.
5.
For Unsafe Interval, type the number of requests that cause the Web Scraping Detected violation if no human activity is detected during the grace period. When the number is reached, reactivate the grace period. The default value is 100.
6.
For Safe Interval, type the number of requests to allow after human activity is detected and before reactivating the grace threshold to check again for non-human clients. The default value is 2000.
7.
In IP Address Whitelist, add the IP addresses or subnets that do not need to be examined for web scraping.
The IP addresses in the whitelist are also added to the centralized IP address exceptions list. Exceptions from the central list having the Ignore in Anomaly Detection setting enabled, also appear on the list here.
8.
Click Save.
9.
To put the security policy changes into effect immediately, click the Apply Policy button in the editing context area.
By default, Application Security Manager allows requests from well-known search engines and legitimate web robots including the following:
ask.com (ask.com)
Bing (msn.com)
Google (googlebot.com)
Yahoo (crawl.yahoo.net)
You can add other search engines to the search engine list, for example, if your web application uses an additional search engine. The list applies globally to all security policies for which web scraping detection is enabled.
1.
On the Main tab, click Application Security then Options.
3.
Click Create.
The New Search Engine screen opens.
4.
For the Search Engine setting, type the name.
5.
For the Bot Name setting, type the search engine bot name, such as googlebot.
6.
For the Domain Name setting, type the search engine crawlers domain name; for example, yahoo.net.
7.
Click Create.
The system adds the search engine to the list.
Table of Contents   |   << Previous Chapter   |   Next Chapter >>

Was this resource helpful in solving your issue?




NOTE: Please do not provide personal information.



Incorrect answer. Please try again: Please enter the words to the right: Please enter the numbers you hear:

Additional Comments (optional)