Viewing the events as implemented on BIG-IQ® eases processing of Web Application Security events, and provides a way to obtain useful insights regarding the activity on client applications. The BIG-IQ platform enables a single view of all filters and log entries (and details for each entry) from multiple BIG-IP® devices.
It also provides a more intuitive navigation path through the log items.
An FPS profile created on the BIG-IP is used to configure when and how events are triggered on the client. The profile then directs security events to a BIG-IQ Logging Node, and the BIG-IQ system retrieves them from that node.
Logging Node uses a search engine that requires separate services for management and traffic. Keeping those services on separate networks reduces unnecessary congestion. The network designs described here are not required, but considered best practice.
This figure illustrates the network topology required to deploy a logging node for your events.
Logging Node network topology
A BIG-IQ Logging Node is a specially-provisioned BIG-IQ® system, which runs the same software version as the BIG-IQ device that you use to manage your security and the rules that determine your alert responses. After you provision the BIG-IQ Logging Node, you discover it from BIG-IQ and then add the service. After you configure the service, the logging node stores events from one or more BIG-IP® systems. The BIG-IQ system can then retrieve and manage those events.
You need snapshots of your alert data to perform software upgrades, hotfix upgrades, and to restore your .
When event snapshots are created, they need to be stored on a machine other than the Logging Node that stores the events. You define the location for the snapshot by editing the fstab file on your Logging Node machines and on the BIG-IQ® and HA peer devices.
|Schedule the interval at which you want to create snapshots:||
You schedule the system to take snapshots indefinitely. Snapshots are created at the frequency you specify.
For example, if you set the frequency to 6 and Hours, the first log event data snapshot is taken immediately (on Save). Subsequent snapshots are taken every 6 hours.
|Schedule specific days on which you want to create snapshots:||
You schedule the system to take snapshots on specific days.
The BIG-IQ® Logging Node runs as a virtual machine in supported hypervisors, or on the BIG-IQ 7000 series platform. You license the Logging Node using the base registration key you purchased. The base registration key is a character string that the F5 license server uses to provide access to Logging Node features.
You license Logging Node in one of the following ways:
When you license the Logging Node, you:
unit_hostname="%unit_hostname%",management_ip_address="%management_ip_address%", http_class_name="%http_class_name%",web_application_name="%http_class_name%",policy_name="%policy_name%", policy_apply_date="%policy_apply_date%",violations="%violations%",support_id="%support_id%", request_status="%request_status%",response_code="%response_code%",ip_client="%ip_client%", route_domain="%route_domain%",method="%method%",protocol="%protocol%",query_string="%query_string%", x_forwarded_for_header_value="%x_forwarded_for_header_value%",sig_ids="%sig_ids%",sig_names="%sig_names%", date_time="%date_time%",severity="%severity%",attack_type="%attack_type%",geo_location="%geo_location%", ip_address_intelligence="%ip_address_intelligence%",username="%username%",session_id="%session_id%", src_port="%src_port%",dest_port="%dest_port%",dest_ip="%dest_ip%",sub_violations="%sub_violations%", virus_name="%virus_name%",uri="%uri%",request="%request%",violation_details="%violation_details%", header="%headers%",response="%response%
You can either create a new virtual server on the BIG-IP® device that creates the event, or you can use a virtual server that already exists on that device.
The format for an IPv4 address is I<a>.I<b>.I<c>.I<d>. For example, 172.16.254.1.
The format for an IPv6 address is I<a>:I<b>:I<c>:I<d>:I<e>:I<f>:I<g>:I<h>..For example, 2001:db8:85a3:8d3:1319:8a2e:370:7348.
There are a number of useful concepts to consider when you manage Logging Nodes for off-box log storage. This reference material might prove helpful in setting up and maintaining your Logging Node configuration.
Logging Nodes are specialized BIG-IQ® devices designed to provide sufficient CPU, memory, and disk capacity to store and search logging data from BIG-IP® devices. The underlying technology to provide these services is Elasticsearch. (Information about general Elasticsearch comments can be found on their website: https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_concepts.html)
Logging Nodes managed by BIG-IQ provide an Elasticsearch (ES) cluster that can scale horizontally (more nodes = more capacity), but each node in that cluster has limits on disk space. To mitigate that, there are a number of configuration elements that control how much disk is used by the system.
|Disk||10 GB (/var file system )|
The /var file system on the Logging Node (which includes ES data) is only 10GB in size. To store more data on the file system, you need to extend the size. Refer to Index rotation policy for details on managing the data requirements. Extending the file system to 500GB is straightforward, assuming overall disk allocation on the BIG-IQ virtual machine is adequate. Log in as root to the Logging Node, and run the following commands.
The system response will be similar to this:
Directory Name Current Size New Size -------------- ------------ -------- /config 1048576 - /shared 10240000 - /var 10485760 - /var/log 7168000 -
The system response will be similar to this:
Directory Name Current Size New Size -------------- ------------ -------- /config 1048576 - /shared 10240000 - /var 10485760 500000000 /var/log 7168000 -
The system response will be similar to this:
Directory Name Current Size New Size -------------- ------------ -------- /config 1048576 - /shared 10240000 - /var 500003840 - /var/log 7168000 -
The following table is a very rough guide to how much data can be stored on a given Logging Node. The estimate assumes that the Logging Node has been configured to the recommended /var filesystem size. This size is outlined in the Index rotation policy. Because all indexes share the same filesystem, the approximate maximum documents per node estimate assumes no other indexes exist on that node.
Average document size (bytes)
Approximate maximum documents per node
|Access||access-event-logs||730||500GB / 730 = 700 million|
|Access||access-stats||730||500GB / 730 = 700 million|
|ASM||asmindex||1400||500GB / 1400 = 350 million|
|FPS||websafe||1400||10GB / 1400 = 70 million|
The optimum settings used to configure your logging node indices depend on a number of factors. Some of the key factors are discussed here.
The system provides the ability to dynamically create new indices based either on a specified interval or on a specified size. The primary goal to consider when you make these decisions is how to maintain a maximum disk allocation for the Logging Node data while maintaining capacity for new data that flows in.
Secondary considerations include search optimization, and the ability to optimize old indices to reduce their size.
Generally, the best policy is one that does not create unnecessary indices. The more indices, the lower the overall performance, because your searches have to deal with more shards. For example, if a module knows that it has a low indexing volume (thousands/day) then it makes the most sense to have a large aggregation per rotation (5 days or 30 days). For components like Web Application Security that probably have high indexing volumes, it makes more sense to rotate every 8 hours (which reduces the number of retained indices).
Index rotation also allows changing sharding and replica counts by changing the template on a given index type. New indices created from that template will contain the new shard and replica count properties.
This table shows the default configuration values for each index running on the BIG-IQ®. These values are based on anticipated data ingestion rates and typical usage patterns.
|Component||Index Name||Minimum Number of Logging Nodes||Rotation Policy||Retained Index Count||Approximate time window||Size of /var file system|
|Access||access-event-logs||2||Time/5 days||19||95 days||500 GB|
|Access||access-stats||2||Time/5 days||19||95 days||500 GB|
|Web Application Security||asmindex||2||Size/100000 MB||5||N/A||500 GB|
|FPS||websafe||2||Time/30 days||100||8 years||10 GB|
If multiple modules are running on a given Logging Node or if you have higher inbound data rates, you might have to adjust these values to keep the /var file system from filling up (there is a default alert to warn of this when the file system becomes 80% full).
The simplest resolution is to revise the retained index count; lowering this value will reduce the disk space requirements but it will also reduce the amount of data available for queries. For details on changing this setting, refer to Modifying event indices.
The event log interface consists of two filter fields and three main screens:
|Request type||Type a request type or select from the list All requests or Illegal requests (log responses for illegal requests only).|
|Support ID||Type the complete support ID (unique ID given for a transaction), or select the Last 4 digits check box and type the last 4 digits of the support ID.|
|Violation||Use this list to select the policy violation that detects attacks, such as Attack Signature Detection or Illegal Cookie Length. You can select a violation type from the list or you can select none of the violations (indicating that any violation type matches).|
|Attack type||Use this list to select the type of service attacks (such as Denial of Service or HTTP Parser Attack) that you want to see. Select nothing (indicating that any attack type matches), or select a specific attack type.|
|Time Period||In the From and To fields, type a date and time in the format: 2015-12-01T15:15:29-05:00. Or click the calendar icon and select dates.|
|Policies||In the field, type a policy name, or click in the field and, from the list, select a policy.|
You can type a query in the filter box in the format method:'value' protocol:'value' severity:'value'. For example: method:'GET' protocol:'HTTPS' severity:'error'.
Or, you can open the filter and use the method described in the following section.
|Method||From the list, select a method.|
|Protocols||From the list, select HTTP or HTTPS, depending on the security requirements.|
|Severity||From the list, select Informational, Critical, or Error.|
key1:'value' key2:'value' (key3:'value' OR key4:'value').For example:
|attack_type||Name of identified attack (string). For example: Non-browser client.|
|date_time||Current date and time. For example: 2016-09-19 13:52:29|
|dest_ip||Requested service IP address, generally, the virtual server IP address. For example: 192.168.5.11.|
|dest_port||Destination port of this transaction (non-negative integer). For example: 80.|
|geo_location||Country/city location information, based on the source IP address. For example: USA/NY.|
|headers||List of request headers found in request logs. For example: Host: myhost.com; Connection: close.|
|http_class_name||Alias of policy name. For example: /Common/topaz4-web4.|
|ip_address_intelligence||List of IP intelligence categories found for an IP category such as proxy, phishing and so on. For example: Scanners.|
|ip_client||Client source (attacker) IP address. For example: 192.168.5.10.|
|management_ip_address||BIG-IP® management IP address.|
|method||HTTP method requested by the client. For example: GET.|
|policy_apply_date||Last apply policy operation date and time.|
|policy_name||Name of the active security policy. For example: ACME security policy.|
|protocol||Transport protocol (string). For example: HTTP.|
|query_string||URI query string. For example: /.|
|request||Request string sent by the client. For example: GET / HTTP/1.0\r\nUser-Agent: Wget/1.12 (linux-gnu)\r\nAccept: */*\r\nHost: 10.4.1.200\r\nConnection: Keep-Alive\r\n\r\n.|
|request_status||Action applied to the client request. For example: Blocked.|
|response_code||The HTTP response code returned by the back-end server (application). This information is relevant only for requests that are not blocked. For example: 200.|
|route_domain||Route domain number (non-negative integer). For example: 0.|
|session_id||ID number (hexadeicmal number) assigned to the request to allow the system administrator to track requests by session. For example: a9141b68ac7b4958.|
|severity||Severity category to which the event belongs. For example: Error.|
|sig_ids||Signature ID number (positive non-zero integer). For example: 200021069.|
|sig_names||Signature name(s). For example: Automated client access %22wget%22.|
|src_port||Client protocol source port of this transaction (non-negative integer). For example: 52974.|
|sub_violations||Comma-separated list of sub-violation strings. for example: Bad HTTP version, Null in request.|
|support_id||Internally-generated integer to assist with client access support. For example: 18205860747014045721.|
|unit_hostname||BIG-IP system FQDN (unit host name).|
|uri||URI requested by the client (string). For example: /.|
|username||User name for the client session. For example: admin.|
|violations||Comma-separated list of the violations that occurred during enforcement of the request or response. For example: Attack signature detected.|
|virus_name||Virus name (string). For example: Melissa.|
|x_forwarded_for_header_value||Value of the XFF HTTP header (string). For example: 192.168.5.10|
|le||Less than or equal to|
|ge||Greater than or equal to|
The BIG-IQ® user interface does not currently support restoring the event snapshots. However, if a logging node fails, you can manually restore the data up to the last snapshot.
Please note the following:
You can manage configuration snapshots for the configurations you have created on the BIG-IQ® Centralized Management system. A snapshot is a backup copy of a configuration. Configuration snapshots are created manually. This type of snapshot does not include events.
You create a configuration snapshot to compare it to another configuration snapshot, or so you can save the current working configuration and then restore from that snapshot if needed.
The system creates the snapshot and adds it to the list of snapshots on the Snapshot and Restore - screen, including information related to the snapshot, including the date it was created, what account created it, and any description.
You can compare two snapshots to view their differences.
You can restore a snapshot to change the working configuration to that of the snapshot. Restoring the snapshot merges objects from the snapshot into the BIG-IQ® configuration and removes all active locks. No objects in the BIG-IQ configuration are removed. Once the restore process starts, you cannot modify the BIG-IQ configuration until the process is completed or canceled. If the process is canceled, all configuration settings are rolled back.