Manual Chapter : Managing Event Logs for Web Application Security

Applies To:

Show Versions Show Versions

BIG-IQ Centralized Management

  • 5.1.0
Manual Chapter

How do I manage events with a Logging Node?

Viewing the events as implemented on BIG-IQ® eases processing of Web Application Security events, and provides a way to obtain useful insights regarding the activity on client applications. The BIG-IQ platform enables a single view of all filters and log entries (and details for each entry) from multiple BIG-IP® devices.

It also provides a more intuitive navigation path through the log items.

To properly configure event log viewing, you should:
  • Discover and activate a BIG-IQ Logging Node.
  • License and provision a BIG-IQ Logging Node.
  • Define an external machine to which periodic data snapshots are sent.
  • Configure a BIG-IP system to collect events and send them to the BIG-IQ Logging Node. Part of this configuration includes a virtual server configured with an FPS profile.
  • Configure the BIG-IQ for HA, if needed.
  • A minimum of 1 logging node is required, but up to 5 logging nodes are supported.

An FPS profile created on the BIG-IP is used to configure when and how events are triggered on the client. The profile then directs security events to a BIG-IQ Logging Node, and the BIG-IQ system retrieves them from that node.

Logging Node uses a search engine that requires separate services for management and traffic. Keeping those services on separate networks reduces unnecessary congestion. The network designs described here are not required, but considered best practice.

BIG-IQ Networks

  • A cluster management network to perform Elasticsearch configuration and status operations
  • A cluster traffic network for inter-node communication

Logging Node Networks

  • A cluster management network to perform Elasticsearch configuration and status operations
  • A cluster traffic network for inter-node communication
  • A listener network to handle inbound data traffic

This figure illustrates the network topology required to deploy a logging node for your events.

Logging Node network topology

Important: F5 Networks strongly recommends that the Listener Network and Management Networks be separate. This separation, can help with data protection and management network availability in case the Listener Network is flooded with data.

What is a BIG-IQ Logging Node?

A BIG-IQ Logging Node is a specially-provisioned BIG-IQ® system, which runs the same software version as the BIG-IQ device that you use to manage your security and the rules that determine your alert responses. After you provision the BIG-IQ Logging Node, you discover it from BIG-IQ and then add the service. After you configure the service, the logging node stores events from one or more BIG-IP® systems. The BIG-IQ system can then retrieve and manage those events.

Note: The software version on the Logging Node must be the same as the version on its partner BIG-IQ system. If you need to upgrade the Logging Node, follow the instructions in Upgrading BIG-IQ Systems.

Discover and activate a logging node

Using BIG-IQ® System Management, you can discover a Logging Node and add it to the Logging Group. The BIG-IQ can then access all event on the discovered Logging Node. You can then receive these events from multiple BIG-IP® systems. This unified view makes browsing easier, and provides a complete view of application event activity.
  1. Log in to BIG-IQ system with your administrator user name and password.
  2. At the top left of the screen, select System Management from the BIG-IQ menu.
  3. On the left, expand BIG-IQ LOGGING.
  4. Under BIG-IQ LOGGING, select Logging Nodes.
  5. Click Add Node.
  6. On the New Logging Device screen, fill in as appropriate:
    1. In IP Address, type the management IP address.
    2. In User Name, type the user name for an administrator on the Logging Node (for example, admin.
    3. In Password, type the password for an administrator on the Logging Node (for example, admin.
    4. In Transport Address, type the IP address of the logging node internal self IP address.
    5. For Transport Port, the default value is 9300. The BIG-IQ uses this port for internal polling and communication with the logging nodes.
  7. Click the Add button at the bottom of the screen to add the Logging Node to the system. Or, click Discard to cancel the operation.
    Note: This operation might take a minute or two.
  8. Repeat these 7 steps for each Logging Node you want to configure.
  9. To activate this logging node for the service you want to monitor, in the Services column, click Add Services.
    The Logging Node Services screen opens.
  10. For the service you want to add, confirm that the Listener Address correctly specifies the external self IP address of the Logging Node, and click Activate.
    When the service is successfully added, the Service Status changes to Active.
  11. Click Close.
Once discovered and activated, this logging node collects the events generated by the configured BIG-IP systems. Thus, BIG-IQ provides a single view of all event entries.
Important: The Total Document Count is not a report of the number of alerts sent to the Logging Node. Instead, it is a sum of various document types sent to the Logging Node. Alerts are included in this list, but this total includes other document types as well.

Modifying event log indices

Event log indices determine the physical characteristics of what is sent to the Logging Node.
  1. Log in to F5® BIG-IQ® Centralized Management with your user name and password.
  2. At the top left of the screen, select System Management from the BIG-IQ menu.
  3. On the left, select Logging Configuration.
    The Logging Configuration screen opens to display the current state of the logging node cluster defined for this device.
  4. In the Web Application Security row in the bottom half of the screen, click the Configure button.
    The Web Application Security Indices screen opens.
  5. For the Rotation Type, keep the default setting: Size Based.
  6. For the Max Index Size, type the maximum size of the indices you want to send to the logging node.
    For example, if you type 1000, when the event log data reaches a size of 1 Gig, it is sent to the logging node.
  7. For the Retained Index Count, type the total number of indexes you want to store on the logging node.
    The maximum amount of data stored on the Logging Node is the product of the Max Index Size and the Retained Index Count. When the amount of data reaches this size, the oldest event data is truncated or discarded.
  8. Click Save to save the indices configuration settings.

Define event snapshot storage locations

Before you can configure the external snapshot storage location, you need the following information on the machine you will use to store the event snapshots:
  • storage-machine-IP-address
  • storage-file-path
  • Read/Write permissions for the storage file path

You need snapshots of your alert data to perform software upgrades, hotfix upgrades, and to restore your .

When event snapshots are created, they need to be stored on a machine other than the Logging Node that stores the events. You define the location for the snapshot by editing the fstab file on your Logging Node machines and on the BIG-IQ® and HA peer devices.

Important: You must perform this task on each Logging Node device, on the BIG-IQ device, and on the BIG-IQ HA peer.
  1. On the first device, in the folder /var/config/rest/elasticsearch/data/, create a new folder named essnapshot.
    mkdir /var/config/rest/elasticsearch/data/essnapshot
  2. Edit the /etc/fstab file to add /var/config/rest/elasticsearch/data/essnapshot.
    For example, //<storage machine ip-address>/<storage-file-path> /var/config/rest/elasticsearch/data/essnapshot cifs iocharset=utf8,rw,noauto,uid=elasticsearch,gid=elasticsearch, 0 0
  3. Run the mount command to mount the snapshot storage location to the new folder.
    For example, from /var/config/rest/elasticsearch/data type: mount essnapshot.
  4. Confirm that the essnapshot folder has full read, write, and execute permissions, (specifically Chmod 777 essnapshot), and that the owner and group are elasticsearch for this folder.
    For example, ls-l would yield: drwxrwxrwx 3 elasticsearch elasticsearch 0 Apr 25 11:27 essnapshot.
  5. Create a test file to confirm that the storage file-path has been successfully mounted.
    For example: touch testfile.
    The test file should be created on the storage machine at the location storage file path.
  6. Repeat these five steps for each Logging Node, the BIG-IQ, and the BIG-IQ HA peer.
The storage location should now be accessible to the BIG-IQ devices and to the logging node machines.

Define Web Application Security snapshot schedules

Before you define snapshot schedules, you must have defined the snapshot storage locations.
Snapshots of the events sent to your Logging Nodes are an essential safeguard for your data. If the machine that stores the events fails, the data can be restored using these snapshots. These snapshots are created based on the snapshot schedules you define. F5 recommends that you schedule snapshots at least every 6 hours and retain at least 4 snapshots.
  1. Log in to BIG-IQ system with your administrator user name and password.
  2. At the top left of the screen, select System Management from the BIG-IQ menu.
  3. On the left, expand BIG-IQ LOGGING.
  4. Under BIG-IQ LOGGING, select Logging Configuration.
  5. For the Snapshot Schedules setting, click Create.
    The New Logging Snapshot screen opens.
  6. For the Snapshot Name Prefix, type the string that you want to use to identify the snapshots created by this schedule.
    For example snapshot_.
  7. In Snapshots to Keep, specify the number of snapshots that you want to accumulate before they are deleted for space constraints.
    For example, if you specify 25, then the system will retain a maximum of 25 snapshots before it starts to delete older snapshots as new snapshots are created. You can save up to 100.
  8. Define how you want the snapshots scheduled.
    Option Description
    Schedule the interval at which you want to create snapshots:

    You schedule the system to take snapshots indefinitely. Snapshots are created at the frequency you specify.

    1. Select Repeat Interval.
    2. Specify the Snapshot Frequency.
    3. Select a time increment.

    For example, if you set the frequency to 6 and Hours, the first log event data snapshot is taken immediately (on Save). Subsequent snapshots are taken every 6 hours.

    Schedule specific days on which you want to create snapshots:

    You schedule the system to take snapshots on specific days.

    1. Select Days of the Week.
    2. For the Days of the Week setting, select the days on which you want backups to occur.
    3. For the Start Date, select the time (date, hour, minute, and AM or PM) on which you want backups to start.
  9. Click Save to save the new schedule.

How do I license and do the basic setup to start using a Logging Node?

The BIG-IQ® Logging Node runs as a virtual machine in supported hypervisors, or on the BIG-IQ 7000 series platform. You license the Logging Node using the base registration key you purchased. The base registration key is a character string that the F5 license server uses to provide access to Logging Node features.

You license Logging Node in one of the following ways:

  • If the system has access to the internet, you can have the Logging Node contact the F5 license server and automatically activate the license.
  • If the system is not connected to the internet, you can manually retrieve the activation key from a system that is connected to the internet, and transfer it to the Logging Node.
  • If your Logging Node is in a closed-circuit network (CCN) that does not allow you to export any encrypted information, you must open a case with F5 support.

When you license the Logging Node, you:

  • Specify a host name for the system.
  • Assign a management port IP address.
  • Specify the IP address of your DNS server and the name of the DNS search domain.
  • Specify the IP address of your Network Time Protocol (NTP) servers and select a time zone.
  • Change the administrator’s default admin and root passwords.

Automatically license BIG-IQ and perform initial setup

You must have a base registration key before you can license the BIG-IQ® system. If you do not have a base registration key, contact the F5 Networks sales group (http://www.f5.com).
If the BIG-IQ® system is connected to the public internet, you can follow these steps to automatically perform the initial license activation and perform the initial setup.
  1. Use a browser to log in to BIG-IQ by typing https://<management_IP_address>, where <management_IP_address> is the address you specified for device management.
  2. Log in to F5® BIG-IQ® Centralized Management with your user name and password.
  3. Click Activate.
    The Base Registration Key field is added to the screen.
  4. In the Base Registration Key field, type or paste the BIG-IQ registration key.
    Important: The registration key you use must support a Logging Node capable license.
  5. In the Add-On Keys field, paste any additional license key you have.
  6. To add another additional add-on key, click the + sign and paste the additional key in the new Add-On Keys field.
  7. For the Activation Method setting, select Automatic, and click the Activate License button.
    The End User Software License Agreement (EULA) displays.
  8. To accept the license agreement, click the Agree button.
  9. Click the Next button at the right of the screen.
    If the license you purchased supports both Logging Node and BIG-IQ Central Management Console, the License Feature Selection popup screen opens. Otherwise the Management Address screen opens.
  10. If you are prompted with the License Feature Selection, select BIG-IQ Logging Node, and then click OK. If you are not prompted, proceed to the next step.
    Important: This choice cannot be undone. Once you license a device as a Logging Node, you cannot change your mind and license it as a BIG-IQ Management Console.
    The Management Address screen opens.
  11. In the Host Name field, type a fully-qualified domain name (FQDN) for the system.
    You cannot change this name after you add it. The FQDN can consist of letters and numbers, as well as the characters underscore ( _ ), dash ( - ), or period ( . ).
  12. In the Management Port IP Address field, type the IP address for the management port IP address.
    Note: The management port IP address must be in Classless Inter-Domain Routing (CIDR) format. For example: 10.10.10.10/24.
  13. In the Management Port Route field that the system creates, type the IP address for the management port route.
  14. Specify what you want the BIG-IQ to use for the Discovery Address.
    • To use the management port, select Use Management Address.
    • To use the internal self IP address, select Self IP Address, and type the IP address.
      Important: If you are configuring a Logging Node device, you must use the internal self IP address.
      Note: The self IP address must be in Classless Inter-Domain Routing (CIDR) format. For example: 10.10.10.10/24.
  15. Click the Next button at the right of the screen.
  16. In the DNS Lookup Servers field, type the IP address of your DNS server.
    You can click the Test Connection button to verify that the IP address is reachable.
  17. In the DNS Search Domains field, type the name of your search domain.
    The DNS search domain list allows the BIG-IQ system to search for local domain lookups to resolve local host names.
  18. In the Time Servers fields, type the IP addresses of your Network Time Protocol (NTP) servers.
    You can click the Test Connection button to verify that the IP address is reachable.
  19. From the Time Zone list, select your local time zone.
  20. Click the Next button at the right of the screen.
  21. In the Old Password fields, type the default admin and root passwords, and then type a new password in the Password and Confirm Password fields.
  22. Click the Next button at the right of the screen.

Manually license BIG-IQ and perform initial setup

You must have a base registration key before you can license the BIG-IQ® system. If you do not have a base registration key, contact the F5 Networks sales group (http://www.f5.com).
If the BIG-IQ® system is not connected to the public internet, use this procedure to manually activate the license and perform the initial setup.
  1. Use a browser to log in to BIG-IQ by typing https://<management_IP_address>, where <management_IP_address> is the address you specified for device management.
  2. Log in to F5® BIG-IQ® Centralized Management with your user name and password.
  3. Click Activate.
    The Base Registration Key field is added to the screen.
  4. In the Base Registration Key field, type or paste the BIG-IQ registration key.
    Important: The registration key you use must support a Logging Node capable license.
  5. In the Add-On Keys field, paste any additional license key you have.
  6. For the Activation Method setting, select Manual and click the Generate Dossier button.
    The BIG-IQ system refreshes and displays the dossier in the Device Dossier field.
  7. Select and copy the text displayed in the Device Dossier field.
  8. Click the Access F5 manual activation web portal link.
    The Activate F5 Product site opens.
  9. Into the Enter your dossier field, paste the dossier.
    Alternatively, if you saved the file, click the Choose File button and navigate to it.
    After a pause, the license key text displays.
  10. Click the Next button.
    The Accept User Legal Agreement screen opens.
  11. To accept the license agreement, select the I have read and agree to the terms of this license, and click Next. button.
    The licensing server creates the license key text.
  12. Copy the license key.
  13. In the License Text field on BIG-IQ, paste the license text.
  14. Click the Activate License button.
  15. Click the Next button at the right of the screen.
    If the license you purchased supports both Logging Node and BIG-IQ Central Management Console, the License Feature Selection popup screen opens. Otherwise the Management Address screen opens.
  16. If you are prompted with the License Feature Selection, select BIG-IQ Logging Node, and then click OK. If you are not prompted, proceed to the next step.
    Important: This choice cannot be undone. Once you license a device as a Logging Node, you cannot change your mind and license it as a BIG-IQ Management Console.
    The Management Address screen opens.
  17. In the Host Name field, type a fully-qualified domain name (FQDN) for the system.
    You cannot change this name after you add it. The FQDN can consist of letters and numbers, as well as the characters underscore ( _ ), dash ( - ), or period ( . ).
  18. In the Management Port IP Address field, type the IP address for the management port IP address.
    Note: The management port IP address must be in Classless Inter-Domain Routing (CIDR) format. For example: 10.10.10.10/24.
  19. In the Management Port Route field that the system creates, type the IP address for the management port route.
  20. Specify what you want the BIG-IQ to use for the Discovery Address.
    • To use the management port, select Use Management Address.
    • To use the internal self IP address, select Self IP Address, and type the IP address.
      Important: If you are configuring a Logging Node device, you must use the internal self IP address.
      Note: The self IP address must be in Classless Inter-Domain Routing (CIDR) format. For example: 10.10.10.10/24.
  21. Click the Next button to save your configuration.
  22. In the DNS Lookup Servers field, type the IP address of your DNS server.
    You can click the Test Connection button to verify that the IP address is reachable.
  23. In the DNS Search Domains field, type the name of your search domain.
    The DNS search domain list allows the BIG-IQ system to search for local domain lookups to resolve local host names.
  24. In the Time Servers fields, type the IP addresses of your Network Time Protocol (NTP) servers.
    You can click the Test Connection button to verify that the IP address is reachable.
  25. From the Time Zone list, select your local time zone.
  26. Click the Next button at the right of the screen.
  27. In the Old Password fields, type the default admin and root passwords, and then type a new password in the Password and Confirm Password fields.
  28. Click the Next button at the right of the screen.

Configuring the BIG-IP logging profile

Each properly-configured BIG-IP® system sends its event log to a BIG-IQ® Logging Node. You configure the BIG-IP system to do so by creating a logging profile and assigning the logging profile to a virtual server, and then deploying it to the BIG-IP system. The logging profile defines the content of the events, and identifies the Logging Node to which the events are sent.
  1. Log in to BIG-IQ system with your administrator user name and password.
  2. Access the BIG-IQ component you are setting up, using the BIG-IQ menu and options near the top of the screen.
    • If you are setting up Web Application Security, select Web Application Security and then Shared Security.
    • If you are setting up Fraud Protection Service, select Network Security and then Shared Security.
  3. On the left, expand SECURITY PROFILES and click Logging Profiles.
  4. On the Logging Profiles screen, click Create.
    The Logging Profiles properties screen opens, showing the Properties tab.
  5. On the Properties tab, edit as appropriate:
    1. In the Name field, type a unique name for this new profile. This field is required.
    2. For Application Security, select Enabled. The Application Security tab appears.
      When you select this option, the Protocol Security option is not available, but you can select any of the other options.
    3. To use Network Firewall, select the Enabled check box; the Network Firewall tab appears.
    4. To use DoS Protection, select the Enabled check box; the DoS Protection tab appears.
  6. On the Application Security tab, select Remote Storage.
    Several new fields appear, including the Protocol list.
  7. Specify the appropriate Logging Format.
    • If the BIG-IP device runs version 12.0 or later, select BIG-IQ.
    • If the BIG-IP device runs a version prior to 12.0, select Comma-Separated Values. Several new fields appear.
      • For Storage Format, select User Defined.
      • In the Selected Items field, paste the following text:
        unit_hostname="%unit_hostname%",management_ip_address="%management_ip_address%",
        http_class_name="%http_class_name%",web_application_name="%http_class_name%",policy_name="%policy_name%",
        policy_apply_date="%policy_apply_date%",violations="%violations%",support_id="%support_id%",
        request_status="%request_status%",response_code="%response_code%",ip_client="%ip_client%",
        route_domain="%route_domain%",method="%method%",protocol="%protocol%",query_string="%query_string%",
        x_forwarded_for_header_value="%x_forwarded_for_header_value%",sig_ids="%sig_ids%",sig_names="%sig_names%",
        date_time="%date_time%",severity="%severity%",attack_type="%attack_type%",geo_location="%geo_location%",
        ip_address_intelligence="%ip_address_intelligence%",username="%username%",session_id="%session_id%",
        src_port="%src_port%",dest_port="%dest_port%",dest_ip="%dest_ip%",sub_violations="%sub_violations%",
        virus_name="%virus_name%",uri="%uri%",request="%request%",violation_details="%violation_details%",
        header="%headers%",response="%response%
        
        Note: The line breaks in the example above were necessary due to screen width; remove all of them after you paste this data. It must be a single string with no white space.
  8. For Protocol, select TCP.
  9. For the Server Addresses settings, specify the address you want to use:
    1. In the IP Address field, type the Logging Node's management IP address.
    2. Specify the port to use for your data.
      • If you are setting up a logging profile for Web Application Security, type 8514 in the Port field.
      • If you are setting up a logging profile for Fraud Protection Service, type 8008 in the Port field.
    3. Click the Add button to add the address and port to the list of servers.
  10. To specify the Storage Format for this data, select the preferred formats in the Available Items list, and click the right arrow to add them to the Selected Items list.
  11. For the Maximum Entry Length, select 64k.
  12. In the Storage Filter area, from the Request Type list, select All requests.
  13. Click Save to save the new profile.

Virtual servers that remote logging uses to route event logs

You can either create a new virtual server on the BIG-IP® device that creates the event, or you can use a virtual server that already exists on that device.

Creating a virtual server for remote logging

If the device for which you are configuring remote logging does not have a virtual server, you need to create one.
  1. Log in to BIG-IQ system with your administrator user name and password.
  2. At the top left of the screen, select ADC from the BIG-IQ menu.
  3. On the left, expand LOCAL TRAFFIC.
  4. Under LOCAL TRAFFIC, select Virtual Servers.
    The screen displays a list of virtual servers defined on this device.
  5. Click Create.
    The Virtual Servers - New Item screen opens.
  6. In the Name field, type in a name for the virtual server you are creating.
  7. From the Device list, select the device on which to create the virtual server.
  8. In the Description field, type in a brief description for the virtual server you are creating.
  9. For the Destination Address, type the IP address of the destination you want to add to the Destination list.

    The format for an IPv4 address is I<a>.I<b>.I<c>.I<d>. For example, 172.16.254.1.

    The format for an IPv6 address is I<a>:I<b>:I<c>:I<d>:I<e>:I<f>:I<g>:I<h>..

    For example, 2001:db8:85a3:8d3:1319:8a2e:370:7348.
  10. In the Service Port field, type a service port number, or select a type from the list.
    When you select a type from the list, the value in the Service Port field changes to reflect the associated default, which you can change.
  11. Click Save.
    The system creates the new virtual server with the settings you specified.
  12. Click Save to save the assignment. Or, click Save & Close to save the assignment and return to the Virtual Servers screen.
A virtual server that can be used to routeevent data to the logging node is created for the BIG-IP® device.
Before the BIG-IP device can actually use this new virtual server, you must deploy it to the device.

Assigning the logging profile to a virtual server

After configuring a logging profile on the BIG-IQ® system, you must assign it to a virtual server and deploy it to the BIG-IP® system from which you want to collect event logs.
  1. Log in to BIG-IQ system with your administrator user name and password.
  2. Access the BIG-IQ component you are setting up, using the BIG-IQ menu and options near the top of the screen.
    • If you are setting up Web Application Security, select Web Application Security and then Shared Security.
    • If you are setting up Fraud Protection Service, select Network Security and then Shared Security.
  3. If necessary, expand ADC, then select Virtual Servers.
    The screen displays a list of virtual servers that are configured with devices that have been provisioned and discovered.
  4. On the Virtual Servers screen, click the name of the virtual server you want to use.
    The Virtual Servers - Properties screen opens.
  5. From the Log Profiles list, under Available, click a logging profile and move it to the Selected list.
  6. Click Save to save the assignment. Or, click Save & Close to save the assignment and return to the Virtual Servers screen.
You are now ready to deploy your virtual server (with logging profile) to the BIG-IP system(s) from which you want to collect event logs.

Logging Node management

There are a number of useful concepts to consider when you manage Logging Nodes for off-box log storage. This reference material might prove helpful in setting up and maintaining your Logging Node configuration.

Logging Node sizing guide

Logging Nodes are specialized BIG-IQ® devices designed to provide sufficient CPU, memory, and disk capacity to store and search logging data from BIG-IP® devices. The underlying technology to provide these services is Elasticsearch. (Information about general Elasticsearch comments can be found on their website: https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_concepts.html)

Logging Nodes managed by BIG-IQ provide an Elasticsearch (ES) cluster that can scale horizontally (more nodes = more capacity), but each node in that cluster has limits on disk space. To mitigate that, there are a number of configuration elements that control how much disk is used by the system.

Logging Node Minimum Recommended Configuration
CPU 8 Cores
Memory 32 GB
Disk 10 GB (/var file system )

The /var file system on the Logging Node (which includes ES data) is only 10GB in size. To store more data on the file system, you need to extend the size. Refer to Index rotation policy for details on managing the data requirements. Extending the file system to 500GB is straightforward, assuming overall disk allocation on the BIG-IQ virtual machine is adequate. Log in as root to the Logging Node, and run the following commands.

  1. tmsh show sys disk directory

    The system response will be similar to this:

    Directory Name                  Current Size    New Size
    --------------                  ------------    --------
    /config                         1048576         -
    /shared                         10240000        -
    /var                            10485760        -
    /var/log                        7168000         -
            
  2. tmsh modify sys disk directory /var new-size 500000000

    tmsh show sys disk directory

    The system response will be similar to this:

    Directory Name                  Current Size    New Size
    --------------                  ------------    --------
    /config                         1048576         -
    /shared                         10240000        -
    /var                            10485760        500000000
    /var/log                        7168000         -
            
  3. Reboot the system and then confirm the size disk size.

    tmsh show sys disk directory

    The system response will be similar to this:

    Directory Name                  Current Size    New Size
    --------------                  ------------    --------
    /config                         1048576         -
    /shared                         10240000        -
    /var                            500003840       -
    /var/log                        7168000         -
            
Logging Node Capacity

The following table is a very rough guide to how much data can be stored on a given Logging Node. The estimate assumes that the Logging Node has been configured to the recommended /var filesystem size. This size is outlined in the Index rotation policy. Because all indexes share the same filesystem, the approximate maximum documents per node estimate assumes no other indexes exist on that node.

Module Index name

Average document size (bytes)

Approximate maximum documents per node

Access access-event-logs 730 500GB / 730 = 700 million
Access access-stats 730 500GB / 730 = 700 million
ASM asmindex 1400 500GB / 1400 = 350 million
FPS websafe 1400 10GB / 1400 = 70 million

Index rotation policy

The optimum settings used to configure your logging node indices depend on a number of factors. Some of the key factors are discussed here.

The system provides the ability to dynamically create new indices based either on a specified interval or on a specified size. The primary goal to consider when you make these decisions is how to maintain a maximum disk allocation for the Logging Node data while maintaining capacity for new data that flows in.

Secondary considerations include search optimization, and the ability to optimize old indices to reduce their size.

Generally, the best policy is one that does not create unnecessary indices. The more indices, the lower the overall performance, because your searches have to deal with more shards. For example, if a module knows that it has a low indexing volume (thousands/day) then it makes the most sense to have a large aggregation per rotation (5 days or 30 days). For components like Web Application Security that probably have high indexing volumes, it makes more sense to rotate every 8 hours (which reduces the number of retained indices).

Index rotation also allows changing sharding and replica counts by changing the template on a given index type. New indices created from that template will contain the new shard and replica count properties.

This table shows the default configuration values for each index running on the BIG-IQ®. These values are based on anticipated data ingestion rates and typical usage patterns.

Component Index Name Minimum Number of Logging Nodes Rotation Policy Retained Index Count Approximate time window Size of /var file system
Access access-event-logs 2 Time/5 days 19 95 days 500 GB
Access access-stats 2 Time/5 days 19 95 days 500 GB
Web Application Security asmindex 2 Size/100000 MB 5 N/A 500 GB
FPS websafe 2 Time/30 days 100 8 years 10 GB

If multiple modules are running on a given Logging Node or if you have higher inbound data rates, you might have to adjust these values to keep the /var file system from filling up (there is a default alert to warn of this when the file system becomes 80% full).

The simplest resolution is to revise the retained index count; lowering this value will reduce the disk space requirements but it will also reduce the amount of data available for queries. For details on changing this setting, refer to Modifying event indices.

How do I use the event log interface?

The event log interface consists of two filter fields and three main screens:

  • Filter fields:
    • Selected devices filter. This filter appears below the Event Logs header. You can use it to select one or more devices for event viewing.
    • Filter field. Appears to the right of the selected devices field. You can use it to type text to rapidly narrow the search scope. You can also save filters that you use often.
  • Screens:
    • Devices. At the far left, use this to select a group of requests, policies, saved filters, or pre-configured tags. The object you select determines the set of items that appears in the next screen.
    • Log items. Use this to browse log items, or select one and view log item details.
    • Details. Displays details of the item selected in the Log items screen.

Viewing event log details

You can view request and response details for a single log item.
  1. Log in to the BIG-IQ system with your administrator user name and password.
  2. From the BIG-IQ list, select WebApplication Security.
  3. Click Event Logs.
  4. On the Log Items screen (list of events), click a single event log.
    The Details screen displays a variety of information about the event.
  5. On the Details screen, click Request to view request details.
    Details include:
    • Raw HTTP[S] request
    • General request details
    • Geolocation
    • Policy details
    • List of related tags
  6. Click Response to view response details.

Using common filters

You can update common filters for requests and security policies.
  1. Log in to the BIG-IQ system with your administrator user name and password.
  2. From the BIG-IQ list, select Web Application Security.
  3. Click Event Logs.
  4. To update log items according to a selected filter (such as Requests or Policies), click any item under Requests or Policies.
The system updates log items according to the selected filter, and results appear in the Log Items screen.

Filtering the event logs (basic)

You can use the filter to refine your searches through the event logs, including searches through logs from multiple BIG-IP® devices.
  1. Log in to BIG-IQ system with your administrator user name and password.
  2. From the BIG-IQ list, select Web Application Security.
  3. Click Event Logs.
  4. In the Filter field, click the triangle to the right of the field.
    The Search filter popup screen opens to the basic view, which is the default.
  5. Complete the fields.
    Setting Description
    Request type Type a request type or select from the list All requests or Illegal requests (log responses for illegal requests only).
    Support ID Type the complete support ID (unique ID given for a transaction), or select the Last 4 digits check box and type the last 4 digits of the support ID.
    Violation Use this list to select the policy violation that detects attacks, such as Attack Signature Detection or Illegal Cookie Length. You can select a violation type from the list or you can select none of the violations (indicating that any violation type matches).
    Attack type Use this list to select the type of service attacks (such as Denial of Service or HTTP Parser Attack) that you want to see. Select nothing (indicating that any attack type matches), or select a specific attack type.
    Time Period In the From and To fields, type a date and time in the format: 2015-12-01T15:15:29-05:00. Or click the calendar icon and select dates.
    Policies In the field, type a policy name, or click in the field and, from the list, select a policy.
  6. Click the Search bar.
    The results of the filtering process appear in the Log Items list.
  7. When you have configured a search that you will use repeatedly or frequently, click Save the current filter, type a filter name, and click Save.
    The saved filter appears in the left panel under Saved filter.

Filtering (advanced)

You can use the filter's advanced setting to refine your searches.

You can type a query in the filter box in the format method:'value' protocol:'value' severity:'value'. For example: method:'GET' protocol:'HTTPS' severity:'error'.

Or, you can open the filter and use the method described in the following section.

  1. Log in to BIG-IQ system with your administrator user name and password.
  2. From the BIG-IQ list, select Web Application Security.
  3. Click Event Logs.
  4. Open the Filter field.
    The Search filter popup screen opens to the basic view, which is the default.
  5. Click Advanced.
  6. Complete the fields.
    Setting Description
    Method From the list, select a method.
    Protocols From the list, select HTTP or HTTPS, depending on the security requirements.
    Severity From the list, select Informational, Critical, or Error.
  7. Click the search bar.
    The results of the filtering process appear in the Log Items list.
  8. When you have configured a search that you will use repeatedly or frequently, click Save the current filter, type a filter name, and click Save.
    The saved filter appears in the left panel under Saved filter.

Filtering by entering query parameters

You can use the BIG-IQ® Filter field to enter a query in ODATA format:

key1:'value' key2:'value' (key3:'value' OR key4:'value').

For example:

policy_name:'/Common/policy1'

Note: The BIG-IQ system supports AND/OR constructs.
  • OR. Use this operator to log the data that meets one or more of the criteria.
  • AND. Use this operator to log the data that meets all of the criteria.
Keys, values, and operators are listed and described in the following text.
  1. Log in to the BIG-IQ system with your administrator user name and password.
  2. From the BIG-IQ list, select Web Application Security.
  3. Click Event Logs.
  4. In the Filter field, type a query in ODATA format.
  5. Type a key from the following list:
    Key Description
    attack_type Name of identified attack (string). For example: Non-browser client.
    date_time Current date and time. For example: 2016-09-19 13:52:29
    dest_ip Requested service IP address, generally, the virtual server IP address. For example: 192.168.5.11.
    dest_port Destination port of this transaction (non-negative integer). For example: 80.
    geo_location Country/city location information, based on the source IP address. For example: USA/NY.
    headers List of request headers found in request logs. For example: Host: myhost.com; Connection: close.
    http_class_name Alias of policy name. For example: /Common/topaz4-web4.
    ip_address_intelligence List of IP intelligence categories found for an IP category such as proxy, phishing and so on. For example: Scanners.
    ip_client Client source (attacker) IP address. For example: 192.168.5.10.
    management_ip_address BIG-IP® management IP address.
    method HTTP method requested by the client. For example: GET.
    policy_apply_date Last apply policy operation date and time.
    policy_name Name of the active security policy. For example: ACME security policy.
    protocol Transport protocol (string). For example: HTTP.
    query_string URI query string. For example: /.
    request Request string sent by the client. For example: GET / HTTP/1.0\r\nUser-Agent: Wget/1.12 (linux-gnu)\r\nAccept: */*\r\nHost: 10.4.1.200\r\nConnection: Keep-Alive\r\n\r\n.
    request_status Action applied to the client request. For example: Blocked.
    response_code The HTTP response code returned by the back-end server (application). This information is relevant only for requests that are not blocked. For example: 200.
    route_domain Route domain number (non-negative integer). For example: 0.
    session_id ID number (hexadeicmal number) assigned to the request to allow the system administrator to track requests by session. For example: a9141b68ac7b4958.
    severity Severity category to which the event belongs. For example: Error.
    sig_ids Signature ID number (positive non-zero integer). For example: 200021069.
    sig_names Signature name(s). For example: Automated client access %22wget%22.
    src_port Client protocol source port of this transaction (non-negative integer). For example: 52974.
    sub_violations Comma-separated list of sub-violation strings. for example: Bad HTTP version, Null in request.
    support_id Internally-generated integer to assist with client access support. For example: 18205860747014045721.
    unit_hostname BIG-IP system FQDN (unit host name).
    uri URI requested by the client (string). For example: /.
    username User name for the client session. For example: admin.
    violations Comma-separated list of the violations that occurred during enforcement of the request or response. For example: Attack signature detected.
    virus_name Virus name (string). For example: Melissa.
    x_forwarded_for_header_value Value of the XFF HTTP header (string). For example: 192.168.5.10
  6. Type an operator from the following list:
    Operator Description
    eq Equal
    ne Not equal
    lt Less than
    le Less than or equal to
    gt Greater than
    ge Greater than or equal to
  7. Type a value in any of the following formats:
    • 'value'. For example: policy_name:'/Common/policy1'
    • '*alue'. For example: policy_name:'*Common/policy1'
    • 'alu*'. For example: policy_name:'Common/policy*'
    • '*ue*'. For example: policy_name:'policy*'
  8. Press Enter or click the search icon to start the search.
The system updates log items according to the typed query, and results appear in the Log Items list.

Restore event log snapshots

To submit the REST API calls required by this task, you must provide the administrator user name and password.

The BIG-IQ® user interface does not currently support restoring the event snapshots. However, if a logging node fails, you can manually restore the data up to the last snapshot.

Please note the following:

  • The restore operation requires a down time during which no BIG-IQ or Logging Node work is performed.
  • During the restore operation, no event data sent to the Logging Node is retained.
  • The restore operation restores only the data from the time before the chosen snapshot was created. Data from the time that the chosen snapshot was created to the current time is not restored.
  • Before initiating a snapshot restore, make sure that sufficient disk space is allocated to the /var folder on the device to which you are restoring the snapshot.
  1. Log in to BIG-IQ system with your administrator user name and password.
  2. At the top left of the screen, select System Management from the BIG-IQ menu.
  3. On the left, expand BIG-IQ LOGGING and select Logging Configuration.
    The Logging Configuration screen opens.
  4. Click the View History button.
    The BIG-IQ Logging Snapshots screen opens.
  5. Browse through the list to get an idea of which snapshot you want to restore.
  6. On the Logging Configuration screen, next to Last Snapshot/Time, click Restore.

Managing Configuration Snapshots

What is snapshot management?

You can manage configuration snapshots for the configurations you have created on the BIG-IQ® Centralized Management system. A snapshot is a backup copy of a configuration. Configuration snapshots are created manually. This type of snapshot does not include events.

Creating a snapshot

You create a configuration snapshot to compare it to another configuration snapshot, or so you can save the current working configuration and then restore from that snapshot if needed.

  1. Log in to F5® BIG-IQ® Centralized Management with your user name and password.
  2. At the top left of the screen, select Change Management from the BIG-IQ menu.
  3. Under SNAPSHOT & RESTORE, select the service configuration from which to create the snapshot: Network Security or Web Application Security.
    The screen displays a list of snapshots that have been created for that service on this device.
  4. At the top of the screen, click Create.
    The Create Snapshot screen opens.
  5. Supply the values on the Create Snapshot screen, and click Create.

The system creates the snapshot and adds it to the list of snapshots on the Snapshot and Restore - screen, including information related to the snapshot, including the date it was created, what account created it, and any description.

Comparing snapshots

You can compare two snapshots to view their differences.

  1. Log in to F5® BIG-IQ® Centralized Management with your user name and password.
  2. At the top left of the screen, select Change Management from the BIG-IQ menu.
  3. Under SNAPSHOT & RESTORE, select the service containing the snapshots to compare: Network Security or Web Application Security.
    The screen displays a list of snapshots that have been created on this device.
  4. Select the check box to the left of each of the two snapshots to be compared.
  5. Click Compare.
    The Compare Snapshots screen displays.
  6. For the Target, select the snapshot to which you want to compare the snapshot listed as the Source.
  7. Compare the snapshots selected:
    • To compare firewall object differences, click Compare in the Compare Firewall row. This option is only available with the AFM™ service.
    • To compare ASM differences, click Compare in the Compare ASM row. This option is only available with the ASM™ service.
    • To compare shared security object differences, click Compare in the Compare Shared Security row.
  8. Analyze the configuration differences between the two snapshots, When you are finished, click Cancel to close the Differences screen, then click Close.
    The screen closes and you return to the Snapshot and Restore - screen.

Restoring a snapshot

You can restore a snapshot to change the working configuration to that of the snapshot. Restoring the snapshot merges objects from the snapshot into the BIG-IQ® configuration and removes all active locks. No objects in the BIG-IQ configuration are removed. Once the restore process starts, you cannot modify the BIG-IQ configuration until the process is completed or canceled. If the process is canceled, all configuration settings are rolled back.

  1. Log in to F5® BIG-IQ® Centralized Management with your user name and password.
  2. At the top left of the screen, select Change Management from the BIG-IQ menu.
  3. Under SNAPSHOT & RESTORE, select the service containing the snapshot to restore: Network Security or Web Application Security.
    The screen displays a list of snapshots that have been created on this device.
  4. Select the check box to the left of the snapshot to use to restore the current working configuration to the configuration of the snapshot.
  5. Click Restore.
    The Restore snapshot to Working Configuration screen opens.
  6. Compare the snapshot to restore to the working configuration:
    • To compare firewall object differences, click Compare in the Compare Firewall row. This option is only available with the AFM™ service.
    • To compare ASM™ differences, click Compare in the Compare ASM row. This option is only available with the ASM service.
    • To compare shared security object differences, click Compare in the Compare Shared Security row.
    The differences screen for the comparison is displayed for you to review. Click Cancel when done.
  7. Click Restore to restore the configuration in the snapshot and have it replace the working configuration.
  8. Click Restore in the popup screen to confirm that you want to restore the configuration, or click Cancel in the popup screen to stop the restore process for this the snapshot.
    You can also click Cancel after starting the restore process to roll back the restore.