Manual Chapter : Managing a Data Collection Device Cluster

Applies To:

Show Versions Show Versions

BIG-IQ Centralized Management

  • 5.2.0
Manual Chapter

Data collection device best practices

There are a number of useful concepts to consider when you manage data collection devices for off-box log storage. This reference material might prove helpful in setting up and maintaining your data collection device (DCD) configuration.

Important: As part of maintaining a DCD cluster, you might need to remove one or more devices from your DCD cluster. When you remove a DCD from the cluster, BIG-IQ® Centralized Management moves the data to another device in the cluster. Whenever you move data, losing part or all of that data is a risk. Therefore, before you remove a DCD from the cluster, F5 recommends creating a snapshot to back up your logging data.

Restore data collection device snapshots

You can use the BIG-IQ® user interface to restore data collection device (DCD) snapshots.

Please note:

  • The restore operation requires a down time during which no BIG-IQ or DCD work is performed.
  • During the restore operation, no data sent to the DCD is retained.
  • The restore operation restores only the data from the time before the chosen snapshot was created. Data from the time that the chosen snapshot was created to the current time is not restored.
  • Before initiating a snapshot restore, make sure that sufficient disk space is allocated to the /var folder on the device to which you are restoring the snapshot.
  1. At the top of the screen, click System.
  2. On the left, expand BIG-IQ DATA COLLECTION and then select BIG-IQ Data Collection Devices.
    The BIG-IQ Data Collection Devices screen opens to display the currently defined data collection device cluster.
  3. Click the Settings button.
    The Settings screen opens to display the current state of the DCD cluster defined for this BIG-IQ device.
  4. You have two options for choosing a snapshot and starting the restore, using the settings in the External Storage & Snapshot area near the bottom of the screen.
    Option Description
    To restore from the most recent snapshot: Next to Last Snapshot/Time, click Restore Latest.
    To select the snapshot that you want to restore:
    1. Click the View History button.
    2. Choose the snapshot you wish to restore, and click Restore.

Delete a data collection device snapshot

If you determine that there are issues with a specific snapshot, you can delete it so that you cannot accidentally restore to it in the future.

Note: You perform this task on the BIG-IQ® Centralized Management device; not on the data collection device (DCD).
  1. At the top of the screen, click System.
  2. On the left, expand BIG-IQ DATA COLLECTION and then select BIG-IQ Data Collection Devices.
    The BIG-IQ Data Collection Devices screen opens to display the currently defined data collection device cluster.
  3. Click the Settings button.
    The Settings screen opens to display the current state of the DCD cluster defined for this BIG-IQ device.
  4. Near the bottom of the screen, click the View History button.
    The BIG-IQ Data Collection Snapshots screen opens.
  5. Browse through the list to find the snapshot you want to delete.
  6. Select the check box for the snapshot you want to delete, and click Delete.

Check data collection device health

You can use the BIG-IQ® Data Collection Device Settings screen to review the overall health and status of the data collection devices you've configured. You can use the data displayed on this screen both before and after an upgrade to verify that your DCD cluster configuration is as you expect it to be.
Note: You perform this task on the BIG-IQ Centralized Management device; not on the data collection device (DCD).
  1. At the top of the screen, click System.
  2. On the left, expand BIG-IQ DATA COLLECTION and then select BIG-IQ Data Collection Devices.
    The BIG-IQ Data Collection Devices screen opens to display the currently defined data collection device cluster.
  3. Click the Settings button.
    The Settings screen opens to display the current state of the DCD cluster defined for this BIG-IQ device.
  4. Inspect the DCD cluster details listed in the Summary area.
    Data Cluster Status
    Look here for information about the current state of the cluster.
    Master Device

    The read-only Master Device field displays the host name of the BIG-IQ device that manages and monitors the health of this DCD cluster.

    Devices in Cluster
    Displays the total number of devices in the cluster including DCDs, the BIG-IQ Centralized Management devices and the optional peer.
    Data Collection Devices in Cluster
    Displays the number of DCDs that have been added to the cluster.
    Total Document Count
    Displays the number of all document types stored on the cluster. Alerts and events are included in this list, but the total includes other types of document as well.
    Total Document Size
    Displays the amount of disk space consumed by the documents stored for this cluster.
    This information provides a fairly detailed overview that describes the DCD cluster you have created to store data. After you complete an upgrade, you can check the health to verify that the cluster restored successfully.

Index rotation policy

The optimum settings used to configure your data collection device (DCD) indices depend on a number of key factors.

  • The system provides the ability to dynamically create new indices based on either a specified interval or a specified size. The primary goal to consider when you make these decisions is how to maintain a maximum disk allocation for the DCD data, while maintaining capacity for new data that flows in.
  • Secondary considerations include search optimization, and the ability to optimize old indices to reduce their size.
  • Generally, the best policy is one that does not create unnecessary indices. The more indices, the lower the overall performance, because your searches have to deal with more shards. For example, if a module knows that it has a low indexing volume (thousands/day) then it makes the most sense to have a large aggregation per rotation (5 days or 30 days). For components like Web Application Security that probably have high indexing volumes, it makes more sense to rotate every 8 hours (which reduces the number of retained indices).
  • Index rotation also allows changing sharding and replica counts by changing the template on a given index type. New indices created from that template will contain the new shard and replica count properties.

This table shows the default configuration values for each index running on BIG-IQ® Centralized Management. These values are based on anticipated data ingestion rates and typical usage patterns.

Component Index Name Minimum Number of DCDs Rotation Policy Retained Index Count Approximate time window Size of /var file system
Access access-event-logs 2 Time/5 days 19 95 days 500 GB
Access access-stats 2 Time/5 days 19 95 days 500 GB
Web Application Security asmindex 2 Size/100000 MB 5 N/A 500 GB
FPS websafe 2 Time/30 days 100 8 years 10 GB

If multiple modules are running on a given DCD or if you have higher inbound data rates, you might have to adjust these values to keep the /var file system from filling up. (There is a default alert to warn of this when the file system becomes 80% full.)

The simplest resolution is to revise the retained index count; lowering this value reduces the disk space requirements, but it will also reduce the amount of data available for queries. For details on changing this setting, refer to the modifying indices topic for the component you are configuring.