Manual Chapter : Reporting Usage Data to an External Analytics Server

Applies To:

Show Versions Show Versions

BIG-IP PEM

  • 11.5.10, 11.5.9, 11.5.8, 11.5.7, 11.5.6, 11.5.5, 11.5.4, 11.5.3, 11.5.2, 11.5.1
Manual Chapter

Overview: Reporting usage data to an external analytics server

In Policy Enforcement Manager, you can create a rule within an enforcement policy that instructs the system to send usage data in high-speed logging (HSL) format to an external analytics server. The rule specifies what type of reporting data you are interested in; one of the actions it can take with the traffic is to send the information collected about it for processing to a centralized analytics server.

The system sends the information as a set of comma-separated values by means of SYSLOG transport. You can choose to use either the session-based or flow-based reporting format, depending on the level of granularity you need.

For example, a rule might collect session-based information about all audio and video traffic. You can specify how often to log the data and set the destination as an HSL server or pool.

Task summary

Creating a publisher

Ensure that at least one destination associated with a pool of remote log servers exists on the BIG-IP system.
Create a publisher to specify where the BIG-IP system sends log messages for specific resources.
  1. On the Main tab, click System > Logs > Configuration > Log Publishers. The Log Publishers screen opens.
  2. Click Create.
  3. In the Name field, type a unique, identifiable name for this publisher.
  4. For the Destinations setting, select a destination from the Available list, and click << to move the destination to the Selected list.
    Note: If you are using a formatted destination, select the destination that matches your log servers, such as Remote Syslog, Splunk, or ArcSight.
  5. Click Finished.

Creating a rule for high-speed logging

Before you can create a high-speed logging (HSL) rule, you need to create a publisher that defines the destination server or pool where the HSL logs are sent.
In an enforcement policy, a rule can specify that statistics about traffic affected by the rule are sent to an external high-speed logging server.
  1. On the Main tab, click Policy Enforcement > Policies. The Policies screen opens.
  2. Click the name of the enforcement policy you want to add rules to. The properties screen for the policy opens.
  3. In the Policy Rules area, click Add. The New Rule screen opens.
  4. In the Name field, type a name for the rule.
  5. In the Precedence field, type an integer that indicates the precedence for the rule in relation to the other rules. Number 1 has the highest precedence. Rules with higher precedence are evaluated before other rules with lower precedence.
    Tip: All rules in policy are run concurrently. Precedence takes effect when there are conflicting rules. The conflict occurs when the traffic matches two rules and the policy actions from these rules differ. For example, if you have a rule 1 with precedence 10 with Gate Status disabled for a search engine and you have rule 2 with precedence 11 with Gate Status enabled, then rule 1 is processed first because it has higher precedence. Rules conflict if they have identical or overlapping classification criteria (for the traffic that matches more than one rule).
  6. Use the Classification, URL, Flow, and Custom Criteria tabs to identify the traffic that you want to be affected by this rule. Other tasks describe how to do this in detail.
  7. From the Reporting list, select Enabled.
  8. From the Report Granularity list, select the appropriate option:
    • To log details about subscribers and application sessions, select Session.
    • For more granular reporting of every TCP connection, select Flow.
  9. In the Volume Threshold setting, specify in octets, the threshold to send HSL reporting records. You can send reporting data from uplink traffic, to downlink traffic and the total traffic volume before logging the information.
  10. In the Destination setting, specify where to send the usage monitoring data:
    • In the Gx field select Enabled for the BIG-IP system to send usage monitoring data over a Gx interface. You can then type a string for the Gx Monitoring Key that is used for usage monitoring.
      Note: When you select Session in the Report Granularity field, the Gx field appears.
    • From the HSL list, select the name of the publisher that specifies the server or pool of remote HSL servers to send the logs.
    Note: If you are using a formatted destination, select the publisher that matches your log servers, such as Remote Syslog, Splunk, or ArcSight.
  11. In the Interval field, type an integer that specifies how frequently HSL reporting data is sent.
  12. Click Finished.
You have created a rule that sends data about the traffic to external high-speed logging servers. The CSV reporting format differs depending on whether the report granularity is flow-based or session-based.

Session-based reporting format

In an enforcement policy, a rule can send session-based information about traffic that matches certain criteria to an external high-speed logging (HSL) server. The logs include the following comma-separated values in the order listed.

Field Description
PEM id Identifies the reporting module (PEM) and the field value is 23003143.
Timestamp seconds The time the information was logged (along with the timestamp in milliseconds), specifies seconds using UNIX time format.
Timestamp msec The time the information was logged (along with the timestamp in seconds), specifies milliseconds using UNIX time format.
Report type The type of report. Always set to 3 for session-based reporting.
Subscriber ID A unique identifier (up to 64 characters) for the subscriber initiating the session, such as a phone number. The subscriber ID type determines the format.
Subscriber ID type The format of the subscriber ID. It can be E.164, IMSI, NAI, or Private.
3GPP parameters The list of 3GPP parameters, which can be imsi, imeisv, tower_id, or username.
Application ID A unique number that represents a particular application, and is used for classifying traffic.
Last Sent The time, in seconds, since the last log entry was sent.
Bytes in The number of bytes received during this session.
Bytes out The number of bytes sent during this session.
Concurrent flows Always 0 (unsupported).
Opened flows Always 0 (unsupported).
Terminated flows Always 0 (unsupported).
Total transactions Always 0 (unsupported).
Successful transactions Always 0 (unsupported).
Aggregated category duration Summary of the duration of all flows for the session.
Reason The reason for sending the record. It can be 0 - reserved, 1 - volume threshold reached, 2- interval time, 3 - subscriber logout, or 4 - inactivity.

Example session-based reporting format

Oct 10 17:19:45 172.31.63.64 23003143,1349914925,546879,3,404234567123456,IMSI,linux,f501, 404234567123456,35827001,16394,1349914913,5469633,308908379, 0,0,0,0,0,5052,1 Oct 10 17:19:57 172.31.63.64 23003143,1349914937,546661,3,404234567123456,IMSI,linux,f501, 404234567123456,35827001,16394,1349914925,5550857,313317479, 0,0,0,0,0,5063,1 Oct 10 17:20:09 172.31.63.64 23003143,1349914949,546676,3,404234567123456,IMSI,linux,f501, 404234567123456,35827001,16394,1349914937,5636605,318053179, 0,0,0,0,0,5074,1

Flow-based reporting format

In an enforcement policy, a rule can send flow-based information about traffic that matches certain criteria to an external high-speed logging (HSL) server. The logs include the following comma-separated values in the order listed.

Field Description
PEM id Identifies the reporting module (PEM) and the field value is 2300314.
Timestamp seconds The time the information was logged in UNIX time format.
Timestamp msec The msecs time value of the timestamp in UNIX time format.
Report type The type of report; 0 – flow start, 1 – flow interim, 2 – flow end.
Subscriber ID A unique identifier (up to 64 characters) for the subscriber initiating the session, such as a phone number. The subscriber ID type determines the format.
Subscriber ID type The format of the subscriber ID. It can be E.164, IMSI, NAI, or Private.
Source IP address The source IP of the subscriber.
Source port The source port the subscriber.
Destination IP address The destination IP of the traffic.
Destination port The destination port for the traffic.
Protocol The protocol of the traffic for this flow, TCP or UDP.
Application ID A unique number that represents a particular application in this flow; it is used for classifying traffic.
Urlcat ID The URL category id that the flow belongs to.
Flow start time seconds The time, in seconds, the flow started in UNIX time format.
Flow start time msecs The time in milliseconds of the flow start time.
Flow end time seconds The time the flow ended in UNIX time format.
Flow end time msecs The time in milliseconds of the flow end time.
Transactions count The count of full transactions seen in the flow.
Bytes in The number of bytes received during this flow.
Bytes out The number of bytes sent during this flow.

Example flow-based reporting format

Sep 13 13:48:58 172.31.63.60 23003143,1347546777,654398,0,4086007577,E164,2001::10,52784,2001::2,80,6, 67,1347546774,628630,4278124286,4278124286,331,156 Sep 13 13:48:58 172.31.63.60 23003143,1347546777,654398,2,4086007577,E164,2001::10,52784,2001::2,80,6, 67,1347546774,628630,1347546775,382473,547,864