Applies To:

Show Versions Show Versions

Manual Chapter: BIG-IP® Network and System Management Guide: 17 - Logging BIG-IP System Events
Manual Chapter
Table of Contents   |   << Previous Chapter   |   Next Chapter >>


17

Logging BIG-IP System Events


Introducing BIG-IP system logging

Viewing and managing log messages is an important part of maintaining a BIG-IP system. Log messages inform you on a regular basis of the events that are happening on the system. Some of these events pertain to general events happening within the operating system, while other events are specific to the BIG-IP system, such as the stopping and starting of BIG-IP system services.

The mechanism that the BIG-IP system uses to log events is the Linux utility syslog-ng. The syslog-ng utility is an enhanced version of the standard UNIX and Linux logging utility syslog.

The types of events that the BIG-IP system logs are:

  • System events
    System event messages are based on Linux events, and are not specific to the BIG-IP system.
  • Packet filter events
    Packet filter messages are those that result from the implementation of packet filters and packet-filter rules.
  • Local traffic events
    Local-traffic event messages pertain specifically to the local traffic management system.
  • Audit events
    Audit event messages are those that the BIG-IP system logs as a result of changes to the BIG-IP system configuration. Logging audit events is optional.

Summarizing logging features

The logging mechanism on a BIG-IP system includes several features designed to keep you informed of system events in the most effective way possible.

One of the primary features of the logging feature is its ability to log different types of events, ranging from Linux system events, to packet filtering events, to local traffic events. Through the BIG-IP system auditing feature, you can even track and report changes that users make to the BIG-IP system configuration, such as adding a virtual server or designating a device to be part of a redundant system. For more information, see Understanding log content and Understanding log types .

When setting up logging on the BIG-IP system, you can customize the logs by designating the minimum severity level, or log level, that you want the BIG-IP system to report when a type of event occurs. The minimum log level indicates the minimum severity level at which the BIG-IP system logs that type of event.

For example, you can specify that, for any change a user makes to the bigdbTM database, the minimum severity level for which the BIG-IP system logs messages is Warning. This means that the BIG-IP system logs Warning and more severe messages such as Error and Critical messages, but not less severe ones such as Notice, Informational, or Debug messages. For more information, see Setting log levels .

You can also use the Configuration utility to search for a string within a log event, that is, filter the display of the log messages according to the string you provide. For more information, see Viewing and filtering log messages .

Finally, you can log BIG-IP system events to a remote logging server. You do this by identifying the IP address or host name of the remote logging server, and creating an encrypted network connection, or tunnel, for sending log information to that remote server. For more information, see Configuring encrypted remote logging .

Tip


You can also configure the system to send email or activate pager notification based on the priority of the logged event.

Understanding log content

The logs that the BIG-IP system generates include several types of information. For example, all logs except the audit log show a timestamp, host name, and service for each event. Some logs show a status code, while the audit log shows a user name and a transaction ID corresponding to each configuration change. All logs contain a 1-line description of each event.

Table 17.1 lists the categories of information contained in the logs and the specific logs in which the information is displayed.

Table 17.1 Log information categories and their descriptions
Information Type
Explanation
Log Type
Timestamp
The time and date that the system logged the event message.
System
Packet Filter
Local Traffic
Host name
The host name of the system that logged the event message. Because this is typically the host name of the local machine, the appearance of a remote host name could be of interest.
System
Packet Filter
Local Traffic
Service
The service that generated the event.
System
Packet Filter
Local Traffic
Status code
The status code associated with the event. Note that only events logged by BIG-IP system components, and not Linux system services, have status codes.
Packet Filter
Local Traffic
Description
The description of the event that caused the system to log the message.
System
Packet Filter
Local Traffic
User Name
The name of the user who made the configuration change.
Audit
Transaction
The identification number of the configuration change.
Audit
Event
A description of the configuration change that caused the system to log the message.
Audit

 

Viewing and filtering log messages

Use the Configuration utility for an easy way to view the log files that the system generates. You can also control which messages the Configuration utility displays, by typing a search string.

To view log messages

  1. On the Main tab of the navigation pane, expand System, and click Logs.
    The Logs screen opens.
  2. On the menu bar, click System, Packet Filter, Local Traffic, or Audit, depending on the type of log messages you want to view.
    This displays a list of this type of log message.
  3. If you want to advance to another screen of messages, first locate the page list at the lower-right corner of the screen. You can either:
    • Display the list and select a page number or Show All.
    • Click the right arrow to advance to the next page of messages.

To filter log messages based on a search string

  1. On the Main tab of the navigation pane, expand System, and click Logs.
    The Logs screen opens.
  2. On the menu bar, click System, Packet Filter, Local Traffic, or Audit, depending on the type of log messages you want to view.
    This displays log messages of the type you selected.
  3. In the Search box (directly above the Timestamp column), type a string, optionally using the asterisk as a wildcard character.
  4. Click Search.
    This displays only those messages containing the string you specified.

Understanding log types

As described in Introducing BIG-IP system logging , the BIG-IP system automatically logs four main event types: system, packet filter, local traffic, and configuration changes (audit). Each type of event is stored in a separate log file, and the information stored in each log file varies depending on the event type. All log files for these event types are in the directory /var/log.

Logging system events

Many events that occur on the BIG-IP system are Linux-related events, and do not specifically apply to the BIG-IP system. The BIG-IP system logs the messages for these events in the file /var/log/messages.

Using the Configuration utility, you can display these system messages. Table 17.2 shows some sample system log entries.

Table 17.2 Sample system log entries
Timestamp
Host
Service
Event
Mon Feb 14 03:34:45 PST 2005
bigip3
syslog-ng[5494]
new configuration initialized
Mon Feb 14 03:35:06 PST 2005
bigip3
syslog-ng[5494]
kjournald starting. Commit interval 5 seconds.
Mon Feb 14 04:38:06 PST 2005
bigip3
EXT3-fs
mounted filesystem with ordered data mode.

 

Logging packet filter events

Some of the events that the BIG-IP system logs are related to packet filtering. The system logs the messages for these events in the file /var/log/pktfilter.

Using the Configuration utility, you can display these packet filter messages.

Logging local traffic events

Many of the events that the BIG-IP system logs are related to local area traffic passing through the BIG-IP system. The BIG-IP system logs the messages for these events in the file /var/log/ltm.

Using the Configuration utility, you can display these local-traffic messages. Table 17.3 shows some sample local-traffic log entries.

Table 17.3 Sample local-traffic log entries
Timestamp
Host
Service
Status Code
Event
Mon Feb 14 03:34:45 PST 2005
bigip2
bcm56xxd(785)
00010013
Starting packet registry event timer
Mon Feb 14 03:35:06 PST 2005
bigip2
bcm56xxd(785)
00010013
Starting HA heartbeat timer tick
Mon Feb 14 04:38:06 PST 2005
bigip2
bcm56xxd(785)
00010013
Successful start. Entering main message loop
Mon Feb 14 o4:36:06 PST 2005
bigip2
bcm56xxd(785)
00010012
Link 2.5 is up

 

Some of the specific types of events that the BIG-IP system displays on the Local Traffic logging screen are:

  • Address Resolution Protocol (ARP) packet and ARP cache events
  • bigdbTM database events (such as populating and persisting bigdb variables)
  • HTTP protocol events
  • HTTP compression events
  • IP packet discard events due to exceptional circumstances or invalid parameters (such as a bad checksum)
  • Layer 4 events (events related to TCP, UDP, and Fast L4 processing)
  • MCP/TMM configuration events
  • Monitor configuration events
  • Network events (layers 1 and 2)
  • Packet Velocity® ASIC (PVA) configuration events
  • iRuleTM events related to run-time iRule processing
  • SSL traffic processing events
  • General TMM events such as TMM startup and shutdown
Note

For information on setting a minimum log level on each of these event types, see Setting log levels for local traffic events , and Appendix B, Configuring bigdb Database Keys .

Auditing configuration changes

Audit logging is an optional feature that logs messages whenever a BIG-IP system object, such as a virtual server or a load balancing pool, is configured; that is, created, modified, or deleted. There are three ways that objects can be configured:

  • By user action
  • By system action
  • By loading configuration data

The BIG-IP system logs the messages for these events in the file /var/log/ltm.

Using the Configuration utility, you can display audit log messages. Table 17.4 shows some sample audit log entries. In this example, the first entry shows that user Janet enabled the audit logging feature, while the second and third entries show that user Matt designated the BIG-IP system to be a redundant system with a unit ID of 1.

Table 17.4 Sample audit log entries
Timestamp
User Name
Transaction
Event
Mon Feb 14 03:34:45 PST 2005
janet
79255-1
DB_VARIABLE modified:
name="config.auditing"
Mon Feb 14 03:35:06 PST 2005
matt
79609-1
DB_VARIABLE modified:
name="failover.isredundant"
value="true"
Mon Feb 14 03:35:06 PST 2005
matt
79617-1
DB_VARIABLE modified:
name="failover.unitid"
value="1"

 

By default, audit logging is disabled. For information on enabling this feature, see Setting log levels , following.

Setting log levels

Using the Configuration utility, you can set log levels on both local traffic and auditing events. For each type of local traffic event, you can set a minimum log level. The minimum log level indicates the minimum severity level at which the BIG-IP system logs that type of event. For more information, see Setting log levels for local traffic events , following.

For auditing events, you can set a log level that indicates the type of event that the system logs, such as the user-initiated loading of BIG-IP system configurations, or system-initiated configuration changes. For more information, see Setting log levels for auditing events .

Setting log levels for local traffic events

For local traffic events, you can set a minimum log level. Thus, for different kinds of local traffic events, such as bigdb configuration events or events related to HTTP compression, you can set different minimum log levels.

The log levels that you can set on certain types of events, ordered from highest severity to lowest severity, are:

  • Emergency
  • Alert
  • Critical
  • Error
  • Warning
  • Notice
  • Informational
  • Debug

For example, if you set the minimum log level for bigdb events to Error, then the system only logs messages that have a severity of Error or higher for those events. If you retain the default minimum log level (Informational), then the system logs all messages that have a severity of Informational or higher (that is, all messages except Debug messages).

There are many different types of local traffic events for which you can set a minimum log level. Table 17.5 shows the types of local traffic events and the minimum log levels that you can configure for them. Because not all log levels are available for every local-traffic event type, the table shows the specific log levels you can set on each event type. Following the table is the procedure for setting the minimum log level on a local traffic event type.

Table 17.5 Local-traffic event types and their available log levels
Local-Traffic Event Type
Available Minimum Log Levels
Default Value
ARP/NDP
Error, Warning, Notice, Informational, Debug
Warning
BigDB
Critical, Error, Warning, Notice, Informational, Debug
Informational
HTTP
Error, Debug
Error
HTTP Compression
Error, Debug
Error
IP
Warning, Notice
Notice
iRules
Error, Informational, Debug
Informational
Layer 4
Notice, Informational, Debug
Notice
MCP
Emergency, Alert, Critical, Error, Warning, Notice, Informational, Debug
Notice
Monitors
Error, Debug
Error
Network
Critical, Error, Warning, Notice, Informational, Debug
Warning
Packet Velocity® ASIC
Informational, Debug
Informational
SSL
Error, Warning
Warning
Traffic Management OS
Emergency, Critical, Error, Notice, Informational
Error

 

To set a minimum log level for local traffic events

  1. On the Main tab of the navigation pane, expand System, and click Logs.
    This opens the Logs screen.
  2. On the menu bar, click Options.
    This displays the screen for setting minimum log levels on local traffic events.
  3. In the Local Traffic Logging area of the screen, locate the event type for which you want to set a minimum log level.
    An example of an event type is HTTP Compression.
  4. Select a minimum log level from the list.
  5. Click Update.
Note

For more information on local traffic event types, see Logging local traffic events . For information on using bigdb configuration keys to set minimum log levels, see Appendix B, Configuring bigdb Database Keys .

Setting log levels for auditing events

An optional type of logging that you can enable is audit logging. Audit logging logs messages that pertain to configuration changes that users or services make to the BIG-IP system configuration. (For more information, see Auditing configuration changes .)

You can choose one of four log levels for audit logging. In this case, the log levels do not affect the severity of the log messages; instead, they affect the initiator of the audit event.

The log levels for audit logging are:

  • Disable
    This turns audit logging off. This is the default value.
  • Enable
    This causes the system to log messages for user-initiated configuration changes only.
  • Verbose
    This causes the system to log messages for user-initiated configuration changes and any loading of configuration data.
  • Debug
    This causes the system to log messages for all user-initiated and system-initiated configuration changes.

To set a minimum log level for audit events

  1. On the Main tab of the navigation pane, expand System, and click Logs.
    This opens the Logs screen.
  2. On the menu bar, click Options.
    This displays the screen for setting minimum log levels on local traffic events.
  3. In the Audit Logging area of the screen, select a log level from the Audit list.
  4. Click Update.

Configuring encrypted remote logging

You can configure the Syslog utility on the BIG-IP system to send BIG-IP system log information to a remote logging host, using an encrypted network connection. To do this, you create a port-forwarding SSH tunnel to the remote logging host, and configure syslog-ng on the BIG-IP system to send log messages through the SSH tunnel.

Before you begin

Before you attempt to configure encrypted remote logging, you must meet the following conditions on the BIG-IP system and your remote logging host:

  • On the BIG-IP system
    You must have a console with root access to the BIG-IP system.
  • On the remote logging host
    You must have a console with root access to the remote logging host, the IP address, or the host name of the remote logging host.
  • For both systems
    You must have both systems connected to the same subnetwork.
Warning

You should attempt this configuration only if you understand the risks associated with making changes to service startup scripts.

Creating the remote encrypted logging configuration

When creating an encrypted remote logging configuration, you must complete the following tasks:

  • Review the SSH syntax required to create this configuration.
  • Create a unique SSH identity key to identify and authorize the BIG-IP system.
  • Edit the syslog-ng service startup script to create and destroy the SSH tunnels.
  • Edit the remote logging host to accept syslog-ng messages through the SSH tunnel.
  • Copy the unique SSH identity key to the remote logging host and append it to the authorized key file.
  • Verify the logging configuration and restart syslog-ng.

Reviewing the SSH syntax required to create this configuration

This configuration requires that the BIG-IP system is able to establish an SSH connection to the remote logging host. On the BIG-IP system, use the ssh command to create the tunnel. Figure 17.1 is an example of the syntax required to create an SSH tunnel.

Figure 17.1 Establish an SSH tunnel from the BIG-IP system to the logging host.
$ ssh -L <local tunnel port>:<remote log hostname>:<remote 
tunnel port> \
  <remote user>@<remote log hostname> \
  -nNCxf \
  -i <key identity file>

 

Table 17.6 contains detailed descriptions of the ssh syntax elements shown in Figure 17.1 .

Table 17.6 Detailed syntax elements for configuring SSH.
SSH syntax
Description
<local tunnel port>
The port SSH listens on for connections in order to forward them to <remote log hostname>:<remote tunnel port>.
<remote log hostname>
The IP address or FQDN of the remote logging server.
<remote tunnel port>
The port to which you want the SSH daemon on the remote logging server to forward connections.
<remote user>
The user name that ssh attempts to authenticate, as on <remote log hostname>.
<key identity file>
A file name from which the identity (private key) for authentication is read.

 

Creating a unique SSH key to identify and authorize the BIG-IP system

After you have reviewed the ssh command syntax, use the ssh command to create the encrypted tunnel on the BIG-IP system, you must create a unique key on the BIG-IP system. The unique key is used to identify and authorize the BIG-IP system to the remote logging host.

Use the following command to create the file syslog_tunnel_ID and syslog_tunnel_ID.pub.

$ ssh -b 2048 -f syslog_tunnel_ID -t rsa -N "" -P ""

Use the following command to make syslog_tunnel_ID readable only by the root account:

$ chmod 600 syslog_tunnel_ID

Use the following command to make the public portion of the unique SSH ID named syslog_tunnel_ID.pub readable by all accounts:

$ chmod 644 syslog_tunnel_ID.pub

Copy syslog_tunnel_ID and syslog_tunnel_ID.pub into /var/ssh with the following command:

$ cp syslog_tunnel_ID* /var/ssh

Editing the syslog-ng start script to open and close the encrypted tunnel

Next change the syslog-ng start script, /etc/init.d/syslog-ng, so that the encrypted tunnel is opened when the syslog-ng script starts up and is closed when the script is restarted or stopped.

Before you edit the syslog-ng start script, save a backup copy to the root directory. Use the following command to save the backup to the root directory:

$ cp /etc/init.d/syslog-ng /root/syslog-ng.backup

After you save a backup of the syslog-ng, edit the startup script /etc/init.d/syslog-ng to automatically create a SSH tunnels when syslog-ng is started, or closed when syslog-ng is restarted or stopped.

The example configuration in this document demonstrates how to create a tunnel to a host using the following IP addresses and ports:

  • IP address of 10.0.0.100
  • Local tunnel port of 5140
  • Remote tunnel port of 5140
  • User name logger on host 10.0.0.100.

Start by adding syntax below the line that reads start). Figure 17.2 is an example of what the section of the syslog-ng start script looks like after you add the new syntax. In this example, the syntax you need to add is shown with bold text).

Figure 17.2 The syntax to add below the start) line.
 start)
       ssh -L 5140:10.0.0.100:5140 \
       logger@10.0.0.100 -nNCxf \
       -i var/ssh/syslog_tunnel_ID
        echo -n "Starting $INIT_NAME: "
        daemon --check $INIT_PROG "$INIT_PROG $INIT_OPTS"

 

Next, add syntax below the line that reads stop). Figure 17.3 shows the syntax you need to add in bold text.

Figure 17.3 The syntax to add below the stop) line
 stop)
        for sshTunnel in \
           `ps -ewo "%p!%a" | \
            grep ssh | \
            grep syslog_tunnel_ID | \
            grep -v grep | \
            cut -f 1 -d !`; do
            if [ -n "$sshTunnel" -a $sshTunnel -gt 10 ]; then
                echo " -- Shutting down SSH tunnel with process $sshTunnel"
                kill -TERM $sshTunnel
            fi
        done
        echo -n "Stopping $INIT_NAME: "

 

Editing syslog-ng to log messages on the remote logging host

After you add the syntax to open and close SSH tunnels, you can edit the syslog-ng configuration to log messages to the remote machine. To do this, you need to create source and filter configuration blocks based on the local environment.

Using the example IP addresses and ports used in the example in the previous section, you would edit the syslog-ng.conf file to look like the syslog-ng.conf in Figure 17.4 .

 

Figure 17.4 The syslog-ng.conf example configuration
# capture all messages
filter f_catchall {
   level(debug...emerg);
};
# send message to localhost through tcp port 5140
destination d_remoteLogTunnel {
   tcp("127.0.0.1" port(5140););
};
# Combine everything to actually perform logging
log {
   source(local);
   filter(f_catchall);
   destination(d_remoteLogTunnel);
};

 

Copying the unique SSH identity to the remote logging host and appending it to the authorized keys file

After you have edited the syslog-ng.conf to log messages on the remote logging host, you must copy the unique SSH identity to the remote logging host. To do this, copy the syslog_tunnel_ID.pub to the remote syslog server, and append this key to the authorized_keys file found in the.ssh folder under the home directory of the user that you want to use to capture remote log messages.

$ cat syslog_tunnel_ID.pub >> ~logger/.ssh/authorized_keys

Note

The following instructions are given as examples. The actual process for setting up the new SSH key to be automatically authorized, and configuring the syslog service may be different.

Verify that the logging facility is configured and ready to receive syslog messages on the <remote tunnel port>. If the remote logging host uses syslog-ng, you need to add a source configuration block like the one in Figure 17.5 :

Figure 17.5 Remote logging host source identification block.
source remote {
   tcp(ip(10.0.0.100) port(5140));
};

 

In addition to the source identification block, you also need to add filter, destination, and log configuration blocks to use the data from the source remote as required by your application.

Verifying the logging configuration and restarting syslog-ng

Finally, verify that the SSH connection is functional and restart the syslog-ng service.

To verify the configuration from a command line and restart the syslog-ng service

  1. Log in as root to the BIG-IP system.
  2. Make an SSH connection to the remote logging host using the new identity key you created.
  3. # ssh logger@10.0.0.100 -i /var/shh/syslog_tunnel_ID

    If everything is configured correctly, you should be able to get shell access to the remote logging host without being challenged for a password. (By adding the new identity key to the remote host's authorized_keys file, the key is used to authenticate the BIG-IP system.)

  4. Exit from the SSH session to the BIG-IP system command line.
  5. Restart the syslog-ng service by typing the following command:
  6. $ /etc/init.d/syslog-ng restart

    The BIG-IP system should now be sending log messages to your remote host.




Table of Contents   |   << Previous Chapter   |   Next Chapter >>

Was this resource helpful in solving your issue?




NOTE: Please do not provide personal information.



Incorrect answer. Please try again: Please enter the words to the right: Please enter the numbers you hear:

Additional Comments (optional)