Gathering VMWare Host Data

Today’s post talks about the configuration and setup for sending and receiving VMWare ESX host logs and system events into Honeycomb Lexicon.

As VMWare host systems become ever more pervasive across business network estates and data centres, it becomes increasingly essential to ensure the host is running smoothly. Awareness of problems and system issues is crucial for these mission-critical systems.

Honeycomb Lexicon includes a VMWare LexApp application that contains the Knowledge Sets, Dashboards, searches, reports etc. to monitor the health of your ESX servers. All that is required is to get VMWare servers forwarding data into Lexicon.

 

Important Lexicon Setup Information

VMWare ESX syslog messages use a standard Linux-style format:

<pri>month day time service: message

This format contains no provision to uniquely identify the message as coming from an ESX server (as opposed to, say, a generic Linux log file which uses the same format). As such, Lexicon (or any syslog server for that matter) has no intrinsic way of determing the event source type as VMWare.

The good news is that Lexicon has a number of configuration options that do allow proper sourcetype detection for generic/standardized message formats. These options are contained within the Lexicon server LexApp’s sourcetypes.conf file installed on the Lexicon server.

To correctly identify VMWare syslog messages as a VMWare type, follow these simple steps:

1  On the Lexicon server that will receive VMWare events, open this file in a text editor:

<Honeycomb install folder>/lexicon/apps/vmware/cfg/sourcetypes.conf

2  Find the line that starts like this:

vmware:syslogvalidators=    (typically about halfway down the file)

For the VMWare LexApp, the default value for this property is:

*:udp:1515                                (this means all incoming traffic on udp port 1515)

3  Edit this value to match the VMWare ESX server(s) syslog configuration. Multiple servers are entered as a comma-delimited list. For example:

vmware:syslogvalidators=192.168.15.12:udp:514,10.0.1.14:udp:515,10.0.1.19:udp:515

It’s important to be careful about using the wildcard (*) value for generic/standard syslog messages, as this means ALL incoming traffic on the given port will be interpreted as VMWare data. This is why the LexApp’s default settings are not set to the standard 514 port – otherwise many other incoming events would be mis-categorized.

Also, as a general rule, using the IP address here tends to work better than a DNS name, as often the ESX servers don’t have access to the same DNS resolution as the Lexicon server(s) receiving the syslog messages.

4  Once you’ve made the changes to reflect your VMWare configuration, save the file, restart the Lexicon service, and your system is ready to go!

Additional Note: There is also a corresponding option in sourcetypes.conf for file validation, in case you are importing log files directly into Lexicon. The VMWare LexApp sourcetypes.conf file has an example of how to use this option.

 

VMWare Syslog Setup

This section steps through the procedure for enabling syslog messages to be sent by your ESX server(s). ESX/ESXi hosts run a syslogd service that can forward messages from VMKernel and other ESX system components.

Here are the steps:

1   Modify the /etc/syslog.conf file to capture and forward events:

Login to the ESX host as root or equivalent using and SSH client (e.g. PuTTY)

Open the /etc/syslog.conf file using a text editor

Add the following line to the end of the file:

*.*   @<IPAddress of your Lexicon server>

For example:

*.*   @192.168.15.10

Important: Note the white space between *.* and the @ symbol is a TAB, not spaces.

Save the file and exit your text editor

2   Restart the syslogd service with this command:

service syslog restart

3   It’s important now to check the ESX firewall is configured to allow outbound syslog traffic, so run this command to check the firewall’s configuration:

esxcfg-firewall -q|grep syslog

If the firewall is configured to allow syslog, you should see output similar to this:

syslog              : port 514 udp.out

If you don’t see any output, outbound syslog traffic is blocked, and needs to be enabled, so run this command to enable outbound syslog traffic:

esxcfg-firewall -o 514,udp,out,syslog && esxcfg-firewall –l

You can run the firewall check command above again to ensure the firewall is configured correctly.

 

Your ESX host is now configured to send its logs and system messages to your Lexicon server.

Honeycomb Lexicon, by default, monitors incoming traffic on UDP port 514, so VMWare data will be automatically indexed as it arrives from your ESX hosts.

You can check incoming VMWare traffic in Lexicon by Launching the Lexicon Browser (meshreporter.exe), and loading the VMWare Dashboard.

Tags: , , , , ,