Honeycomb Lexicon Pre-Installation Guidelines

This post outlines some useful guidelines and best practices prior to installing the Honeycomb Lexicon® and mesh® services.

 

Windows Event Logs

The recommended settings for all in-scope Windows servers’ Event Logs should be set as follows:

Application Log          32768KB     Overwrite events as needed

Security Log              196608KB    Overwrite events as needed

System Log               32768KB     Overwrite events as needed

Other Logs                Default settings

Note that no pre-v6 Windows machines (e.g. Server 2003, XP etc.) should have Event Log sizes with a combined total exceeding 300MB. The above settings are optimal for log collection and performance. Whilst changing these settings is a server configuration change, it has no adverse impact on system performance, as long as there is at least 256MB of system disk drive space to house the Event Log database.

Note that for Windows Server 2008/Win7, the 300MB limit is not relevant, as the new Crimson event log mechanism in these Windows products is completely redesigned. The above settings, however, should still be applied to Event Logs on monitored Windows v6+ products (Server 2008, Windows 7) to ensure there is sufficient capacity.

Using Active Directory Group Policy is the most efficient way to set Event Log sizes.

 

Disk Storage

Collecting data from numerous machines for centralized/distributed storage can involve a significant disk storage requirement.

Here are some points that are worth considering when determining the data collection and disk requirements:

  • Windows Event Logs
    • Monitor the Security Event Log on your Domain Controller(s), and note the number of records and time range stored (using the Event Log sizing below). This will give a rough indication of the amount of data that will be collected from Windows Event Logs across the network (DCs are a good choice, because they tend to be the ‘chattiest’).
  • Note that Windows 2008/7 generate much more verbose Event Log events than in previous versions. This places additional storage requirements on attached Lexicon® storage. If you run a 2008+ Active Directory Domain, there are some mechanisms available to help bring down the size of generated events.
  • File Integrity Monitoring
    • It is a worthwhile exercise to target File Integrity Monitoring to include only those files/areas that you really want/need to monitor/capture. It’s easy to say ‘monitor everything’, and the system will faithfully gather all data – however, note this will significantly increase the resources needed to store such data. mesh® File Integrity Monitor Policies can be tailored to each individual server if needed, to ensure efficient collection and storage.
  • Network device syslog monitoring
    • Network devices such as firewalls, VPNs, switches etc. should have their syslog data feed filtered to emit only the data that is required to be monitored. Network devices, left unfiltered can generate very large amounts of data.

 

Agentless Remote Event Log Monitoring

Ideally, a separate machine should be used for the Agentless Windows Event Log collection. Any reasonable specification machine will suffice for this purpose.

If Real-Time mode File Integrity Monitoring is targeted for the same machines as Windows Event Log Monitoring, some, most or all Windows Event Log monitoring can be done locally on each server, which would reduce or preclude the need for agentless monitoring. This can reduce the usage of WMI resources on your Windows systems, and also allows for detailed filtering of Wdinows Event Log traffic.

 

Reporting

If many/large reports are configured/run, it can be a good idea to configure these reports to run on separate hardware. If reports are configured that search across and/or produce very large data sets, this can require significant resources from a machine.

LexReporter can be distributed across multiple machines. Refer tot he LexReporter documentation included with Lexicon®.

 

Distributed Options

Honeycomb Lexicon® has very flexible options for distributing data collection, storage, search and reporting across multiple devices, thus allowing these operations to occur ‘closer’ to where it is generated and/or needed, whilst minimizing hardware requirements.

 

Data Backup

Lexicon® includes both and automated and on-demand snapshot capability that can be used for backing up data. Alternatively, a replication server can be setup, with backups taken from the replica.
 

Network Devices

Devices such as firewalls, web filters, VPNs etc. can easily be configured to send their syslog data directly to Lexicon®.

It’s worth spending a bit of time configuring these devices to send only the data required, so as not to fill up disk storage with unnecessary events.

 

Task Checklist

The following is a handy list of task items that can be performed prior to installation, which can prove helpful:

  • Decide which Windows server Event Logs need to be monitored, noting their physical location
  • Decide which NTFS volumes need to be monitored
  • Of the required NTFS volumes to be monitored, decide which data areas should be monitored for File Integrity Monitoring, the type of monitoring (Real-Time, Scanner, Access Monitoring), and which should be excluded (note that monitoring file access opens generates the most amount of traffic)
  • Decide which, if any, Microsoft Exchange servers should have Message Tracking monitored. Ensure any required servers have Message Tracking enabled

Note: When enabling Message Tracking on Exchange servers for the first time, test the systems thoroughly to ensure the server hardware can handle the extra load placed on it by Message Tracking – particularly at peak times 

  • Decide which network devices are to be monitored
  • Configure all monitored network devices to only send the data required via syslog
  • Decide which custom applications/log files need to be monitored, noting their location(s), names etc.
  • Gather a list of email recipients to be used for reporting/alerting etc.
  • Decide on disk share location(s) for generated reports
  • Decide which events/types/searches are needed for alerting, including the type of action to take (email recipients, scripts, programs etc.)
  • Decide what access control mechanisms are required for your site – e.g. who has access to the mesh®/Lexicon® consoles, etc.
  • Decide how long you wish to hold ‘live’ data within Lexicon® storage, and provision disk storage accordingly
  • Provision the required hardware for running Lexicon® server(s), including installation of the operating system and any required updates/patches/roles/accounts. In an Active Directory environment, the Lexicon® server should be a member of the Active Directory domain for which it will primarily monitor
  • Have ready, your organization’s backup strategy to capture automated snapshots to copy to offline/long term storage. It is a good idea to take regular backups of the configured snapshot folder, and to remove live index areas from your backup’s configuration (these will have many in-use/locked files, and won’t produce a reliable backup – the snapshots folder is used for this purpose)
  • Make sure you exempt Honeycomb Lexicon® index folders from anti-virus/anti-malware etc. configurations, as these can have an adverse impact on performance (the folders to exempt will look something like: D:/Honeycomb/lexicon and all subfolders)
  • Create a dedicated (AD) Admin user account for use by mesh® services to access remote resources (e.g. for Agentless Event Log monitoring). Typically, this is a Domain Admin account with an extremely strong password
  • The software installation of mesh® and Lexicon® will require a Domain Admin account to perform the installation
  • For segmented LANs, DMZs etc., ensure any required firewall rules/ports are enabled to allow mesh®/Lexicon® traffic to flow (e.g. syslog, WMI, DCOM traffic) mesh® uses rmi:tcp:8238 by default for its communications. Lexicon® uses http:8983 by default for search communications. The most common syslog port is udp:514 (but you can and often do configure multiple ports if there are many syslog devices)

Tags: , , , , ,