Troubleshooting

This section discusses common problems encountered by BMC Defender Agent for Windows administrators and developers, and how to troubleshoot and solve these problems.

The BMC Defender Agent for Windows is designed to be flexible and open, and cover a variety of different application areas. It is common to experience problems, especially when getting started. The range of the typical problems experienced is quite small and can address by the notes in this section.

Prior to opening a trouble ticket, you might want to review the list of issues in this section. In addition to assistance with solving problems, this section can be used to achieve greater insight into the workings of the BMC Defender Agent for Windows. After reviewing this section, if you still have questions or issues, contact BMC Support.

See also the BMC Defender Framework, that has a similar list of frequently asked questions and troubleshooting techniques, useful when working with the BMC Defender and framework components.

I am not receiving any syslog messages.

The most likely reason is that a firewall is preventing the transmission of the message to the destination host. You need to open a hole in the firewall to permit UDP port 514 to be accessible at the remote machine. This is the most common problem with the BMC Defender Agent for Windows since this port number is frequently shut off by many firewalls.

How do I verify that the CO-sysmsg.exe program is actually running?

You can run the CO-sysmsg.exe program at a command prompt using the -foreground option to the program. You can also check the CO-sysmsg.log file to see what errors have been encountering during normal execution. You can enable the logging of all syslog messages to this file by setting the LogLocal directive to True in the CO-sysmg.cnf file. These techniques are all useful for assessing what messages are sent and how the CO-sysmsg.exe program is running.

I want to use the sendlog.exe program, but not the CO-sysmsg.exe program.

The sendlog.exe program is a completely stand-alone utility. It does not require any special DLL files and does not require the execution of the CO-sysmsg.exe program. Simply copy the sendlog.exe program to any location, and it can run just fine. This is very handy for supporting batch files and scripting applications. (The program can easily launch, the CO-sysmsg.exe program never sees them via Perl, PHP, and many other scripting languages.)

My CO-sysmsg.exe program appears to be ignoring certain events.

Make sure that the Event Viewer program is not filtering the events. (Use the Control Panel > Admin Tools > Event Viewer > Properties dialog, and click on the Filters tab.) Microsoft Event Viewer can selectively shut off certain events at the source, so that the CO-sysmsg.exe program never sees them, and never relayed to the syslog server.

My CO-sysmsg.exe program appears to ignore the Security or other Windows event logs.

Assuming that you do not have the configuration file completely filtering all messages, this situation can occur if you make the event log too small, and the event log is rapidly updating.

Can I reduce the Microsoft event log sizes to their minimum size?

No. The system requires ample buffer space, in the form of non-trivial event log sizes. Some users might think that, because the CO-sysmsg.exe program is in place, the event logs can be made very small (because the CO-sysmsg.exe program relays events to the syslog Server host). In practice, this is partially true. However, the system requires around 1024 Kbytes as the maximum size of any event log, or more.

How can I decrypt an encrypted syslog message?

You can’t. The BMC Defender encryption routine uses a secret key available only to the BMC Defender Sigma Web Framework. The encryption is a one-use pad type encryption with a time-based rotating cipher. Contact the vendor for enhanced encryption techniques, including public key encryption schemes.

I enabled encryption in the configuration file, but I see no change.

The BMC Defender Server is decrypting the messages correctly, and you might not see any change. However, if the destination for these syslog messages is some other program (such as a UNIX syslogd program), it is readily apparent by inspecting the syslog that the message content is encrypted.

How do I run multiple copies of CO-sysmsg.exe?

Do not install the CO-sysmg service. Instead, use the Use CO-svc.exe program, that is a regular component of the BMC Defender Web Framework. Install this service, configure the sched.cnf file to launch the CO-sysmsg –foreground program, and install the CO-sysmsg program in several different directories, one directory for each destination. This provides the added benefit of providing a separate configuration file for each destination, that is the flexibility that might probably be needed in any management scenario.

How do I interface to the Windows Performance Monitor?

The easiest way to send syslog messages to monitor performance is to use the Windows performance monitor to send alerts. Although it is not commonly known, the Windows Performance Monitor contains a very sophisticated alerting facility, that permits event log messages (of arbitrary text content) is generating whenever a performance counter is at a limit. The event log messages are subsequently converting into to syslog messages, that arrive at the syslog receiver.  As an alternative, you can actually launch external programs when Windows Performance Alerts occur. In this case, you can configure the Performance Monitor to execute the end log program, passing as the syslog message the counter name, counter values, and other items of interest. This is all available via the Admin Tools > Performance > Performance Logs And Alerts screen, that is a standard part of the Windows system.

How do I detect when a user logs on or off the system?

This usually requires no modifications to the system. However, it might be that Windows Audit policies (in the Local Security settings screen of the Windows Control Panel > Administrative Tools screen) have not been enabled. Launch this standard windows tool, drill down into the Security Settings > Local Policy > Audit Policy settings and make sure that the Audit Logon Events settings are set to send event messages on both Success and Failure type events.

I am generating a lot of messages that say “No data available.” What does that mean?

This message is actually a briefer message that is logged in the event log whenever the registry does not contain an event log key. This is probably the result of a third-parties sloppy uninstaller program or might indicate a problem with the registry. You can filter out all of these messages via the CO-sysmsg.cnf file or you can investigate what software components are missing from the system. This is a common problem of older machines or machines that have migrated from older operating system versions to newer versions (such as Vista).  These keys can be inspected via the regedit program, by looking at the SYSTEM\CurrentControlSet\Services\Eventlog portion of the Windows registry.

I am receiving messages that begin “data://” followed by garbage.

This is due to the encryption mechanism of the CO-sysmsg.exe and sendlog programs. You are receiving this data at some server OTHER than the BMC Defender Server. The data encryption works only between the BMC Defender Agent for Windows and BMC Defender Server. Turn off the data encryption in the CO-sysmsg.exe program via the EncryptData directive in the CO-sysmsg.cnf file. To turn off the encryption associated with the sendlog.exe program you need to unset the SIGMA_ENCRYPT_DATA environmental variable. This can take care of the problem.

The BMC Defender Server does not receive the syslog message until several minutes after the message generates.

The CO-sysmsg.exe program parses events in the order that they were received. The program contains specific logic to prevent messages from being sent faster than the MsgDelayMsecs directive. The minimum value for this is 100 msecs, regardless of what minimum value you have configured. Therefore, most messages that can be sent by the CO-sysmsg.exe program are 10 messages per second. You are probably in a situation where a flood of messages enters into the Windows event log, and the CO-sysmsg.exe program is pacing itself to report all of these messages. 

The BMC Defender tunneling software is not working.

There are various configuration issues that can occur with the BMC Defender tunneling, ranging from firewall issues to mismatched encryption keys. Check the Troubleshooting information, found in this space. In particular, you can set the LogLocal configuration directive in the CO-tsend.cnf and CO-trecv.cnf file to True and restart these two processes. This logs all the connections and transfers of these two programs from the CO-tsend.exe to the CO-trecv.exe programs, so you can easily find the communication failure point by examining the CO-tsend.log and CO-trecv.log files.