BMC Defender File Transfer Queue adapter

This section provides a detailed description of the BMC Defender File Transfer Queue adapter, an optional set of files and executables added to the BMC Defender Server to expand the role of the BMC Defender to include transmission of entire files (using syslog) to a receiving program. By using the BMC Defender File Transfer Queue adapter, you can easily create agentless system in which BMC Defender (or some other syslog receiver program) receives files that are periodically added to an arbitrary folder (installationDirectory\file-queue) or queue. Replace installationDirectory with the directory in which you installed the product. The default directory is C:\Program Files\BMC Software\BMC Defender.

The BMC Defender File Transfer Queue adapter consists of several components. A background process continuously monitors a user-specified directory for new text files. This background process reads a configuration file that you can edit through the BMC Defender web interface (or manually). The BMC Defender File Transfer Queue adapter also includes a dashboard gadget that you can use to continuously monitor the progress of the File Transfer process. 

The BMC Defender File Transfer Queue adapter extends the BMC Defender system to send entire text files to BMC Defender at regular intervals. This function augments the log file monitor functions of the Windows agent that enables you to monitor streaming log files.

Unlike the Windows log file monitor agent, the BMC Defender File Transfer Queue adapter monitors and transmits entire files that are asynchronously added to a folder through some external process, such as an FTP script. The installationDirectory\file-queue is created by the installation procedure. To redefine this directory, see Installing-BMC-Defender-File-Transfer-Queue-adapter. You can include up to four other queue directories.

By using the BMC Defender File Transfer Queue adapter, you can construct a simple system to permit a log file to be fetched at periodic intervals from a machine and copied to a queue. The contents of the file are automatically transferred to BMC Defender, where it can be searched and correlated. This provides a simple and efficient agentless collection system.

Configure and monitor the background process by using a tightly coupled integration with the main BMC Defender web interface. You can configure various parameters, including the queue name, message rates, facility and severity of messages, by using the Messages > Adapters > File Queue tab.

You can install the BMC Defender File Transfer Queue adapter without BMC Defender Server, such as on a remote machine. In this case, you can install the process as a service and then manually edit the configuration file to specify the queue directory, destination address, and other transfer parameters. The BMC Defender File Transfer Queue adapter runs as a proper windows service. You can configure the process by making manual edits of the configuration file. For more information, see Installing-BMC-Defender-File-Transfer-Queue-adapter.

System diagram

The BMC Defender File Transfer Queue adapter program consists of a single background process, the CO-queue.exe program, that runs at the BMC Defender Server or another network location. This process reads configuration data that has been configured by the operator, and watches for new files to be entered into a user specified folder.

When a new file is entered into the queue folder, the program waits for a configured period of time (by default 10 seconds) for the file to stabilize. When no writes or further modifications to the file have occurred for this period of time, the file is moved into an intermediate queue, and then transferred through syslog messages to a destination address. The file transfer takes place as a separate process, freeing the main process to continue monitoring the queue while the file is transferred. When the file is completely transferred, the queued file is deleted from the system.

Various transfers can simultaneously take place, up to a maximum of fifty concurrent processes. Once a transmission has been started, the BMC Defender File Transfer Queue adapter immediately waits for more files. This technique allows the user to simply copy a file into a designated directory in order to send the file to the syslog receiver. An external program, such as an automated FTP batch file or a scheduled job, provides the actual file copy into the queue directory.

The following diagram illustrates a file that is copied into a file queue, which is a user-specified folder. The CO-queue.exe service periodically polls the queue, based on the configuration data that you provide, and sends files form the queue to the BMC Defender Server using syslog messages, and immediately removes the file from the queue.

You can configure the file queue using the Messages > Adapters tab of the main BMC Defender Server web interface to point to a standard Windows folder.

You must define at least one file queue directory. You can define up to eight other queues as described in Installing-BMC-Defender-File-Transfer-Queue-adapter. All File Queue directories must exist on the same hardware platform as the CO-queue.exe process, but otherwise are ordinary directories.

Caveats

Several minor caveats exist to installing and using the software, as follows:

• Do not install the file queue (installationDirectory\file-queue) on a remote network disk. This can interfere with the Max Queue Wait Seconds setting and prevent proper transmission of files.
• Copy the files to the queue instead of moving them to the queue. The modify timestamp of the file is used to determine when the file is finished, and therefore the modify time of the file is an essential consideration. No transfer of the file begins until the modify time of the file is greater than the current time plus the queue wait time seconds value.
• The maximum number of messages sent by the program is significant. Setting the maximum number of threads (that is, fifty threads) and the maximum messages per second (that is, 1000 messages per second) can overwhelm a syslog receiver with data. Ensure that the syslog receiver is not flooded with messages that might cause data loss.
• Queued files are permanently deleted from the system. When constructing the file queue system, the designer must preserve files or data as needed, that is, if you need to retain the data somewhere other than the syslog receiver. No facility exists to preserve the data other than at the syslog receiver.
• The administrator must ensure that the CO-queue.exe program has permissions in the input folders specified in the configuration file. If permissions are not correctly set, you can see indications logged in the CO-queue.log file, which you can view through the web interface.
  
  

This section provides information about the following topics: