Understanding syslog debug events

This section contains the following topics that describe the debug events that a user might see when syslog debugging is enabled for a given device agent:

To learn how to enable syslog debugging, see Managing device agents. To learn how to manage syslog external event filters, see Managing external event filters.

Each debug event displays INFO in the Severity column, Debug in the Category column, Syslog Debugging in the Event column, and nothing in the Target column.

The Date/Time column displays the timestamp when the event was created at the BMC Network Automation application server, not when the underlying message was created by the device agent. The Source column shows System for events from the Local agent, and System-{Remote_Agent_Name} for events from remote device agents.

Events fired when a syslog processing queue becomes full at an agent, or when a full queue frees up again

These events are not debug events. The full event displays Syslog message queue full; one or more messages dropped in the Event column, while the not-full event shows Syslog message queue was full, but it is now down to 95% capacity. The Source column for these events displays System for the Local agent, and System-<Remote_Agent_Name> for remote agents. The Description column for these events displays the name of the queue involved (typically InetMessageQueue ).

The syslog processor is designed to handle a small but steady stream of messages, with short bursts at higher volume. When the syslog queue gets full, it indicates that BMC Network Automation is unable to process a constant high volume of syslog messages. The solution is to reduce the messages at the source (the most frequent culprits are PIX/ASA devices), or filter the messages using a relay (such as kiwi or syslog-ng).

Agent initialization sequence

The following table shows an example sequence of debug events during initialization of an agent at address 172.21.127.10 in a system containing two enabled syslog External Event Filters: Configuration Changes and Received message with Critical severity.

Description	Comment
`local addresses: [172.21.127.10]`	Indicates that the agent has a single local address of 172.21.127.10.
`adding SYSLOG filter Log All Configurations Changes`
`adding SYSLOG filter Received message with Critical severity`
`consuming messages from InetMessageQueue ...`	Indicates that the agent thread to pull syslog messages off the queue and filter them is running.
`listening for syslog messages on port 514 ...`	Indicates that the agent thread to listen for incoming syslog packets is running on port 514.

Message processing sequence - rejected local echo

The following table shows an example sequence of debug events in which a device agent receives a syslog message encapsulated in a single packet directly from device bcan-cisco1760-02 at address 172.21.125.4.

The message indicates that someone connected to the device from the device agent's platform triggered the firing of the syslog message, therefore the agent does not forward it to the BMC Network Automation server as a potential syslog event, because it is probably an echo of server's own device communications. Logging to a file named C:\syslog_info is enabled for the agent.

Description	Comment
`received message: headerAddress=172.21.125.4, protocol=SYSLOG, payload=<189>2: *Oct 6 09:42:00: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (172.21.127.10)`	Indicates a syslog packet has been read off the socket by the agent and added to the queue to await parsing and filtering.
`built syslog message: deviceAddress=172.21.125.4, fromRelay=false, severity=5, msg=2: *Oct 6 09:42:00: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (172.21.127.10)`	Indicates that the agent has successfully parsed the syslog packet into a syslog message object.
`log to C:\syslog_info: [Oct 06 09:42:31 172.21.125.4 2: *Oct 6 09:42:00: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (172.21.127.10)]`	Indicates that the agent is about to append the contents of the syslog message payload to its syslog log file (if logging is enabled for this agent). The contents are first massaged to remove any leading facility/security code, and prepend the current date followed by the address of the originating device. The address is a symbolic name, unless the `syslogLogSymbolicNames` global property is set false.
`rejecting local echo message: deviceAddress=172.21.125.4, fromRelay=false, severity=5, msg=2: *Jul 16 15:17:50: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (172.21.127.10)`	Indicates that the agent found that the syslog message originated from the agent platform's own communication with the device, and therefore is not forwarded to the BMC Network Automation server as a potential event.

Message processing sequence - rejected via filtering

The following table shows an example sequence of debug events in which a device agent receives a syslog message encapsulated in a single packet directly from device ban-cisco4003-01 at address 172.21.127.68.

The message content does not match either of the enabled filters, therefore the agent does not forward it to the BMC Network Automation server as a potential syslog event. Logging to a file named C:\syslog_info is enabled for the agent.

Description	Comment
`received message: headerAddress= 172.21.127.68, protocol=SYSLOG, payload=<189>2: *Oct 6 09:42:00: %MGMT-6-LOGINPASS:User admin logged in from 172.21.126.21 via SSH`
`built syslog message: deviceAddress=172.21.127.68, fromRelay=false, severity=5, msg=2: *Oct 6 09:42:00: %MGMT-6-LOGINPASS:User admin logged in from 172.21.126.21 via SSH`
`log to C:\syslog_info: [Oct 06 09:42:31 172.21.127.68 2: *Oct 6 09:42:00: %MGMT-6-LOGINPASS:User admin logged in from 172.21.126.21 via SSH`
`rejecting unwanted message: deviceAddress=172.21.127.68, fromRelay=false, severity=5, msg=2: *Oct 6 09:42:06: %MGMT-6-LOGINPASS:User admin logged in from 172.21.126.21 via SSH`	Indicates that the agent found that the syslog message did not pass through at least one syslog External Event Filter, and therefore is not forwarded to the BMC Network Automation server as a potential event.

Message processing sequence - accepted syslog event

The following table shows an example sequence of debug events in which the Local agent receives a syslog message fragmented across two packets, originating from device bcan-cisco1760-01 at address 172.21.127.58, relayed from a known syslog relay at address 172.21.127.25.

The message content matches one of the enabled filters, therefore the device agent forwards it to the BMC Network Automation server which determines that it came from a known device and turns it into a syslog event. Logging to a file named C:\syslog_info is enabled for the agent.

Note that syslog messages received directly from devices is never fragmented. Fragmentation is only possible in relayed messages. It is also possible for relayed packets to contain multiple embedded syslog messages.

Description	Comment
`received message: headerAddress=172.21.127.58, protocol=SYSLOG, payload=<189>2: *Oct 6 09:42:00: %SYS-5-CONFIG_I: Configured from tftp://s-mcwilli-02/bcan-01.tmp`	This packet is the first fragment of a syslog message which extends over two packets
`message is from relay 172.21.127.25, message: <189>2: *Oct 6 09:42:00: %SYS-5-CONFIG_I: Configured from tftp://s-mcwilli-02/bcan-01.tmp`	Indicates that the syslog packet the agent is attempting to parse was sent to us from one of the known relay addresses defined in the System Parameters.
`received message: headerAddress=172.21.127.58, protocol=SYSLOG, payload=by admin on vty (127.21.126.21)`	This packet is the second fragment of a syslog message which extends over two packets.
`message is from relay 172.21.127.25, message: by admin on vty (127.21.126.21)`
`built syslog message: deviceAddress=172.21.127.58, fromRelay=true, severity=5, msg=2: *Oct 6 09:42:00: %SYS-5-CONFIG_I: Configured from tftp://s-mcwilli-02/bcan-01.tmp by admin on vty (127.21.126.21)`	Indicates that the agent has successfully parsed the two syslog packet into a single syslog message object.
`log to C:\syslog_info: [Oct 06 09:42:31 172.21.127.58 2: *Oct 6 09:42:00: %SYS-5-CONFIG_I: Configured from tftp://s-mcwilli-02/bcan-01.tmp by admin on vty (127.21.126.21)`	Unlike a message received directly from a device, the payload from a relayed message is logged to file without any modifications.
`forwarding filtered message: deviceAddress=172.21.127.58, fromRelay=false, severity=5, msg=2: *Oct 6 09:42:06: %SYS-5-CONFIG_I: Configured from tftp://s-mcwilli-02/bcan-01.tmp by admin on vty (127.21.126.21)`	Indicates that the agent found that the syslog message did pass through at least one syslog External Event Filter, and therefore is forwarded to the BMC Network Automation server as a potential event.
`processing filtered message from Local, message: deviceAddress=172.21.127.58, fromRelay=false, severity=5, msg=2: *Oct 6 09:42:06: %SYS-5-CONFIG_I: Configured from tftp://s-mcwilli-02/bcan-01.tmp by admin on vty (127.21.126.21)`	Indicates that the BMC Network Automation server has received the filtered syslog message from the agent for final processing.
`found originating device bcan-cisco1760-01 managed by Local, message: deviceAddress=172.21.127.58, fromRelay=false, severity=5, msg=2: *Oct 6 09:42:06: %SYS-5-CONFIG_I: Configured from tftp://s-mcwilli-02/bcan-01.tmp by admin on vty (127.21.126.21)`	Indicates that theBMC Network Automation server has identified the syslog message as having originated from a known device associated with the Local agent. At this point the BMC Network Automation server fires an event with a Category value of `External`, Source value of `Syslog`, Event value of `Configuration Changes`, Target value of `bcan-cisco1760-01`, and a Description value of `2: *Oct 6 09:42:06: %SYS-5-CONFIG_I: Configured from tftp://s-mcwilli-02/bcan-01.tmp by admin on vty (127.21.126.21)`.

Miscellaneous debug events

The following table shows some less common, miscellaneous syslog debug events:

Description	Comment
`did not find any originating devices managed by <Agent_Name>, message: <Syslog_Message>`	Indicates that the BMC Network Automation server received a filtered syslog message from the agent, but the device the message originated from is not a known device, so no syslog event is fired.
`done consuming messages from InetMessageQueue`	Emitted after an unexpected error. Indicates that the thread to pull messages off the queue and filter them has stopped.
`done listening for syslog messages`	Emitted after an unexpected error. Indicates that the thread to listen for syslog messages has stopped.
`error consuming messages from InetMessageQueue: <Exception>`	Emitted after an unexpected error in the thread to pull syslog messages off the queue and filter them.
`error listening for syslog messages: <Exception>`	Emitted after an unexpected error in the thread to listen for syslog messages.
`found candidate device <Device_Name> but it is not managed by <Agent_Name>, message: <Syslog_Message>`	Indicates that the BMC Network Automation server received a filtered syslog message from the agent, but the device the message originated from is not one being managed by the agent in question, so no syslog event is fired.
`PartialMessageFlusher checking for orphans ...`	Indicates that a packet representing a fragment of a syslog message has been detected and cached. Further processing of the message is delayed until the missing fragments are found. The agent starts up a thread (unless one is already running) which checks every five seconds to see if any cached packet ages beyond a default time limit. Such orphaned packets are removed from the cache and processed as incomplete syslog messages. The default time limit is 15 seconds, unless otherwise specified by the `syslogPartialMessageAgeSeconds` global property.
`PartialMessageFlusher done`	Indicates that the thread that was baby sitting cached syslog message fragments has finished running because the fragment cache has become empty.
`PartialMessageFlusher found orphan: <Message>`	Indicates that the thread baby sitting cached syslog message fragments has found that one has aged beyond the allowed time limit for finding missing fragments belonging to the same message. The orphan is removed from the cache and processed as an incomplete syslog message now.
`removing SYSLOG filter <Filter_Name>`	Emitted during agent re-initialization, or whenever a syslog External Event Filter is disabled or significantly modified.
`unable to log to syslog file: <Exception>`	Indicates that a write error occurred attempting to append a new entry to the syslog log file.
`unable to process unknown message from <Agent_Name>, message: <Message>`	Indicates that the BMC Network Automation server received a message of unknown type from the agent (not a syslog message or an internal message), which it ignores.
`unknown filter <Filter_Key> reference in results for message: <Syslog_Message>`	Indicates that the BMC Network Automation server received a filtered syslog message from the agent, but the External Event Filter that it passed through no longer exists in the database, so no syslog event is fired.