This topic covers the architecture of the BMC ProactiveNet Server, Integration Service, and database. These best practices are also posted as a webinar on BMC Communities.
The following diagram illustrates the high-level architecture of the BMC ProactiveNet Performance Management core solution components.
BMC PATROL Agents collect performance data and generate events for availability metrics. Both performance data and events from BMC PATROL are streamed though the BMC ProactiveNet Integration Service nodes. (This assumes that versions 9.5 of the BMC ProactiveNet Server and the BMC ProactiveNet Integration Service are in use).
The Integration Service nodes forward the performance data to the BMC ProactiveNet Server. Not all performance data has to be forwarded. Performance data can be collected and stored at the BMC PATROL Agents and visualized as trends in the BMC ProactiveNet console without having to stream the data to the BMC ProactiveNet Server. This is configurable for each PATROL parameter. It is a best practice to limit the streaming of performance data to the BMC ProactiveNet Server for the following purposes only:
Performance data for all parameters designated as Key Performance Indicators (KPIs) must be streamed to the BMC ProactiveNet Server to support baselines, abnormality detection, and predictive alarming.
For performance reporting in BMC ProactiveNet Performance Management Reporting, stream the data for all the parameters that are required in performance reports. This must be limited to KPI parameters, but can be extended.
Include parameters that are necessary or required for probable cause analysis leveraging baselines and abnormalities.
The Integration Service processes forward events to event management cells running on the Integration Service hosts. These event management cells filter and enhance events, and then forward the events to event management cells used for correlation.
BMC strongly recommends that the environments you set up for BMC ProactiveNet development and test be separate from the production environment.
Much of the overall architecture remains unchanged from the previous release; however, there are some significant changes. The major high-level changes are listed below:
The Integration Service process has been significantly simplified.
Support for multiple Oracle schemas in the same Oracle instance is provided. This applies to both the BMC ProactiveNet and the BMC ProactiveNet Performance Management Reporting databases.
Connectivity between the Integration Service processes and the BMC ProactiveNet Central Monitoring Administration has been consolidated. Central Monitoring Administration now communicates with each Integration Service through the BMC ProactiveNet Server that the Integration Service is connected to.
In BMC ProactiveNet 9.0, data was sent from the PATROL Agents to the Integration Service nodes, but the BMC ProactiveNet Server polled a single data point every five minutes from the Integration Services. In version 9.5, data is streamed from the PATROL Agents through the Integration Service to the BMC ProactiveNet Server. Consequently, every data point is now collected by the server and stored in the database for performance parameters that are streamed.
In version 9.5, events are now streamed from the PATROL Agents to the Integration Services on the same port that performance data streams to. The Integration Service then sends the events either to the remote cells or directly to the BMC ProactiveNet Server. In version 9.0, the PATROL Agents sent events directly to the remote cells on a separate port.
The BMC ProactiveNet Performance Management solution supports installing a single BMC ProactiveNet Server or multiple servers in the same environment. The overall architecture diagram illustrates a single server environment. The following diagram illustrates a multiple BMC ProactiveNet Server environment with a BMC ProactiveNet Central Server that has Central Monitoring Administration and Child Servers.
A multiple BMC ProactiveNet Server implementation supports distributed service models so that specific configuration items (CIs) in one BMC ProactiveNet Server can be visible in another model in a separate server. This is supported by web services installed with the BMC ProactiveNet Server(s). The BMC ProactiveNet Central Server acts as a single point of entry for users and provides a common point to access service models.
Although not required, BMC recommends installing the top tier BMC ProactiveNet Server as a Central server with Central Monitoring Administration included, for most environments
A single implementation of the BMC ProactiveNet Performance Management solution cannot support mixed versions of BMC ProactiveNet Servers. This includes Central Monitoring Administration. All BMC ProactiveNet Server versions must be the same in a single environment.
The BMC ProactiveNet Server is supported with one of the two database options. You can install the embedded Sybase database that comes with the product, or you can leverage an Oracle database that you provide. If you choose the Sybase option, the Sybase database is installed with the BMC ProactiveNet Server on the same host with the application server and web server components. The Sybase database cannot be installed on a separate server.
Use the Sybase database in the following situations:
When an Oracle database license is not available
When no Oracle DBA is available
When robust database availability is not required
In small and medium environments in which Oracle is not available
If you choose the Oracle database option you must provide an Oracle instance. The Oracle instance must be installed on a host separate from the BMC ProactiveNet Server. You can either allow the BMC ProactiveNet Server installer to create the schema for BMC ProactiveNet in the Oracle database, or you can create the schema manually using scripts provided with the installer. For more information, see Installing the BMC ProactiveNet Server on Microsoft Windows with Oracle as database.
In BMC ProactiveNet 9.5, multiple BMC ProactiveNet Servers can be supported on a single Oracle instance. This is accomplished by creating/allocating separate Oracle schemas in the single Oracle instance, one for each server. Database resources and sizing of the Oracle instance SGA must be increased to support this. The following diagram illustrates how multiple BMC ProactiveNet Servers can share a single Oracle instance.
This is not possible with the Sybase database.
Similarly, multiple BMC ProactiveNet Performance Management Reporting instances can share the same Oracle instance.
An Oracle instance must never contain a schema or schemas for the BMC ProactiveNet Server while also containing a schema or schemas for BMC ProactiveNet Performance Management Reporting. The BMC ProactiveNet Server instance(s) and reporting instance(s) must be separated for performance reasons. Additionally, Oracle database instances for BMC components must be dedicated for BMC products and must not contain any third party application data or code.
The following diagram below illustrates requirement of the Oracle instance:
Each schema in an Oracle instance must have a unique Oracle database user that owns the schema. When you install the BMC ProactiveNet Server, the installer prompts you for the user who owns the schema for the current instance as shown in the following screen. Ensure that you enter a unique user for each BMC ProactiveNet Server instance that you install.
Additionally, each unique BMC ProactiveNet Server schema must be installed in separate data files and corresponding tablespaces in the Oracle Instance. The BMC ProactiveNet Server installer allows you to specify these criteria as shown in the following figure:
The BMC ProactiveNet Server installer requires remote connectivity to the Oracle instance and must be able to “connect as sysdba” remotely. You must validate this connectivity before trying to install the BMC ProactiveNet Server. BMC recommends that you install SQL*Plus or the Oracle Instant Client on the target BMC ProactiveNet Server and test/validate the Oracle database connectivity as “sysdba” from that server before starting BMC ProactiveNet Server installation. For more information, see Manually setting up the Oracle database.
You can configure Oracle so that each database instance has a unique Oracle listener, or a single listener that can support multiple database instances. BMC recommends that you designate a unique Oracle listener for each database instance. This isolates listener issues to a single instance. Additionally, high availability (HA) must be set up for the Oracle listeners and the databases. BMC recommends leveraging Oracle RAC for database HA. For more information, see Setting up a customized Oracle database.
BMC recommends leveraging the same database platform for all BMC ProactiveNet Server databases across the environment. Although it is technically possible to install some BMC ProactiveNet Servers using the embedded Sybase database and others using Oracle, standardizing on one platform provides a common way to manage HA, backup/restore, and export/import data from one instance to another. Database export/import is only possible between the exact same versions and patch levels.
In previous releases of BMC ProactiveNet, each instance of both the server and BMC ProactiveNet Performance Management Reporting components, required a dedicated Oracle instance. (This assumes Oracle was the chosen database and not Sybase.)
Use the Oracle database in the following situations:
When an Oracle license is already available
Oracle is a standard database platform used in the environment
When robust database availability is required
The following are additional best practices when using Oracle as the database platform:
Use Oracle RDBMS v 184.108.40.206.0 or v220.127.116.11.0
Create at least two BMC ProactiveNet users, one for data storage and one for data views. Consider a third “backend user” to manage issues like locked accounts.
Physically co-locate the BMC ProactiveNet Server and the database server on the same subnet.
Use BMC Database Recovery Management or an Oracle tool such as RMAN.
Enable archive logging.
Use Oracle RAC for HA.
Use Oracle Data Guard for disaster recovery.
The following diagram illustrates how the Integration Service nodes fit into the BMC ProactiveNet 9.5 architecture. A reference to the 9.0 architecture is provided on the left for comparison.
The BMC ProactiveNet Integration Service version 9.5 accepts streaming of PATROL data and events using a common connection port. The default port is 3183. This includes all the data points and events from PATROL for parameters that you select. After the events arrive at the Integration Service, they are separated and follow a unique path to one of the following based on configuration:
BMC PATROL sends performance data, streaming it to the BMC ProactiveNet Server. This is not summarized data. The data does get summarized in the BMC ProactiveNet Server (as in previous versions) but raw data is sent from the PATROL Agents. This includes all the data points for parameters that you decide to send.
The architecture also supports buffering of PATROL performance data and events at the PATROL Agents in case there is a network connectivity issue or if the Integration Service cannot be reached. When the PATROL Agent reconnects to an Integration Service process, the buffered data is sent. This capability is not intended to support buffering for very large amounts of data. It is intended to support a few minutes of lost connectivity, not hours or days. Testing has shown that the process can support up to 30 minutes of data collected by the PATROL Agents across 1000 managed servers.
The Integration Service processes are generally “stateless” meaning the following:
The 9.5 Integration Services do not cache PATROL Agent namespace data and data points as was happening in version 9.0. The data is now streamed directly through to the BMC ProactiveNet Server. The server now gets every data point rather than only a snapshot every 5 minutes from the Integration Service cached data points.
There are no adapters associated with the PATROL data collection.
The Integration Service acts as a proxy to receive and forward both data and events that are sent to it from the PATROL Agents. It also receives PATROL Agent and Knowledge Module (KM) configuration data from Central Monitoring Administration and passes that data to the PATROL Agents.
You can optionally install the following components and configure them on the Integration Service host depending on whether or not they are required in the environment. Before installing any of these additional components, consider scalability and additional resources that you may require.
Event management cell
The event management cell is the event management process installed locally on the same server with the Integration Service. BMC recommends and it is a best practice to install the event management cell on all of the Integration Service host computers.
This assumes the environment includes the PATROL Central Console which is not required. See the PATROL documentation for RT Server requirements. The Console Server must be installed on a separate machine.
BMC Event Adapters
BMC Event Adapters work with the event management cell to consume non-PATROL events. For example, SNMP traps and so on. BMC recommends that significant non-PATROL event collection be dedicated to other event management cells as recommend in previous BMC ProactiveNet versions. The default event adapter classes, rules, and files are installed with the cell that is installed with the Integration Service.
PATROL Agent and Knowledge Module (KM)
The PATROL Agent and Knowledge Module (KM) monitor the Integration Service host processes.
Certain processes that ran on the older Integration Service hosts are no longer needed and must not be installed or used with the Integration Service version 9.5. These include the following:
A PATROL Agent acting as a notification server
Integration Service data collection adapters (used in version 9.0 and earlier)
The Integration Service in version 9.5 can consume and forward both performance data and events. “Technically” the cell is not required in order to forward events to the BMC ProactiveNet Server. Therefore, the cell “technically” does not have to be installed with the Integration Service.
For most environments, BMC recommends propagating events from the Integration Service to a lower tier event management cell. This is especially important in environments that meet any of the following conditions:
Include multiple events sources other than PATROL
Has more than few users
A medium or large environment involving more than 100 managed servers
The event management cells allow you to further process events (event enrichment, filtering, correlation, de-duplication, auto closure, and so on) before sending them on to the BMC ProactiveNet Server. This type of event processing must avoided on the BMC ProactiveNet Server as much as possible. Event processing in the BMC ProactiveNet Servers must be controlled and limited to the following:
Event presentation of actionable events only
Collection of events for Probable Cause Analysis
Events used in service modeling
Events sent to the BMC ProactiveNet Servers must be closely controlled and limited for the following reasons:
Event presentation in the BMC ProactiveNet Server must not be cluttered with un-actionable events that distract or otherwise reduce the efficiency of end users.
The new capability in BMC ProactiveNet 9.5 to view PATROL performance data in BMC ProactiveNet without having to forward and store the data in the BMC ProactiveNet database, is likely to decrease the number of parameters that trend in the BMC ProactiveNet Server for most environments. This may increase the number of events propagated from PATROL for parameters that do not require baselines, but do require static thresholds. This increase will increase the load on the event management cell in the BMC ProactiveNet Server.
PATROL events are approximately twice the size in bytes compared to events generated in the BMC ProactiveNet Server. A larger volume of PATROL events increases the memory consumption of the event management cell on the BMC ProactiveNet Server and additionally increases the BMC ProactiveNet Server startup time. The overall startup time for a BMC ProactiveNet Server at full capacity ranges from 15 to 20 minutes.
The automated event to monitor association introduced in BMC ProactiveNet 9.5 has a slightly increased load on the event management cell that is embedded in the BMC ProactiveNet Server.
The BMC ProactiveNet Central Server can act as a presentation server for all events processed in the Child Servers. Tto accomplish this, events can be propagated from the Child Servers to the Central Server. Additionally, event management cells on the Integration Service hosts must be integrated with the BMC ProactiveNet Server so that events in the remote cells are accessible in the BMC ProactiveNet Operations Console, under the Other Cells view.
BMC recommends integrating the Child Servers with BMC Remedy IT Service Management Suite and other BMC products such as BMC Atrium Orchestrator for event processing related to products such as these. These integrations must not be configured in the Central Server. The only exceptions are the BMC Atrium Single Sign-On and BMC Change Management components.
The following are additional best practices for Integration Services, event management cells, and the Integration Service host computers:
Limit the usage of HTTPS between the Integration Service nodes and the BMC ProactiveNet Server(s). HTTPS is not as scalable as HTTP and requires more administration.
Do not send raw events directly to the BMC ProactiveNet Server. Every environment must have at least one lower tier event management cell.
Install the event management cell on all Integration Service nodes.
Additional event management cells must not be installed on the BMC ProactiveNet Server.
Install additional event management cells on Integration Service hosts and remote host as needed.
Do not configure Integration for BMC Remedy Service Desk, notifications, or other global event forwarding integrations on the lower tier event processing cells. Global event forwarding integrations must configured on the Child Server(s).
The number and placement of event management cells must be based on the number of events, event source domains (secure zones, geography, and so on), and major event sources. Always deploy multiple event management cells in the following scenarios:
Configure the display of remote event management cells in the BMC ProactiveNet Server when necessary.
Install dedicated event processing cells to manage large volumes of events from common sources such as SNMP traps, SCOM, and other significant sources of events.
Distribute event management cells as required, based on event loads and event sources.
Deploy event management cells close to or on the same node as event sources for third party sources.
Filter, enrich, normalize, de-duplicate and correlate events at the lowest tier event management cells as much as possible before propagating to the next level in the event flow path.
Do not collect unnecessary events. Limit event messages sent from data sources to messages that require action or analysis.
Do not try to use the event management cells as a high volume SNMP trap forwarding mechanism.
Use dedicated Integration Service hosts for large domain data collection. For example VMware vSphere, remote operating system monitoring, and other large sources of data.
Install Integration Service hosts close to the data sources for which they process data. Deploy by geography, department, business, or applications especially if multiple Integration Services are required from a single source.
Do not collect excessive or unnecessary performance data. Review the need for lower polling intervals considering server performance and database size.
Do not collect trends for availability metrics.
Both performance data and events are sent to the Integration Service from BMC PATROL Agents over the same TCP communication path. The Integration Service then forwards events to the event management cell that is running locally on the same host with the Integration Service. The event management cell further processes the events (filtering, enrichment, correlation, and so on) and forwards them to the BMC ProactiveNet Server. Performance data is sent from the Integration Service directly to an enterprise event correlation cell. The event correlation cell then forwards the events to the BMC ProactiveNet Server.
BMC PATROL Agent events are considered internal intelligent events in BMC ProactiveNet 9.5. In earlier versions, they were considered external events and were managed like third party events. BMC PATROL Agent events in BMC ProactiveNet 9.5 are mapped to the instance object they belong to (previously, the mapping was only at the device level). This monitor instance association of BMC PATROL events improves Probable Cause Analysis leveraging categorizations (for example, Database, Application, Server, Network, and so on).
The following are best practice recommendations for Integration Services with respect to event and data flow processing:
The Integration Service installed with the BMC ProactiveNet Server (locally on the server) must be configured as a staging Integration Service (in a POC environment, it can be used for data collection).
At least one remote Integration Service node must be deployed for all environments.
BMC recommends installing the Integration Service and event management cell in pairs so that each Integration Service process has a corresponding event management cell installed on the same host computer. In this configuration, events are propagated from the Integration Service to the event management cell running on the same host. The installation of an event management cell is an option available when you install the Integration Service.
It is very important to maintain the event flow path so that all events from any BMC PATROL Agent are always processed through the same event management cell(s) (including cell HA pairs). This ensures event processing continuity where automated processing of one event is dependent on one or more other events from the same agent. An example of this type of processing is the automated closure of critical events that is triggered by “OK” events for the same object that was in a state of critical alarm. If you do not maintain the same event flow path per agent through the same event management cell(s), event correlation of all events from the same agent is not possible because the necessary events are not received and available in the same cell(s).
Some environments may require more than two Integration Service nodes in a cluster and/or more than two Integration Service nodes defined for each agent that is sends the data (events and performance) through a third party load balancer to the Integration Service nodes. This is acceptable as long as all events from any one agent always flow through the same HA cell pair and the event processing continuity is maintained. For example, if four Integration Service nodes are clustered, then each node in the cluster must not have a cell configured on it. Instead, the cell must be on other systems (in an HA pair) so that the event path remains the same for all events coming from the agents that the cluster handles.
Configuration regarding performance and event data sent from the PATROL Agents to the BMC ProactiveNet Server is defined in policies that are automatically applied to the required PATROL Agents. The PATROL Agent assignment is defined in each policy configuration based specific criteria. The details of agent selection criteria per policy are discussed at Central Monitoring Administration best practices. BMC PATROL events and performance data are completely controlled at the PATROL Agent based on these policies. This means data, events, data and events, or no data and no events are controlled as per the parameter. You can edit or change these configuration settings as and when you want without having to rebuild any configurations or restart any processes.
This section describes in detail the connection details between BMC ProactiveNet and BMC PATROL Agent.
The following diagram illustrates the default ports on which connections are made regarding communications from the BMC PATROL Agents through the Integration Service to the BMC ProactiveNet Server. The direction of the arrows indicates the connection requests. For details about the new architecture in BMC ProactiveNet version 9.5, see BMC ProactiveNet architecture with the integration of the BMC PATROL Agent.
The following diagram illustrates the ports and connections related to BMC ProactiveNet Administration Console and the BMC PATROL consoles.
The BMC ProactiveNet Administration Console must always be installed on a computer separate from the BMC ProactiveNet Server. An instance of the Administration Console is installed on the BMC ProactiveNet Server by default. Use this instance of the Administration Console only in an emergency or if another instance is not available.
A BMC PATROL Console is not required in every environment. The need to use PATROL Consoles and BMC PATROL Configuration Manager is reduced in BMC ProactiveNet 9.5 due to the enhancements in Central Monitoring Administration. Install the BMC PATROL Console only in those environments in which specific BMC PATROL Console functionality is required.
The following points are reasons to include BMC PATROL Consoles and/or BMC PATROL Configuration Manager. These will not apply to all environments.
The environment has a legacy BMC PATROL implementation and the PATROL Console functionality needs to be continued for some period of time for migration and process related reasons.
Specific functionality in the BMC PATROL Console is required that is not available in the BMC ProactiveNet consoles. Examples of functionality limited to the BMC PATROL Console include the following:
In certain situations, detailed analysis of the PATROL Agent and KM operations may be necessary for troubleshooting. This must be accomplished using a BMC PATROL Central Console if it is required in production. The BMC PATROL Classic Console must be used only for KM development and never in a production environment.
Development of custom KMs to be loaded in Central Monitoring Administration requires the BMC PATROL Classic Console. It can also be used to analyze content in the BMC PATROL events at the PATROL Agent and must be done only in a development environment. Use the BMC PATROL Classic Console primarily for custom KM development.
When detailed understanding of a KM’s functionality and how it is configured cannot be understood without analyzing the KM using the BMC PATROL Console. This must be done in a development environment.
The following diagram illustrates the connections between the Administration Console and other BMC ProactiveNet components. The ports listed are default ports and the arrows indicate the direction of the connection requests. For more information about ports, see BMC ProactiveNet ports.