Data collection redundancy

This topic provides the following information that you can use for data collection planning redundancy.

Redundancy overview

IT Data Analytics provides you a mechanism of collecting and searching data. Redundancy is supported only for the data collection function.

The data collection function is concerned with the following product components:

Collection Station: Acts as a receiver for data collected by Collection Agents and forwards it to Indexers for indexing.
Indexer: Receives data from Collection Stations and indexes the data so that it is available for searching.

Therefore, to support redundancy for data collection, you need to maintain the availability of the Collection Station and Indexer components.

Data collection redundancy is concerned with securing the following channels:

Channel between the Collection Agent and Collection Station (after data is collected).
Channel between the Collection Station and Indexer (before data collected reaches the Indexer for indexing).

Note

Redundancy only applies to securing data collection if the Collection Station goes down. It does not apply if the Collection Agent goes down. Because the Collection Agent is expected to be located on the same computer as the target host (where data resides), the Collection Agent is likely to go down only if the target host goes down.

The following diagram points the channels that you can secure by applying data collection redundancy.

redundancy diag.jpg

Need for redundancy

The need for redundancy depends on two factors:

Business needs – If you want to increase data continuity and availability or if the data you are collecting is critical.
Size and scale of your business – If you are operating in a large scale, multiple-server deployment.

How redundancy works?

To implement data collection redundancy, it is imperative that the Collection Station(s) deployed in your environment is included in a common pool. This pool is automatically created when you select the check box for enabling redundancy while installing the Collection Station. If the Collection Station is installed while performing a typical installation of the product, then it is assumed that you want to enable redundancy. Therefore, in a typical installation the Collection Station is automatically added to the pool. The pool is saved with the name "ITDAPool" in the agent.properties file. For more information, see Modifying-the-configuration-files.

When you upgrade from an earlier version of the product, it is assumed that you want to enable redundancy. Therefore, when you upgrade the Collection Station is automatically added to the pool.

After you implement redundancy, if a Collection Station goes down, the Collection Agents communicating with that Collection Station are redirected to the next available Collection Station and data collection continues. When a Collection Agent contacts the Collection Station for the first time, the Collection Station informs the Collection Agent about the following information:

Details about the other Collection Stations operating in the pool.
Details about the sequence in which the Collection Agent must contact the other Collection Stations in case it goes down.

Thus, if the Collection Station goes down, the Collection Agent uses the preceding details to redirect data collection work to the next available Collection Station.

Best practice

Note that if there are any data collectors configured on the Collection Station and if the Collection Station goes down, those data collectors stop working. It is recommended to use the Collection Station only as a receiver for data forwarded by the Collection Agents and not for actually collecting data. For more information, see Agent-types.

If you already have data collectors configured on the Collection Station, you might want to move them. For more information, see the CLI command for moving data collectors.

Collection Agents and redundancy

To successfully implement redundancy, it is important that the Collection Agents deployed in your environment are connected with the pool in which the Collection Stations are operating. This happens automatically when you install or upgrade to the latest version of the Collection Agents.

If you upgrade to the latest version of the product but continue using older Collection Agents, the Collection Agents will not have details of any new Collection Stations that you install. They will continue to contact only those Collection Stations of which they are aware.

The following information compares how data collection works with versions 2.0 and 2.1 of the Collection Agent:

Collection Agents 2.1.00 or later

While installing version 2.1.00 of the Collection Agent, you need to provide details of any one Collection Station available in the pool. When the Collection Agent contacts this Collection Station for the first time, the Collection Station provides details of all other Collection Stations in the pool. Furthermore, it provides the order in which the Collection Agent must contact the next Collection Station if the initial Collection Station goes down.

Additionally, if a new Collection Station is added to the pool, then details of this Collection Station is automatically made available to the Collection Agent. Thus, all the Collection Agents in the pool are communicated details about the Collection Stations newly added to the pool.

Collection Agents 2.0.00 or earlier

If you are using Collection Agents with version earlier than 2.1, then the Collection Agents communicate with the Collection Stations in the same way as they did in the earlier version.

In the earlier version, if you provided details of three Collection Stations while installing the Collection Agent, the Collection Agent will first contact the Collection Station that was added initially. If the first Collection Station goes down, then the Collection Agent will contact the second Collection Station. If the second Collection Station also goes down, then the Collection Agent will contact the third Collection Station. The Collection Agent contacts the Collection Stations in the same order as they were added.

Where to go from here

If you are installing the product for the first time, proceed as follows:

Install multiple Collection Stations in your environment. For more information, see Installing-in-a-multiple-server-environment.
Install Collection Agents that can automatically communicate with the Collection Station pool. For more information, see Setting-up-Collection-Agents.

If you are upgrading from an earlier version of the product, proceed as follows:

Upgrade to the new version of the Collection Station. For more information, see Upgrading.
Upgrade to the new version of the Collection Agents. For more information, see Upgrading-the-standalone-Collection-Agent.