Indexer redundancy can help you achieve the following main goals:
Applying Indexer redundancy provides an added benefit of better search response times with high number of concurrent users.
A minimum of 3 Indexers are recommended for applying redundancy.
The following information can help you understand how Indexer redundancy works and how it can be enabled in your environment.
The Indexer indexes data received from the Collection Stations to make the data search-ready. If the Indexer goes down, you will start losing data even though the data was successfully collected. Furthermore, data lost will not be searchable. Also, in case of disk failures, you can lose data.
Indexer redundancy covers the following aspects:
The following diagram identifies the channels covered by maintaining the availability of Indexers.
Before you understand how Indexer redundancy works, you need to understand how data is indexed and how Indexers function:
After data is collected, it is added to an index on an Indexer node. An index contains a number of shards and all the data indexed is stored in the shards. An index is a high-level place that points to the location where particular data might be stored. A shard is a low level unit of an index that holds a small portion of all the data in the index.
A shard can be a primary shard or a replica shard. Each document in the index is stored in a single primary shard. The total number of primary shards determines the maximum amount of data that an index can hold. A replica shard is an identical copy of the primary shard. Multiple replicas can be created for a single primary shard. For more information, see Index replication.
All Indexers deployed in your environment form part of a cluster. Data collected is stored across all the Indexers.
In a single Indexer cluster, by default, that Indexer functions as the master. In a multiple Indexer cluster, you can have more than one Indexer functioning as the master.
If you are operating in a multiple-server deployment, the Indexer that you install for the first time (on the primary node) must be the master. All other Indexers are considered as peers and they must be provided with details of the master. The master plays an important role of the coordinator. Deploying multiple Indexers in your environment can be helpful to support better performance, longer data retention period, and continued data availability.
As a best practice, it is recommended that you use the shutdown CLI command to stop all the Indexers in the cluster, instead of stopping individual Indexer nodes.
If you try to stop an individual Indexer node, it is possible that index replication starts. This might lead to performance degradation. If you have a large set of Indexers deployed in your environment and if you want to stop all the Indexers, it is better that you use the shutdown CLI command instead of individually stopping each node.
The master Indexer communicates and coordinates with other peer Indexers. The master Indexer is like any other member of the Indexer cluster. This means like any other peer, the master continues to index data.
Any member of the Indexer cluster can play the role of a master, however at any given time, the Indexer cluster can have only one Indexer playing the role of a master. This means that when you configure more than one Indexer to act as the master, those Indexers are master-enabled (in other words, enabled to act as masters). If the current master goes down, the next master-enabled Indexer is elected to act as the new master. You can configure masters by running the configuremasters CLI command. This command can be also used to modify masters in the Indexer cluster.
If the master goes down, the cluster goes down, resulting in data loss. To avoid data loss, you must enable more than one Indexer to act as master in the cluster.
If a peer Indexer goes down, data already indexed by that Indexer remains unavailable (partial data loss). When a peer goes offline temporarily, the data held by the peer is not available for search, for the time when the peer is offline. The peer can go offline due to various reasons, for example, the node on which the peer is deployed goes down. After the peer comes online, the past data held by the peer becomes available for search. But during the time when the peer is offline, the data collectors are unable to connect and send data (for indexing) to the peer. Data sent to the peer in the time when it was offline is lost. To avoid data loss in such a scenario, you need to enable index replication, a process where a copy of the indexed data is stored on multiple Indexer nodes. For more information, see Index replication.
If the master Indexer goes down, the Indexer cluster goes down, thereby resulting in data loss. To avoid such a scenario, you need to enable more than one Indexer to act as master in the cluster. With this, if one master goes down, the next master-enabled Indexer assumes the responsibility of the master. In this way, the Indexer cluster remains up and data loss is avoided. Determining the number of masters required in an Indexer cluster depends on the total number of Indexers in the cluster. For more information, see the configuremasters CLI command.
The following table summarizes the scenarios when you can experience data loss due to an Indexer becoming unavailable.
|If a peer goes down|
The indexing and search functions continue to work, but you can experience partial data loss.
|Ensure that index replication is enabled.|
|If a master goes down|
The indexing and search functions stop working and you can experience data loss.
|Enable more than one Indexer to act as master in the cluster by running the configuremasters CLI command.|
Indexer redundancy means that if one Indexer goes down, the remaining Indexers in the cluster take up the job of the first Indexer. This means data indexing requests coming from the Collection Station and search requests coming from the Search components are taken up by the remaining Indexers. In this way, data collected continues to be indexed and data indexed continues to be searchable.
Indexer redundancy also provides the capability of enabling index replication, a process that allows you to store multiple copies of the data indexed across various Indexers. In case of a failure, replication enables the copy (or replica) of the data held by the Indexer (that just went down) to remain available for search. This process minimizes the possibilities of data loss.
You can configure Indexers in your environment such that each Indexer's data is replicated on other Indexer nodes. This means when an Indexer indexes data, a replica or an identical copy of that data is immediately created on other Indexer nodes. You can configure the number of replicas that you want to maintain, also known as the index replication factor. The replication factor can be set while enabling redundancy.
The degree of resiliency that the Indexer cluster possesses is directly proportional to the number of copies of the data it maintains. However, to maintain additional copies of the data, you need additional storage. You can control the number of copies of the data to be maintained by changing the index replication factor.
Depending on the data availability needs of your business, you need to manage the trade-off between the benefits of data availability and the cost of storage, and an increased processing load. For more information, see Sizing drivers and their impact.
The following figure depicts a three-node Indexer cluster (in a horizontally scaled environment) with primary and replica shards (index replication factor set to 1).
In the preceding figure, you can see the allocation of primary shards and corresponding replica shards. For example, primary shard P0 located on Node 1 has a replica R0 on Node 2. Similarly, primary shard P2 located on Node 3 has a replica R2 located on Node 2. The main difference between primary and replica shards is that data indexing requests can only be accepted by the primary shards. However, both primary and replica shards can serve search requests. Thus, by maintaining multiple copies of the data, not only do you protect against data loss, but also increase search performance. When multiple users make search requests on the data indexed, these requests are served by pulling data from the primary shards or replica shards.
Before you enable redundancy, ensure that you have more than one Indexer deployed in your environment.
Indexer redundancy is applied dynamically while the Indexers are functioning in the Indexer cluster. You need to ensure that all the Indexers in your environment are up and running. To check whether all the Indexers are in the running state, navigate to the Administration > Components page. If the master Indexer goes down, the Indexer cluster goes down. Therefore, to decrease the possibility of the cluster going down, it is recommended that you enable more than one Indexer to act as master in the cluster. To do this, run the configuremasters CLI command.
To enable (or disable) Indexer redundancy, you need to run the configureredundancy CLI command. While running this command, you need to set the index replication factor (default value set to 1). The default value of the index replication factor indicates that by enabling redundancy, atleast one copy of the data indexed is automatically replicated. You can optionally change the replication factor to a value greater than 1.
To achieve Indexer redundancy, setting the index replication factor is a must.
To check whether Indexer redundancy is successfully enabled, run the configureredundancy CLI command with the
Configureredundancy CLI doesn't have status option as mentioned at the bottom of this page please check and confirm.