Indexer redundancy
Indexer redundancy can help you achieve the following main goals:
- Minimize potential data loss if an Indexer goes down and thereby enable search requests to be continuously serviced.
- Protect the indexed data against various scenarios of data loss by keeping multiple copies of the data stored (by enabling index replication).
Applying Indexer redundancy provides an added benefit of better search response times with high number of concurrent users.
Recommendation
A minimum of 3 Indexers are recommended for applying redundancy.
The following information can help you understand how Indexer redundancy works and how it can be enabled in your environment.
Indexer redundancy overview
The Indexer indexes data received from the Collection Stations to make the data search-ready. If the Indexer goes down, you will start losing data even though the data was successfully collected. Furthermore, data lost will not be searchable. Also, in case of disk failures, you can lose data.
Indexer redundancy covers the following aspects:
- Maintaining the Indexer availability (in other words, data availability) so that indexing function continues (in other words, data continues to get indexed).
- Enabling index replication (where multiple copies of the data are stored) to protect against data loss.
The following diagram identifies the channels covered by maintaining the availability of Indexers.
How Indexers function
Before you understand how Indexer redundancy works, you need to understand how data is indexed and how Indexers function:
How data is indexed
After data is collected, it is added to an index on an Indexer node. An index contains a number of shards and all the data indexed is stored in the shards. An index is a high-level place that points to the location where particular data might be stored. A shard is a low level unit of an index that holds a small portion of all the data in the index.
A shard can be a primary shard or a replica shard. Each document in the index is stored in a single primary shard. The total number of primary shards determines the maximum amount of data that an index can hold. A replica shard is an identical copy of the primary shard. Multiple replicas can be created for a single primary shard. For more information, see Index replication.
About the Indexer cluster
All Indexers deployed in your environment form part of a cluster. Data collected is stored across all the Indexers.
In a single Indexer cluster, by default, that Indexer functions as the master. In a multiple Indexer cluster, you can have more than one Indexer functioning as the master.
If you are operating in a multiple-server deployment, the Indexer that you install for the first time (on the primary node) must be the master. All other Indexers are considered as peers and they must be provided with details of the master. The master plays an important role of the coordinator. Deploying multiple Indexers in your environment can be helpful to support better performance, longer data retention period, and continued data availability.
About the master Indexer
The master Indexer communicates and coordinates with other peer Indexers. The master Indexer is like any other member of the Indexer cluster. This means like any other peer, the master continues to index data.
Any member of the Indexer cluster can play the role of a master, however at any given time, the Indexer cluster can have only one Indexer playing the role of a master. This means that when you configure more than one Indexer to act as the master, those Indexers are master-enabled (in other words, enabled to act as masters). If the current master goes down, the next master-enabled Indexer is elected to act as the new master. You can configure masters by running the configuremasters CLI command. This command can be also used to modify masters in the Indexer cluster.
Note
If the master goes down, the cluster goes down, resulting in data loss. To avoid data loss, you must enable more than one Indexer to act as master in the cluster.
What happens if an Indexer goes down
If a peer Indexer goes down, data already indexed by that Indexer remains unavailable (partial data loss). When a peer goes offline temporarily, the data held by the peer is not available for search, for the time when the peer is offline. The peer can go offline due to various reasons, for example, the node on which the peer is deployed goes down. After the peer comes online, the past data held by the peer becomes available for search. But during the time when the peer is offline, the data collectors are unable to connect and send data (for indexing) to the peer. Data sent to the peer in the time when it was offline is lost. To avoid data loss in such a scenario, you need to enable index replication, a process where a copy of the indexed data is stored on multiple Indexer nodes. For more information, see Index replication.
If the master Indexer goes down, the Indexer cluster goes down, thereby resulting in data loss. To avoid such a scenario, you need to enable more than one Indexer to act as master in the cluster. With this, if one master goes down, the next master-enabled Indexer assumes the responsibility of the master. In this way, the Indexer cluster remains up and data loss is avoided. Determining the number of masters required in an Indexer cluster depends on the total number of Indexers in the cluster. For more information, see the configuremasters CLI command.
The following table summarizes the scenarios when you can experience data loss due to an Indexer becoming unavailable.
| Scenario | Impact | Solution |
|---|---|---|
| If a peer goes down | The indexing and search functions continue to work, but you can experience partial data loss. | Ensure that index replication is enabled. |
| If a master goes down | The indexing and search functions stop working and you can experience data loss. | Enable more than one Indexer to act as master in the cluster by running the configuremasters CLI command. |
How Indexer redundancy works
Indexer redundancy means that if one Indexer goes down, the remaining Indexers in the cluster take up the job of the first Indexer. This means data indexing requests coming from the Collection Station and search requests coming from the Search components are taken up by the remaining Indexers. In this way, data collected continues to be indexed and data indexed continues to be searchable.
Indexer redundancy also provides the capability of enabling index replication, a process that allows you to store multiple copies of the data indexed across various Indexers. In case of a failure, replication enables the copy (or replica) of the data held by the Indexer (that just went down) to remain available for search. This process minimizes the possibilities of data loss.
Index replication
You can configure Indexers in your environment such that each Indexer's data is replicated on other Indexer nodes. This means when an Indexer indexes data, a replica or an identical copy of that data is immediately created on other Indexer nodes. You can configure the number of replicas that you want to maintain, also known as the index replication factor. The replication factor can be set while enabling redundancy.
The degree of resiliency that the Indexer cluster possesses is directly proportional to the number of copies of the data it maintains. However, to maintain additional copies of the data, you need additional storage. You can control the number of copies of the data to be maintained by changing the index replication factor.
Note
Depending on the data availability needs of your business, you need to manage the trade-off between the benefits of data availability and the cost of storage, and an increased processing load. For more information, see Sizing drivers and their impact.
The following figure depicts a three-node Indexer cluster (in a horizontally scaled environment) with primary and replica shards (index replication factor set to 1).
In the preceding figure, you can see the allocation of primary shards and corresponding replica shards. For example, primary shard P0 located on Node 1 has a replica R0 on Node 2. Similarly, primary shard P2 located on Node 3 has a replica R2 located on Node 2. The main difference between primary and replica shards is that data indexing requests can only be accepted by the primary shards. However, both primary and replica shards can serve search requests. Thus, by maintaining multiple copies of the data, not only do you protect against data loss, but also increase search performance. When multiple users make search requests on the data indexed, these requests are served by pulling data from the primary shards or replica shards.
How to enable Indexer redundancy
Before you enable redundancy, ensure that you have more than one Indexer deployed in your environment.
Indexer redundancy is applied dynamically while the Indexers are functioning in the Indexer cluster. You need to ensure that all the Indexers in your environment are up and running. To check whether all the Indexers are in the running state, navigate to the Administration > Components page. If the master Indexer goes down, the Indexer cluster goes down. Therefore, to decrease the possibility of the cluster going down, it is recommended that you enable more than one Indexer to act as master in the cluster. To do this, run the configuremasters CLI command.
To enable (or disable) Indexer redundancy, you need to run the configureredundancy CLI command. While running this command, you need to set the index replication factor (default value set to 1). The default value of the index replication factor indicates that by enabling redundancy, atleast one copy of the data indexed is automatically replicated. You can optionally change the replication factor to a value greater than 1.
Best practice
To achieve Indexer redundancy, setting the index replication factor is a must.
To check whether Indexer redundancy is successfully enabled, run the configureredundancy CLI command with the status parameter.
Comments
Log in or register to comment.