Long-term data retention recommendations
Use the following hardware configuration recommendations if you want to retain data for a longer period as compared to the recommendations available for horizontal scaling and vertical scaling of the product.
These recommendations can also be useful if you want to run tabular commands on a bigger set of data. For more information about tabular commands, see Search commands.
Notes
- These recommendations are aimed at providing a 90-day data retention period and are meant for reference purposes only.
- These recommendations are based on a data indexing rate of 100 GB per day.
- Each time you double the data indexing rate, you need to multiply the recommended RAM by 1.5 times.
- The maximum memory required for an Indexer is recommended to be less than 32 GB. Therefore, if you cross the limit of 32 GB, it is recommended that you add additional Indexers to your setup.
- If you enable Indexer redundancy in your environment, you need double the resources stated in this topic. This means you need to add additional servers with the same configuration as that recommended in this topic for various kinds of setups.
Example: Identifying the required RAM
Suppose you plan to index 200 GB data per day and suppose you are operating in a single Indexer setup. As per the recommendations in this topic, for a single Indexer setup the recommended RAM equals 32 GB. Therefore the RAM required to index 200 GB per day can be calculated as 32 GB * 1.5 = 48 GB.
However, the required RAM exceeds the limit of 32 GB. Therefore, in this scenario, it is recommended that you add an additional Indexer to your setup.
Refer to the following configuration recommendations for achieving a 90-day data retention period:
Understanding the configuration recommendations
A higher data retention impacts the following resources:
- CPU
- RAM
- Disk space
- Maximum memory required for the Indexer
This topic covers recommendations for a multiple-server deployment with the Indexer component scaled-out. For more information, see the following links:
- To understand how Indexers functions in a multiple-server deployment, see Indexer redundancy.
- To understand how to deploy Indexers in a multiple-server deployment, see Deploying multiple Collection Stations and Indexers.
The recommended setups consider the following distribution of components:
- One server hosting the Console Server, Search, Collection Station, and Configuration Database components.
- Additional servers with one Indexer per server.
The number of Indexers required can be determined by the size of your environment (medium or large setup). For more information, see Horizontal scaling recommendations.
As the number of servers hosting the Indexer component increases, the disk space, RAM, and maximum memory required for the Indexer component reduces proportionately.
Setup with one Indexer server
The following table lists the configurations recommendations for a setup with one Indexer.
1 server hosting 1 Indexer
| Resource | Recommendation |
|---|---|
| CPU | 8 |
| RAM | 32 GB |
| Disk space | 12 TB HDD |
| x 1 server | |
The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 27 GB.
For more information, see Component configuration recommendations for horizontal scaling.
Setup with two Indexer servers
The following table lists the configurations recommendations for a setup with two Indexers.
2 servers hosting 1 Indexer each
| Resource | Recommendation |
|---|---|
| CPU | 8 |
| RAM | 21 GB |
| Disk space | 6 TB HDD |
| x 2 servers | |
The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 15 GB.
Setup with five Indexer servers
The following table lists the configurations recommendations for a setup with five Indexers.
5 server hosting 1 Indexer each
| Resource | Recommendation |
|---|---|
| CPU | 8 |
| RAM | 16 GB |
| Disk space | 2.4 TB HDD |
| x 5 servers | |
The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 10 GB.
Setup with all other components
The following table lists the configurations recommendations for a setup with one instance of the Console Server, Search, Collection Station, and Configuration Database components.
1 server hosting Console Server, Search, Collection Station, and Configuration Database
| Resource | Recommendation |
|---|---|
| CPU | 8 |
| RAM | 16 GB |
| Disk space | 1 TB HDD |
| x 1 server | |
Notes to keep in mind
This note is applicable only if you have configured Collection Agents by using the TrueSight console or the PATROL infrastructure. For optimum performance of such Collection Agents, you need to change certain properties based on the number of Collection Agents set up in your environment.
The following properties need to be modified:
http.server.workers.thread.pool.size: Determines the maximum number of concurrent HTTP requests that can be handled by the Collection Station.
This property needs to be added with the new values at %BMC_ITDA_HOME%\station\collection\custom\conf\agent.properties.http.response.timeout.millis: Determines the period after which the Configuration Channel request must timeout.
This property needs to be added with the new values at %BMC_ITDA_HOME%\agent\collection\custom\conf\agent.properties.agent.request.poll.heartbeat.per.node.retry: Determines the number of failed HTTP requests before accepting that the Collection Station is not reachable.
This property needs to be added with the new values at %BMC_ITDA_HOME%\agent\collection\custom\conf\agent.properties.
Use the following table as a guideline to determine the preceding property values. These values are based on internal performance tests.
| Number of Collection Agents | http.server.workers. | http.response. | agent.request.poll. |
|---|---|---|---|
| Upto 450 | (Default) 15 | (Default) 30,000 | (Default) 10 |
| More than 450 and upto 950 | 60 | 60,000 | 30 |
| More than 950 and upto 1750 Read more | 60 | 60,000 | 30 |
Note
If you are operating in an environment with 950-1750 Collection Agents, then at a minimum you need two Collection Stations setup on different servers to receive data from those Collection Agents.
Comments
Log in or register to comment.