Long-term data retention recommendations



Use the following hardware configuration recommendations if you want to retain data for a longer period as compared to the recommendations available for horizontal scaling and vertical scaling of the product.

These recommendations can also be useful if you want to run tabular commands on a bigger set of data. For more information about tabular commands, see Search commands

Notes

  • These recommendations are aimed at providing a 90-day data retention period and are meant for reference purposes only.
  • These recommendations are based on a data indexing rate of 100 GB per day.

  • For long-term data retention, it is recommended that the Retention Days should be set as 90 days for Large Index Blocks. For more information on Index Blocks, see Understanding Indexers, Indexes and Index Blocks

  • Eaich time you double the data indexing rate, you need to multiply the recommended RAM by 1.5 times.
  • The maximum memory required for an Indexer is recommended to be less than 32 GB. Therefore, if you cross the limit of 32 GB, it is recommended that you add additional Indexers to your setup.
  • If you enable Indexer-redundancy in your environment, you need double the resources stated in this topic. This means you need to add additional servers with the same configuration as that recommended in this topic for various kinds of setups.
Example: Identifying the required RAM

Suppose you plan to index 200 GB data per day and suppose you are operating in a single Indexer setup. As per the recommendations in this topic, for a single Indexer setup the recommended RAM equals 32 GB. Therefore the RAM required to index 200 GB per day can be calculated as 32 GB * 1.5 = 48 GB.

However, the required RAM exceeds the limit of 32 GB. Therefore, in this scenario, it is recommended that you add an additional Indexer to your setup.

Refer to the following configuration recommendations for achieving a 90-day data retention period:

Related topics

Variables that affect sizing and scalability

Sizing-drivers-and-their-impact

Understanding the configuration recommendations

A higher data retention impacts the following resources:

  • CPU
  • RAM
  • Disk space
  • Maximum memory required for the Indexer

This topic covers recommendations for a multiple-server deployment with the Indexer component scaled-out. For more information, see the following links:

The recommended setups consider the following distribution of components:

  • One server hosting the Console Server, Search, Collection Station, and Configuration Database components.
  • Additional servers with one Indexer per server.
    The number of Indexers required can be determined by the size of your environment (medium or large setup). For more information, see Horizontal-scaling-recommendations.

As the number of servers hosting the Indexer component increases, the disk space, RAM, and maximum memory required for the Indexer component reduces proportionately.

Setup with one Indexer server

The following table lists the configurations recommendations for a setup with one Indexer.

1 server hosting 1 Indexer

The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 27 GB.

For more information, see Component-configuration-recommendations-for-horizontal-scaling.


Setup with two Indexer servers

The following table lists the configurations recommendations for a setup with two Indexers.

2 servers hosting 1 Indexer each

The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 15 GB.

For more information, see Component-configuration-recommendations-for-horizontal-scaling.


Setup with five Indexer servers

The following table lists the configurations recommendations for a setup with five Indexers.

5 server hosting 1 Indexer each

The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 10 GB.

For more information, see Component-configuration-recommendations-for-horizontal-scaling.

Setting up Archive Indexer nodes

The following table lists the configurations recommendations for setting up Archive Indexer nodes.

The number of Archive nodes must be equal to or more than the number of Live nodes.

Setting up Restore Indexer node

The following table lists the configurations recommendations for setting up Restore Indexer nodes.

The number of Restore nodes depends on the amount of data being restored from the Live nodes.

Data archiving storage requirements

Archive disk storage=1.3 times size of the original data you want to archive.
 So, If you want to archive one year's worth of data and are indexing 100 GB per day, then the disk archive storage space= 100 GB/day * 365 days * 1.3.

Setup with all other components

The following table lists the configurations recommendations for a setup with one instance of the Console Server, Search, Collection Station, and Configuration Database components.

1 server hosting Console Server, Search, Collection Station, and Configuration Database


Notes to keep in mind

This note is applicable only if you have configured Collection Agents by using the TrueSight console or the PATROL infrastructure. For optimum performance of such Collection Agents, you need to change certain properties based on the number of Collection Agents set up in your environment.

The following properties need to be modified:

  • http.server.workers.thread.pool.size: Determines the maximum number of concurrent HTTP requests that can be handled by the Collection Station.
    This property needs to be added with the new values at %BMC_ITDA_HOME%\station\collection\custom\conf\agent.properties.
  • http.response.timeout.millis: Determines the period after which the Configuration Channel request must timeout.
    This property needs to be added with the new values at %BMC_ITDA_HOME%\agent\collection\custom\conf\agent.properties.
  • agent.request.poll.heartbeat.per.node.retry: Determines the number of failed HTTP requests before accepting that the Collection Station is not reachable.
    This property needs to be added with the new values at %BMC_ITDA_HOME%\agent\collection\custom\conf\agent.properties.

Use the following table as a guideline to determine the preceding property values. These values are based on internal performance tests.

Note

 If you are operating in an environment with 950-1750 Collection Agents, then at a minimum you need two Collection Stations setup on different servers to receive data from those Collection Agents.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*