Long-term data retention recommendations

Use the following hardware configuration recommendations if you want to retain data for a longer period as compared to the recommendations available for horizontal scaling and vertical scaling of the product.

These recommendations can also be useful if you want to run tabular commands on a bigger set of data. For more information about tabular commands, see Search-commands.

Related topics

Variables that affect sizing and scalability

Sizing-drivers-and-their-impact

Notes

  • These recommendations are aimed at providing a 90-day data retention period and are meant for reference purposes only.
  • These recommendations are based on a data indexing rate of 100 GB per day.
  • Each time you double the data indexing rate, you need to multiply the recommended RAM by 1.5 times.
  • The maximum memory required for an Indexer is recommended to be less than 32 GB. Therefore, if you cross the limit of 32 GB, it is recommended that you add additional Indexers to your setup.
  • If you enable Indexer-redundancy in your environment, you need double the resources stated in this topic. This means you need to add additional servers with the same configuration as that recommended in this topic for various kinds of setups.
Example: Identifying the required RAM

Suppose you plan to index 200 GB data per day and suppose you are operating in a single Indexer setup. As per the recommendations in this topic, for a single Indexer setup the recommended RAM equals 32 GB. Therefore the RAM required to index 200 GB per day can be calculated as 32 GB * 1.5 = 48 GB.

However, the required RAM exceeds the limit of 32 GB. Therefore, in this scenario, it is recommended that you add an additional Indexer to your setup.

Refer to the following configuration recommendations for achieving a 90-day data retention period:

Understanding the configuration recommendations

A higher data retention impacts the following resources:

  • CPU
  • RAM
  • Disk space
  • Maximum memory required for the Indexer

This topic covers recommendations for a multiple-server deployment with the Indexer component scaled-out. For more information, see the following links:

The recommended setups consider the following distribution of components:

  • One server hosting the Console Server, Search, Collection Station, and Configuration Database components.
  • Additional servers with one Indexer per server.
    The number of Indexers required can be determined by the size of your environment (medium or large setup). For more information, see Horizontal-scaling-recommendations.

As the number of servers hosting the Indexer component increases, the disk space, RAM, and maximum memory required for the Indexer component reduces proportionately.

Setup with one Indexer server

The following table lists the configurations recommendations for a setup with one Indexer.

1 server hosting 1 Indexer

The [liveData] macro is a standalone macro and it cannot be used inline. Click on this message for details.

The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 27 GB.

For more information, see Component-configuration-recommendations-for-horizontal-scaling.

Setup with two Indexer servers

The following table lists the configurations recommendations for a setup with two Indexers.

2 servers hosting 1 Indexer each

The [liveData] macro is a standalone macro and it cannot be used inline. Click on this message for details.

The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 15 GB.

For more information, see Component-configuration-recommendations-for-horizontal-scaling.

Setup with five Indexer servers

The following table lists the configurations recommendations for a setup with five Indexers.

5 server hosting 1 Indexer each

The [liveData] macro is a standalone macro and it cannot be used inline. Click on this message for details.

The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 10 GB.

For more information, see Component-configuration-recommendations-for-horizontal-scaling.

Setup with all other components

The following table lists the configurations recommendations for a setup with one instance of the Console Server, Search, Collection Station, and Configuration Database components.

1 server hosting Console Server, Search, Collection Station, and Configuration Database

The [liveData] macro is a standalone macro and it cannot be used inline. Click on this message for details.

Notes to keep in mind

This note is applicable only if you have configured Collection Agents by using the TrueSight console or the PATROL infrastructure. For optimum performance of such Collection Agents, you need to change certain properties based on the number of Collection Agents set up in your environment.

The following properties need to be modified:

  • http.server.workers.thread.pool.size: Determines the maximum number of concurrent HTTP requests that can be handled by the Collection Station.
    This property needs to be added with the new values at %BMC_ITDA_HOME%\station\collection\custom\conf\agent.properties.
  • http.response.timeout.millis: Determines the period after which the Configuration Channel request must timeout.
    This property needs to be added with the new values at %BMC_ITDA_HOME%\agent\collection\custom\conf\agent.properties.
  • agent.request.poll.heartbeat.per.node.retry: Determines the number of failed HTTP requests before accepting that the Collection Station is not reachable.
    This property needs to be added with the new values at %BMC_ITDA_HOME%\agent\collection\custom\conf\agent.properties.

Use the following table as a guideline to determine the preceding property values. These values are based on internal performance tests.

Note

 If you are operating in an environment with 950-1750 Collection Agents, then at a minimum you need two Collection Stations setup on different servers to receive data from those Collection Agents.