Long-term data retention recommendations



Use the following hardware configuration recommendations if you want to retain data for a longer period as compared to the recommendations available for horizontal scaling and vertical scaling of the product.

These recommendations can also be useful if you want to run tabular commands on a bigger set of data. For more information about tabular commands, see Search commands

Notes

  • These recommendations are aimed at providing a 90-day data retention period and are meant for reference purposes only.
  • These recommendations are based on a data indexing rate of 100 GB per day.
  • For long-term data retention, it is recommended that the Retention Days should be set as 90 days for Large Index Blocks. For more information on Index Blocks, see Understanding Indexers, Indexes and Index Blocks
  • Eaich time you double the data indexing rate, you need to multiply the recommended RAM by 1.5 times.
  • The maximum memory required for an Indexer is recommended to be less than 32 GB. Therefore, if you cross the limit of 32 GB, it is recommended that you add additional Indexers to your setup.
  • If you enable Indexer redundancy in your environment, you need double the resources stated in this topic. This means you need to add additional servers with the same configuration as that recommended in this topic for various kinds of setups.

Example: Identifying the required RAM

Suppose you plan to index 200 GB data per day and suppose you are operating in a single Indexer setup. As per the recommendations in this topic, for a single Indexer setup the recommended RAM equals 32 GB. Therefore the RAM required to index 200 GB per day can be calculated as 32 GB * 1.5 = 48 GB.

However, the required RAM exceeds the limit of 32 GB. Therefore, in this scenario, it is recommended that you add an additional Indexer to your setup.

Refer to the following configuration recommendations for achieving a 90-day data retention period:

Understanding the configuration recommendations

A higher data retention impacts the following resources:

  • CPU
  • RAM
  • Disk space
  • Maximum memory required for the Indexer

This topic covers recommendations for a multiple-server deployment with the Indexer component scaled-out. For more information, see the following links:

The recommended setups consider the following distribution of components:

  • One server hosting the Console Server, Search, Collection Station, and Configuration Database components.
  • Additional servers with one Indexer per server.
    The number of Indexers required can be determined by the size of your environment (medium or large setup). For more information, see Horizontal scaling recommendations.

As the number of servers hosting the Indexer component increases, the disk space, RAM, and maximum memory required for the Indexer component reduces proportionately.

Setup with one Indexer server

The following table lists the configurations recommendations for a setup with one Indexer.

1 server hosting 1 Indexer

ResourceRecommendation
CPU8
RAM32 GB
Disk space12 TB HDD
x 1 server

The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 27 GB.

For more information, see Component configuration recommendations for horizontal scaling.


Setup with two Indexer servers

The following table lists the configurations recommendations for a setup with two Indexers.

2 servers hosting 1 Indexer each

ResourceRecommendation
CPU8
RAM21 GB
Disk space6 TB HDD
x 2 servers

The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 15 GB.

For more information, see Component configuration recommendations for horizontal scaling.

Setup with five Indexer servers

The following table lists the configurations recommendations for a setup with five Indexers.

5 server hosting 1 Indexer each

ResourceRecommendation
CPU8
RAM16 GB
Disk space2.4 TB HDD
x 5 servers

The preceding details requires the maximum memory required for the Indexer (wrapper.java.maxmemory property) to be set to 10 GB.

For more information, see Component configuration recommendations for horizontal scaling.

Setting up Archive Indexer nodes

The following table lists the configurations recommendations for setting up Archive Indexer nodes.

ResourceRecommendation
CPU8
RAM16 GB
Disk space

1 TB HDD

(For 400 GB data ingestion per day and with retention period of 2 days.)

Note: The disk space required on Archive node depends on the following:

  • Amount of data ingestion per day
  • Retention period
x 1 servers

The number of Archive nodes must be equal to or more than the number of Live nodes.

Setting up Restore Indexer node

The following table lists the configurations recommendations for setting up Restore Indexer nodes.

ResourceRecommendation
CPU8
RAM16 GB
Disk space

Depends on the amount of data that is being restored from the Live nodes.

x 1 servers

The number of Restore nodes depends on the amount of data being restored from the Live nodes.

Data archiving storage requirements

Archive disk storage=1.3 times size of the original data you want to archive.
So, If you want to archive one year's worth of data and are indexing 100 GB per day, then the disk archive storage space= 100 GB/day * 365 days * 1.3.

Setup with all other components

The following table lists the configurations recommendations for a setup with one instance of the Console Server, Search, Collection Station, and Configuration Database components.

1 server hosting Console Server, Search, Collection Station, and Configuration Database

ResourceRecommendation
CPU8
RAM16 GB
Disk space1 TB HDD
x 1 server

Notes to keep in mind

This note is applicable only if you have configured Collection Agents by using the TrueSight console or the PATROL infrastructure. For optimum performance of such Collection Agents, you need to change certain properties based on the number of Collection Agents set up in your environment.

The following properties need to be modified:

  • http.server.workers.thread.pool.size: Determines the maximum number of concurrent HTTP requests that can be handled by the Collection Station.
    This property needs to be added with the new values at %BMC_ITDA_HOME%\station\collection\custom\conf\agent.properties.
  • http.response.timeout.millis: Determines the period after which the Configuration Channel request must timeout.
    This property needs to be added with the new values at %BMC_ITDA_HOME%\agent\collection\custom\conf\agent.properties.

  • agent.request.poll.heartbeat.per.node.retry: Determines the number of failed HTTP requests before accepting that the Collection Station is not reachable.
    This property needs to be added with the new values at %BMC_ITDA_HOME%\agent\collection\custom\conf\agent.properties.

Use the following table as a guideline to determine the preceding property values. These values are based on internal performance tests.

Number of Collection Agentshttp.server.workers.
thread.pool.size
http.response.
timeout.millis
agent.request.poll.
heartbeat.per.node.retry
Upto 450(Default) 15(Default) 30,000(Default) 10
More than 450 and upto 9506060,00030
More than 950 and upto 1750
Read more
60 60,00030

Note

If you are operating in an environment with 950-1750 Collection Agents, then at a minimum you need two Collection Stations setup on different servers to receive data from those Collection Agents.

Was this page helpful? Yes No Submitting... Thank you

Comments