This topic provides answers to some frequently-asked questions about performance and scalability.
For information about the various factors that affect deployment sizing, see Variables that affect sizing and scalability.
The storage required for your deployment is 1.25 times the volume of data (indexed per day) depending on the data retention period defined.
Storage required = 1.25 * Volume of data (per day) * Retention period (in days)
For more information, see Variables that affect sizing and scalability.
For information about the impact of increasing the retention period, see Sizing drivers and their impact.
There is no theoretical limit to the number of data collectors that can be configured on a single Collection Station or Collection Agent.
However, we have tested upto 1500 data collectors on a single Collection Station (or Collection Agent) on an independent server, that does not host the application for monitoring. BMC recommends that you do not cross this limit for optimum performance, based on the following scenarios:
The overhead of the Collection Agent on the target host is minimal.
During the performance tests, it was observed that the CPU overhead was around 2 - 6 % with 256 MB of RAM.
The following table provides some indicators regarding the performance tests carried out on a virtual setup using Intel® Xeon® CPU E5-2660 @ 2.20GHz processor with a storage of 300 IOPS.
Polling frequency (in minutes) | Average CPU utilization | Number of data collectors | Data collection rate |
---|---|---|---|
1 | 2% | 1 | 67 MB per day |
1 | 2% | 5 | 338 MB per day |
1 | 2% | 10 | 677 MB per day |
1 | 4% | 10 | 1.4 GB per day |
1 | 6% | 10 | 2.7 GB per day |
By default, the product stores the collected data in time-based indices of 6 hours each. In each index, the metadata is stored in the memory to optimize the searches. The additional RAM requirement arises due to this metadata that is stored in the Indexer memory.
For more information, see Variables that affect sizing and scalability.
The browser response time can be slow due to the following reasons:
For more information, see:
The Java heap sizes of the component processes (including for the Configuration Database) might be over-allocating the available RAM.
For more information about configuration recommendations for various components, see Component configuration recommendations.
The search performance can be impacted due to various reasons as described in the following table:
Reason | Solution |
---|---|
High number of fields in the Filter Pane | Reduce the number of fields in the Filter Pane |
High cardinality fields (fields with a large number of unique values) in the Filter Pane | Remove the high cardinality fields from the Filter Pane |
High time range of the search query | Reduce the time range of the search query |
Navigate to the agent.properties file of the Collection Station and increase the collection.thread.pool.size
property value to 40. For more information, see Modifying the configuration files.
If the collection.thread.pool.size
property is already set at 40 and the system seems to be running at full capacity, you can try increasing the polling interval.
You can increase the maximum Java heap size in the server.conf file. For more information, see Component configuration recommendations.
Out Of Memory Error
for the Java heap size in the Indexer logs. What can I do to prevent this error from recurring?You can perform the following steps:
intervalInHrs
property is set to either 2 or 1 (instead of the default 6).Remove high cardinality fields (fields with a large number of unique values) from the Filter Pane.
Reduce the number of concurrent searches
Reduce the data retention period
For more information, see Component configuration recommendations.
For information about the possible steps you can take, see Troubleshooting performance issues.
The number of events that can be indexed is a factor of the average size of each event. In our performance tests, we focus on the size of the data indexed, because the size of the data is a widely known unit of data that is generated by applications. For example, 100 GB data per day on a reference hardware server.
The data might take longer than the expected time if the system is heavily loaded due to a large number of collectors. At times if the system is stabilizing after a downtime, it attempts to index old data that remained pending in the Collection Station. In such a case, the data will take longer than expected, before it is available for searching.
No, it is not mandatory to create a data pattern before indexing data.
The product offers a list of default data patterns for most of the common log formats that you can use while creating a data collector. For more information, see Default data patterns.
In case there is no matching data pattern, the product tries to identify the matching time stamp format (and treats all other data as raw data). Alternatively, you can also index data as free text, in which case no time stamp is extracted from the log file, but the time of indexing of data is associated with the entries. For more information, see Managing data patterns.
The network bandwidth requirements can vary depending on the data generation rate and the type of data collectors used. The network utilization calculation is the same irrespective of the volume of data generated.
For example, for 100 Kb of data generated in one second, the network bandwidth required is approximately 90 Kbps (for data collected using a Collection Agent).
The following formulas illustrate the network bandwidth calculation:
Local file collection using Collection Agent: 0.9 * Data Transfer Rate (or Data Volume)
Data flow |
---|
Target host → Collection Station |
Target host → Collection Agent → Collection Station |
SSH / Windows Share collection: 1.8 * Data Transfer Rate (or Data Volume)
Data flow |
---|
Target host → Collection Station |
Target host → Collection Agent → Collection Station |