Understanding data retention and deletion

• Upload file used for uploading a file into the system and indexing all the data present in the file.
• Monitor Remote Windows Events used for collecting events generated by a Windows computer.
• Monitor using External Configuration used for collecting event data from external systems integrated with the product.

By default, the product defines the data retention period as seven days (since the default index block is Small). You can change this value for an individual data collector at the time of creating the data collector or for a bulk of data collectors on the Administration > Data Collectors page by selecting an index block. The upper limit for setting the maximum data retention limit is set to 30 days, but is configurable. For more information, see Changing system settings.

The data retention period acts as a moving window (depicted in green in the following figure).

Consider that on the following scale of time, you created a data collector at time T1, now data collection starts from T1 when the first poll happens. Data collected at T1 remains in the system until T1+7. As time passes, the data older than the seven days period starts getting deleted and is no longer available for searching.

Data deletion and retention in an upgrade scenario

In an upgrade scenario from earlier versions, data that has been indexed prior to version 11.3.01 continues to be deleted as per the data retention period defined prior to upgrading. However the data that is collected by the data collectors after the upgrade gets bucketed in index blocks according to the retention period that was defined. In such cases, data will be deleted as per the retention period with which its associated index block is configured after the upgrade.

Example:  Consider that there were five data collectors dc1, dc2, dc3, dc4 and dc5, having retention periods as 2, 7, 14, 28 and 35 days respectively, defined before upgrading from earlier versions of TrueSight IT Data Analytics.  So, the maximum retention was 35 days and the minimum was 2 days. Now, the default index blocks are defined as Small, Medium, and Large.

After the upgrade, when data is collected, data collectors will get associated with three index blocks that will get created with retention days as following:

• Small: Having the retention days as defined under Administration> System Settings page, say 7 days.
• Medium: Having retention days as (maximum retention days plus minimum retention days divided by two) rounded to the lower value (2+35)/2 that is, 18 days.
• Large: Having retention days as the maximum retention days of the data collectors that is, 35 days.

Hence, indexes from data collectors dc1 (retention period of 2 days) and dc2 (retention period  of 7 days) will get indexed in the Small index block.

Indexes from data collector dc3 (retention period 14 days) will get indexed in the Medium index block.

Indexes from the data collectors dc4 (retention period 28 days)  and dc5 (retention period 35 days) will get indexed in the Large index block.

So, we can say that when we are upgrading from an earlier version,

• All data collectors and collection profiles whose retention period is less than or equal to the retention period of the Small index block get associated with the Small index block
• All data collectors and collection profiles whose retention period is greater than retention period of the Small index block, and equal to or smaller than the retention period of the Medium index block get associated with the Medium index block.
• All data collectors and collection profiles whose retention period is greater than the retention period of the Medium index block, and equal to or smaller than the retention period of the Large index block get associated with the Large index block.
• A collection profile with retention period more than the retention period of the Large index block also gets associated with the Large index block.