Archiving and restoring logs

To retain logs for longer duration than the default period, archive the logs. You might want to retain the logs for a longer duration for on-demand analysis, compliance, or other purposes. As an administrator, when you archive the logs, they are moved to cold storage. In cold storage, search indexes are removed and you can analyze data on demand, at a lower cost.

Lifecycle of logs

The following important terms will help you understand the lifecycle of the collected logs:

• Retention period—Time for which logs are available for analysis, after which they are archived. During this period, the logs are saved in hot storage where they are indexed for search. You can analyze and search the logs. When the retention period is over, logs are moved to cold storage or are archived. At the same time, the search indexes are also removed from these logs, which means you cannot access the archived logs on the Discover page. 
• Archive period—Time period for which logs are retained and archived, after which they are purged. During this period, logs are stored in cold storage and search indexes are removed. Therefore, you cannot access the archived logs on the Discover page. Logs are archived for the time defined in the archive period, which starts from the log collection date.
  When the archive period is over, logs are purged and cannot be restored.
• Restore period—Time period for which logs are restored for analyzing, after which they are autoarchived. When the logs are archived, you can restore them for analysis. Restored logs are available in the Explorer for analysis. After the restore period is over, logs are archived automatically. However, you can archive the restored logs manually to free up the space to restore other logs if the maximum data storage limit is reached.

The retention, archive, and restore periods depend upon your license entitlement and are shown to you on the Archive & Restore page.

The following image illustrates the lifecycle of logs:

Watch this video (3:16) to understand the archiving and restoring capability.

Watch the YouTube video about the archiving and restoring capability of BMC Helix Log Analytics.

Changes to index pattern after archiving is enabled

Logs are archived in a log index with a unique name. An index stores 100 GB data. If more than 100 GB of data is collected in a tenant in a day, multiple indexes are created for that date. You can identify the archived data by viewing the Date column which shows the date on which the logs were collected. You require this date to determine which log index you want to restore.

After archiving is enabled, a new index pattern is added whose format is logarc_*.  All the logs that were collected since the time archiving was enabled for your tenant are shown in the new index pattern. The data that was collected before archiving was enabled continues to appear in the earlier index pattern. Archived and restored data are available in the new index pattern only. Therefore, to analyze logs collected after archiving is enabled, use the logarc_* index pattern.

To restore archived logs

1. Select Configurations > Archive & Restore.
2. Search for content in the logs you want to restore, use the Date column or the Search Data field.
   
3. Select the check box for the archived logs index, and select Actions > Restore.
4. Restore multiple archived logs by selecting the check boxes for those log indexes and clicking the Restore button.

Automatic archiving of restored logs

Logs that you restore remain available for analysis for the restore period. When the restore period is over, the restored logs are archived automatically. The number of days until the logs are autoarchived is shown in the Autoarchived Days column. This value is also based on your license entitlement.

For example, the restore period for your tenant is five days. You restored a log index on June 1st. This log index will be autoarchived on June 6th. 

In some cases, the autoarchived period can be less than the restore period for your tenant. This situation occurs because the archive period for the particular log index is over before the restore period ends. For example, you restored a log index on June 1st. The restore period is five days, so the log index will be autoarchived on June 6th. However, you see that a 3 is displayed in the Autoarchived Days column because the log index that you restored is due to be purged in three days, on June 4th.

To archive restored logs manually

1. Select Configurations > Archive & Restore.
2. Search for the logs that you want to archive, use the Date column or the Search Data field.
3. Select the check box for the restored logs, and select Actions > Archive.
4. Archive multiple restored logs by selecting the check boxes for those log indexes and click the Archive button.
5. To verify that the logs are archived, on the Discover page, in the logarc_* index pattern, search for the logs in the time range that you just archived.
   If no results are shown, the logs are archived correctly.

Archiving logs in different platforms

The feature is available on the following platforms and different storage options are used with each platform:

• SaaS: For SaaS deployments, the feature is available on Amazon Web Services and OCI tenants. Logs are stored for search and analysis in the hot and ultrawarm storages. When the logs are archived, they are moved to the cold storage.
• On-Premises: For on-premises deployments, the logs that can be searched are stored in Elasticsearch and archival is managed by using the MinIO storage.

Where to go from here

Deriving-insights-from-logs

Learn more

Read the following blog to learn how you can archive and retain logs for a longer time at a cheaper cost Archive logs to optimize storage & gain full visibility.