Archiving and restoring logs
By default, logs are retained for specific number of days (depending on your license entitlements) and are available for analysis. To retain the logs for a longer duration for on-demand analysis, compliance, or other purposes, as an administrator, archive the logs and move them to cold storage.
Life cycle of logs
The following important terms will help you understand the life cycle of the collected logs:
- Retention period—Time for which logs are available for analysis, after which they are archived. During this period, the logs are saved in the hot storage where they are indexed for search. You can analyze and search the logs. When the retention period is over, logs are moved to the cold storage or are archived. At the same time, the search indexes are also removed from these logs that means you cannot access the archived logs on the Discover page.
- Archive period—Time period for which logs are retained and archived, after which they are purged. During this period, logs are stored in the cold storage and search indexes are removed. Logs are archived for the time defined in the archive period that starts from the log collection date. Therefore, you cannot access the archived logs on the Discover page.
When the archive period is over, logs are purged and cannot be restored. - Restore period—Time period for which logs are restored for analyzing, after which they are autoarchived. When the logs are archived, you can restore them for analysis. Restored logs are available in the Explorer for analysis. After the restore period is over, logs are archived automatically. However, you can archive the restored logs manually to free up the space to restore other logs if the maximum data storage limit is reached.
The retention, archive, and restore period depend upon your license entitlement and are shown to you on the Configurations > Archive & Restore page.
The following image illustrates logs life cycle:
Watch this video (3:16) to understand the archiving and restoring feature.
Changes to index pattern after archiving is enabled
Logs are archived in a log index with a unique name. 100 GB data is stored in an index. If more than 100 GB data is collected in a tenant in a day, multiple indices are created for a particular date. You can identify the archived data by viewing the Date column that shows the date on which the logs were collected. You require this date to determine which log index you want to restore.
After archiving is enabled, a new index pattern is added whose format is logarc_*. All the logs collected since the time archiving is enabled for your tenant are shown in the new index pattern. The data before enabling archiving continues to show in the earlier index pattern. Archived and restored data are available in the new index pattern only. Therefore, to analyze logs collected after archiving is enabled, use the logarc_* index pattern.
To restore archived logs
- Click Configurations > Archive & Restore.
- To search the logs you want to restore, use the Date column or the Search Data field.
- Select the check box for the archived logs index, and select Actions > Restore.
- To restore multiple archived logs, select the check boxes for those log indices and click the Restore button.
Automatic archiving of restored logs
Logs that you restore remain available for analysis for the restore period. When the restore period is over, logs are archived automatically. The number of days in which the logs will be autoarchived are shown in the Autoarchived Days column. This value is also based on your license entitlement.
For example, the restore period for your tenant is 5 days. You restored a log index on June 1st. This log index will be autoarchived on June 6th.
In some cases, the autoarchived period can be less than the restore period for your tenant. This occurs because the archive period for the particular log index gets over before the restore period. For example, you restored a log index on June 1st. The restore period is 5 days and it should be autoarchived in 5 days, on June 6th. However, you see that 3 is showing in the Autoarchived Days column. It is because the log index that you restored is due to be purged in 3 days, on June 4th.
To archive restored logs manually
- Click Configurations > Archive & Restore.
- To search the logs that you want to archive, use the Date column or the Search Data field.
- Select the check box for the restored logs, and select Actions > Archive.
- To archive multiple restored logs, select the check boxes for those log indices and click the Archive button.
- To verify that the logs are archived, on the Discover page, in the logarc_* index pattern, search for the logs in the time range that you just archived.
If no results are shown, logs are archived correctly.
Archiving logs in SaaS
For SaaS deployments, the feature is available on Amazon Web Services tenants. Logs are stored for search and analysis in the hot and ultrawarm storages. When the logs are archived, they are moved to the cold storage.
Where to go from here
Learn more
Read the following blog to learn more about gaining full visibility by archiving logs Archive logs to optimize storage & gain full visibility.