Creating collection policies
Collection policies are predefined rules and configurations to manage log data ingestion from various sources. Use collection policies to optimize resource usage, maintain data quality, and gain valuable insights from their log data. Efficient data collection helps you to manage storage and processing costs by avoiding unnecessary data ingestion.
To start collecting logs, add all the collection-related configurations to a collection policy and save time by reusing these configurations in multiple collection policies.
The following image displays the configurations in a collection policy.
The following table describes the configurations that you can add in a collection policy.
Configuration | Description |
|---|---|
Collection type | BMC Helix Log Analyticseffectively gathers and ingests the log data from different sources in your environment. BMC Helix Log Analytics supports the following collection types:
|
Connector | Specify the type of connector and selection criteria that identifies the connector for collection. For more information, see Installing-and-managing-connectors. |
Log source | Specify the source of log collection like a file path or collection interval. These configurations differ for each source. |
Pre-filtering rule | Filter logs before parsing them, add a separate rule for the pre-filtering process in addition to the filtering process. By prefiltering logs, you limit the number of logs that are parsed and thereby enhance system performance. You can prefilter only the file type of logs. For more information, see Creating-a-filtering-rule. |
Parsing rule | Select the parsing rule you created to parse the logs you are collecting. If you have not created a parsing rule and directly move on to create a collection policy, you get a link to create a parsing rule from the collection policy page. For file type log collection, you can apply two-level parsing for all types of logs. For more information, see Creating-a-parsing-rule. |
Filtering rule | Select the filtering rule that includes or excludes the logs from the collection. It is optional to add these rules. However, these rules help you to optimally utilize the storage space for logs. For more information, see Creating-a-filtering-rule. |
User group | Select one or more user groups to assign them to collection policies. You can implement role-based access for collection policies by assigning user groups to policies while creating or editing policies. Scenario Sarah is a tenant administrator at Apex Global, which uses BMC Helix Log Analytics to collect and analyze logs. As an administrator, Sarah has created the Security, Network, and Application user groups to implement role-based access. Sarah does not want any of these groups to view the other group’s data. Sarah can achieve this by associating the correct user group with a collection policy while creating or editing the collection policy. If you do not associate a user group with a collection policy, data collected with the policy is available to all user groups. You can use user groups to implement access control to all log data, which includes logs in BMC Helix Log Analytics and log events in BMC Helix Operations Management. For more information, see Controlling-access-to-the-log-data. |
To create a collection policy
- In BMC Helix Log Analytics, go to Collection > Collection Policies.
- Click Create.
- Depending on the logs that you want to collect, configure the details of the collection type, connectors, tags, fields, filtering rules, and user groups.
For detailed information about these configurations, see the following topics:- AWS Cloudwatch logs (AWS log type on the Collection Policies page)
- Application logs (File log type on the Collection Policies page)
- Syslog messages (Syslog log type on the Collection Policies page)
- Windows events
- Enable and save the policy after all the configurations are complete.
After the collection policy runs, you can see the logs on the Explorer page. In the log records, you can see the log_source_host field that provides information about the data source where the logs originated. With this information, you can perform accurate root-cause analysis because the logs are enriched with the host or server name that caused service degradation.
Restricting access to log records
As an administrator, use user groups while creating or editing collection policies to restrict access to log records.
The following scenario describes the log collection behavior if you change the user group association in collection policies.
Scenario: What happens if associated user groups are changed in collection policies?
Sarah is an administrator in Apex Global, which uses BMC Helix Log Analytics to collect and analyze logs. Sarah has created the Operators and Administrators user groups to implement role-based access in the system. She has created the following collection policies to collect log data:
- Common collection policy: The Operators and Administrators groups are associated with this policy. The data collected from this policy is visible to users belonging to both groups.
- Restricted collection policy: The Administrators group is associated with this policy. The data collected from this policy is visible only to the Administrators group.
Sarah decides to remove the association of the Operators group from the Common collection policy. In this scenario, these actions happen:
- At the time of the next data ingestion after the group association change, the Operators group can no longer see the data collected by the Common collection policy. However, they can still see the data collected up to the data ingestion.
- The Administrators group continues seeing the data. There is no change in behavior for the Administrators group.