Example cleanup plan for large operational environments

This topic walks you through an example end-to-end process of setting up a comprehensive plan for cleaning up your TrueSight Server Automation (BSA) database in a large operational environment, where the database fills up quickly, and the standard weekly cleanup plan (described in Example-weekly-database-cleanup-plan) is not sufficient. This cleanup plan can also be used in situations where you have accumulated large amounts of BladeLogic data in the database due to failures in the basic weekly cleanups, and you need to temporarily apply a more forceful cleanup plan to catch up on database maintenance.

The cleanup plan described here involves a combination of cleanup jobs organized within Batch Jobs and scheduled to run on a frequent basis (mostly daily, some weekly). Setting up the cleanup plan involves the following tasks:

Define retention periods.
Create multiple cleanup jobs based on the default cleanup job that is provided out-of-the-box. Each of these jobs is designed to perform an individual cleanup function.
Organize the various cleanup jobs that you created (in the previous task) as the building blocks in more complex Batch Jobs — one for the daily cleanup of historical data, and the other for weekly hard deletion of additional data that is not fully deleted by the daily job.

This topic includes the following sections:

How to set and enable retention times

Before running Database Cleanup Jobs, you must set retention periods for two types of BSA data. You must also enable a retention policy, so that data that is older than the defined retention periods is marked for deletion. For more information about tasks related to retention periods and the retention policy, see Marking-data-for-deletion.

	Step	Example screen
1	Set a retention period for job run data and job results through the Property Dictionary. In this example we define a retention period of 30 days (the length of time recommended by BMC) for all jobs of all types. In the TrueSight Server Automation Console, from the Configuration menu, select Property Dictionary View. In the Property Class Navigation pane, expand the Built-in Property Classes node, and select the Job property class. In the property list on the right, select the Properties tab. Select the RESULTS_RETENTION_TIME property, and then click Edit Property. Ensure that the Use this default value check box is selected, and then, in the Default value column, enter 30 as your retention period value (in days). Note To see how job run data is displayed in the GUI, click here. Job runs are visible in any job's Results panel; each job run is accompanied by a green (success) or red (failure) icon and a date stamp. For information about additional options for setting retention times, click here. In this example we have set one retention period for all jobs of all types. You can, optionally, set different retention periods for different types of jobs or even for specific jobs. The more specific retention periods override the more general retention periods. For more information about these options, see To set the retention period for job runs. For more information about setting properties through the Property Dictionary, see Adding-or-modifying-properties. This retention time is not applied to the audit trail, job run events (that is, job run log messages), and job schedules. These data types accumulate faster than the other types of job data. Therefore, they have a shorter default retention time of 14 days.
2	We will be using the TrueSight Server Automation Application Server console (the blasadmin utility) to set a retention period for auto-generated objects and to enable the retention policy. To start the Application Server Administration console, perform the following step: Windows: From the Start menu, select Programs > BMC Software > BladeLogic Server Automation Suite > Utilities > Application Server Administration. UNIX: From the directory where TrueSight Server Automation is installed, enter ./br/blasadmin For alternative methods, see Starting-the-Application-Server-Administration-console.
3	Auto-generated objects typically include BLpackages involved in compliance remediation or patch remediation, as well as Deploy Jobs and Batch Jobs from these remediation processes. To specify a retention period for auto-generated objects, enter the following command: set Cleanup AutoGeneratedRetentionTime 30 In this example, we are setting the retention time to 30 days, as recommended by BMC.
4	To enable the retention policy. Enter the following command:set Cleanup EnableRetentionPolicy true
5	For your changes to take effect, restart the Application Server. Perform the following steps: In the TrueSight Server Automation Console, select Configuration > Infrastructure Management. Expand the Application Servers node. Right-click the Application Server and select Restart. For alternative methods, see Restarting Application Servers.

Important

In order to clean data from delete task statistics, you must configure a separate cleanup job that will call the blcli, see Delete-deleteCleanupTaskStatus.

How to create jobs for individual cleanup functions

For greater control over the various cleanup functions, we will create multiple copies of an out-of-the-box NSH Script Job named BSA Recommended Database Cleanup Job, and customize each new job to perform an individual cleanup function (that is, to execute an individual delete command). In each of these new jobs, we will customize the job parameters, which are inherited from the BSA Recommended Database Cleanup Script. All these jobs will serve as the building blocks in the next task, when we organize the individual cleanup jobs in Batch Jobs. (For more about the out-of-the-box script and job, see Performing online database cleanup.)

Step

Example screen

To create your first copy of the default cleanup job, perform the following steps:

Expand the Jobs folder and navigate to BMC Maintenance > BSA Recommended Database Cleanup Job.
Right-click the job and select Copy.
Right-click the BMC Maintenance folder and select Paste.
A dialog box tells you that the copy operation is occurring in the background. Shortly afterwards, a new job named Copy of BSA Recommended Database Cleanup Job appears in the folder, alongside the original job that you copied.

To open the new job, right-click Copy of BSA Recommended Database Cleanup Job, and select Open.

In the content editor on the right, on the General tab, change the name of this new job to Cleanup AuditTrail, and the description to Cleanup of the Audit Trail. Then click Save in the toolbar.

To customize job parameters, click the Parameters tab, and then make the following changes to parameter settings:

Through the Value column, set new values for the following parameters:
Parameter
Example value
ExecutionMode
HISTORY_ORDR
ObjectType
AuditTrail
RetentionTime
30 (same value as set for RESULTS_RETENTION_TIME)
Through the Flag runtime usage column, set the following parameters to Ignore. This will force the NSH Script Job to use default values for these parameters without requiring input.
- MaxDuration
- DurationDistribution
- RoleName
- TargetSize
Leave the remaining parameters (ContinueOnError and TargetDevices) with their default settings.

For more information about the script parameters, see Script parameters.

Save the job again to apply all changes.

Repeat steps 1 to 5 for another 12 cleanup functions. The following table lists all 13 cleanup jobs that you need to create, and includes the name for each job (to rename in step 3) and the parameter settings that you need to adjust (in step 4). You need to create Cleanup Retention job only once.

Tip: To speed up the process of creating these jobs, choose a source job (in step 1) with parameter settings that are similar to those of the target job. For example, to create the Cleanup JobRunEvent job, copy the Cleanup AuditTrail job and adjust just the job name and description and the ObjectType parameter. You can also paste multiple copies of the same source job.

Job names and parameters to adjust in the cleanup jobs for individual cleanup functions

Job name	Execution Mode	Object Type	Retention Time	Other parameters
Cleanup AuditTrail	HISTORY_ORDR	AuditTrail	30	Set to ignore: MaxDuration DurationDistribution RoleName TargetSize Do not change: ContinueOnError TargetDevices
Cleanup JobRunEvent	HISTORY_ORDR	JobRunEvent
Cleanup JobSchedule	HISTORY_ORDR	JobSchedule
Cleanup SharedData	HISTORY_ORDR	SharedData
Cleanup AuditResult	HISTORY_ORDR	AuditResult	set to ignore	Set to ignore: MaxDuration DurationDistribution RetentionTime RoleName TargetSize Do not change: ContinueOnError TargetDevices
Cleanup ComplianceResult	HISTORY_ORDR	ComplianceResult
Cleanup Deploy	HISTORY_ORDR	Deploy
Cleanup PatchResult	HISTORY_ORDR	PatchResult
Cleanup SnapshotResult	HISTORY_ORDR	SnapshotResult
Cleanup OldVersionComponent (Available in 8.9.02 and later versions)	HISTORY_ORDR	OldVersionComponent
Cleanup OldVersionJob (Available in 8.9.02 and later versions)	HISTORY_ORDR	OldVersionJob
Cleanup AppServerCaches	CLEAN_ALL_AS	set to ignore	30	Set to ignore: MaxDuration DurationDistribution ObjectType RoleName TargetSize Do not change: ContinueOnError TargetDevices
Cleanup Database	CLEAN_DB	set to ignore	set to ignore	Set to ignore: MaxDuration DurationDistribution RetentionTime ObjectType RoleName TargetSize Do not change: ContinueOnError TargetDevices
Cleanup Shared Objects	CLEAN_SHARED_OBJECTS
Cleanup FileServer	CLEAN_FS
Cleanup Retention	RETENTION
Cleanup ListBLValue	HISTORY_ORDR	ListBLValue	Set to ignore	Set to ignore

How to organize the individual cleanup jobs into Batch Jobs for a comprehensive cleanup plan

We will now organize the various cleanup jobs that we prepared into two Batch Jobs — one for the daily cleanup of historical data, and the other for weekly hard deletion of additional data that does not get deleted by the daily job. The daily cleanup job is a relatively complex Batch Job, with another two Batch Jobs nested within it (one within the other). The hierarchical organization of cleanup jobs within these Batch Jobs is as follows:

Daily cleanup of historical data:
Batch Job named Daily Cleanup. This job contains the following jobs, set to run sequentially:
1. Cleanup Retention
2. Batch Job named Cleanup Historical, with the following jobs set to run in parallel:
  1. - Cleanup JobRunEvent
    - Cleanup AuditTrail
    - Cleanup JobSchedule
    - Cleanup SnapshotResult (If you are using TrueSight Server Automation 8.9.03.001 and later, you cannot run this job in parallel.)
      However, you cannot execute two or more Cleanup Historical batch jobs in parallel
  - Batch Job named Cleanup Results, with the following cleanups set to run sequentially:
    1. Cleanup AuditResult
    2. Cleanup ComplianceResult
    3. Cleanup PatchResult
    4. Cleanup Deploy
    5. Cleanup SharedData
    6. Cleanup SnapshotResult (This is applicable for TrueSight Server Automation 8.9.03.001 and later.)
    7. Cleanup ListBLValue
    8. Cleanup OldVersionComponent
    9. Cleanup OldVersionJob
3. Cleanup AppServerCaches
Weekly hard delete cleanup:
Batch Job named Weekly Cleanup. This job contains the following jobs, set to run sequentially:
1. Cleanup Retention
2. Cleanup Database
3. Cleanup Shared Objects
4. Cleanup FileServer

The following instructions guide you through the creation of these Batch Jobs. For the daily cleanup job, we will start with the creation of the most nested, lowest-level Batch Job, and work our way upward and outward until we have created the final daily Batch Job.

	Step	Example screen
1	To create the most nested Batch Job, which groups together the cleanup of job results, perform the following steps: Under the Jobs folder, right-click the BMC Maintenance folder and select New > Batch Job. The New Batch Job window and the Select Jobs to Add to Batch Job dialog box open. From the displayed tree, select the jobs for cleanup of job results and add them to the Batch Job. Use the Ctrl key to select multiple jobs. Then click OK. Click for the list of jobs: Cleanup AuditResult Cleanup SnapshotResult Cleanup ComplianceResult Cleanup PatchResult Cleanup Deploy The jobs that you added to the Batch Job appear in a hierarchical list on the left. If necessary, use the Move Up and Move Down arrows to organize the jobs in the correct order. Select the New Batch Job node in the hierarchical list on the left. Several tabs of Batch Job settings appear on the right. On the Batch Job Options tab, enter Cleanup Results as the name of the job, and (optionally) enter a description for the job. Under Execution Options, ensure that Execute jobs sequentially is the selected option. Then click OK to save the job.
2	To create the next Batch Job, which groups together the cleanup of historical data, perform the following steps: Right-click the BMC Maintenance folder and select New > Batch Job. Select the jobs for cleanup of historical data and add them to the Batch Job. Use the Ctrl key to select multiple jobs. Then click OK. Click for the list of jobs: Cleanup Results (the Batch Job that you created in step 1) Cleanup JobRunEvent Cleanup AuditTrail Cleanup JobSchedule Cleanup SharedData Select the New Batch Job node in the hierarchical list on the left to display the tabs of Batch Job settings. On the Batch Job Options tab, enter Cleanup Historical as the name of the job, and (optionally) enter a description for the job. Under Execution Options, ensure that Execute jobs in parallel is the selected option. Then click OK to save the job.
3	To create the final Batch Job for daily cleanup of historical data, perform the following steps: Right-click the BMC Maintenance folder and select New > Batch Job. Select the jobs included in the daily cleanup Batch Job and add them to the Batch Job. Use the Ctrl key to select multiple jobs. Then click OK. Click for the list of jobs: Cleanup Retention Cleanup Historical (the Batch Job that you created in step 2) Cleanup AppServerCaches If necessary, use the Move Up and Move Down arrows to organize the jobs in the correct order in the hierarchical list on the left. Select the New Batch Job node in the hierarchical list on the left to display the tabs of Batch Job settings. On the Batch Job Options tab, enter Cleanup Historical as the name of the job, and (optionally) enter a description for the job. Under Execution Options, ensure that Execute jobs sequentially is the selected option. To schedule the job, click the Schedules tab, and then click New Schedule. Afterwards, in the Scheduling box, schedule a daily recurrence of the job, and click OK. In this example, we want to run the job 6 days a week (rather than all 7 days). Therefore, we select the Weekly option, and then select all days except for Sunday. We set a time late at night when the workload is normally low. Click OK to save the job. Note Cleanup of shared data might involve very large tables with large amounts of data about configuration files. If the Cleanup SharedData job takes takes a very long time (several hours) to execute, terminate the long-running cleanup job, and then execute the following SQL command to increase batch size from 10000 table rows to a much larger value (1 million in this example): update dbm_module set batch_size=1000000 where module_id=18; After running this SQL command, run the cleanup job again.
4	To create the weekly Batch Job for hard deletion of other types of data, perform the following steps: Right-click the BMC Maintenance folder and select New > Batch Job. Select the jobs included in the weekly cleanup Batch Job and add them to the Batch Job. Use the Ctrl key to select multiple jobs. Then click OK. Click for the list of jobs: Cleanup Retention Cleanup Database Cleanup Shared Objects Cleanup FileServer If necessary, use the Move Up and Move Down arrows to organize the jobs in the correct order in the hierarchical list on the left. Select the New Batch Job node in the hierarchical list on the left to display the tabs of Batch Job settings. On the Batch Job Options tab, enter Cleanup Historical as the name of the job, and (optionally) enter a description for the job. Under Execution Options, ensure that Execute jobs sequentially is the selected option. To schedule the job, click the Schedules tab, and then click New Schedule . Afterwards, in the Scheduling box, schedule a weekly recurrence of the job (in this example, every Sunday evening), and click OK. Note If you use TrueSight Smart Reporting for Server Automation to create reports, be careful that the weekly cleanup job does not overlap the weekly ETL job for loading data to the reporting data warehouse. Typically, you first run the ETL job, and only after the ETL job has finished you run the database cleanup job. It is not recommended to run cleanup while running updates of database statistics (Oracle or SQL Server), so as not to impact the performance of the environment. Click OK to save the job.

Where to go from here

If you set up this cleanup plan as a temporary means for catching up on the deletion of old data that accumulated in your database, you can follow the current plan for several weeks and then revert back to the basic plan described in Example-weekly-database-cleanup-plan.

For more complicated situations, where you have accumulated large amounts of BladeLogic data in the database, you might need to plan a period of downtime for your TrueSight Server Automation Application Server so that you can perform a one-time offline database cleanup, a more forceful cleanup mechanism.

In addition to the jobs described here, schedule a monthly cleanup of the agents and repeater server. Create cleanup jobs set to individual modes CLEAN_AGENT and CLEAN_REPEATER, and optionally include the two jobs within a batch job. For best results, keep these cleanup jobs separate from database cleanup (for example, do not include database cleanup in the same batch job).