Example cleanup plan for large operational environments

This topic walks you through an example end-to-end process of setting up a comprehensive plan for cleaning up your BMC Server Automation (BSA) database in a large operational environment, where the database fills up quickly, and the standard weekly cleanup plan (described in Example weekly database cleanup plan) is not sufficient. This cleanup plan can also be used in situations where you have accumulated large amounts of BladeLogic data in the database due to failures in the basic weekly cleanups, and you need to temporarily apply a more forceful cleanup plan to catch up on database maintenance.

The cleanup plan described here involves a combination of cleanup jobs organized within Batch Jobs and scheduled to run on a frequent basis (mostly daily, some weekly). Setting up the cleanup plan involves the following tasks:

  1. Define retention periods.
  2. Create multiple cleanup jobs based on the default cleanup job that is provided out-of-the-box. Each of these jobs is designed to perform an individual cleanup function. 
  3. Organize the various cleanup jobs that you created (in the previous task) as the building blocks in more complex Batch Jobs — one for the daily cleanup of historical data, and the other for weekly hard deletion of additional data that is not fully deleted by the daily job.

This topic includes the following sections:

Failed to execute the [excerpt-include] macro.

How to create jobs for individual cleanup functions

For greater control over the various cleanup functions, we will create multiple copies of an out-of-the-box NSH Script Job named BSA Recommended Database Cleanup Job, and customize each new job to perform an individual cleanup function (that is, to execute an individual delete command). In each of these new jobs, we will customize the job parameters, which are inherited from the BSA Recommended Database Cleanup Script. All these jobs will serve as the building blocks in the next task, when we organize the individual cleanup jobs in Batch Jobs.  (For more about the out-of-the-box script and job, see Default-database-cleanup-script-folders-and-files.)

 

Step

Example screen

1

To create your first copy of the default cleanup job, perform the following steps:

  1. Expand the Jobs folder and navigate to BMC Maintenance > BSA Recommended Database Cleanup Job.
  2. Right-click the job and select Copy.
  3. Right-click the BMC Maintenance folder and select Paste.
    A dialog box tells you that the copy operation is occurring in the background. Shortly afterwards, a new job named Copy of BSA Recommended Database Cleanup Job appears in the folder, alongside the original job that you copied.

copyJob.png

2

To open the new job, right-click Copy of BSA Recommended Database Cleanup Job, and select Open.

opencopiedjob.png

3

In the content editor on the right, on the General tab, change the name of this new job to Cleanup AuditTrail, and the description to Cleanup of the Audit Trail. Then click Save saveButton.png in the toolbar.

nshJobGeneral.png

4

To customize job parameters, click the Parameters tab, and then make the following changes to parameter settings:

  • Through the Value column, set new values for the following parameters:

    Parameter

    Value

    ExecutionMode

    HISTORY_ORDR 


    ObjectType

    AuditTrail

    RetentionTime

  • Through the Flag runtime usage column, set the following parameters to Ignore. This will force the NSH Script Job to use default values for these parameters without requiring input.
    • MaxDuration
    • DurationDistribution
    • RoleName
    • TargetSize
  • Leave the remaining parameters (ContinueOnError and TargetDevices) with their default settings.

For more information about the script parameters, see Script parameters.

parameters_auditTrail.png

5

Save the job again to apply all changes.

 

6

Repeat steps 1 to 5 for another 12 cleanup functions. The following table lists all 13 cleanup jobs that you need to create, and includes the name for each job (to rename in step 3) and the parameter settings that you need to adjust (in step 4).

Tip: To speed up the process of creating these jobs, choose a source job (in step 1) with parameter settings that are similar to those of the target job. For example, to create the Cleanup JobRunEvent job, copy the Cleanup AuditTrail job and adjust just the job name and description and the ObjectType parameter. You can also paste multiple copies of the same source job.

 

Job names and parameters to adjust in the cleanup jobs for individual cleanup functions

Job name

Execution
Mode

Object
Type

Retention
Time

Other
parameters

Cleanup AuditTrail

HISTORY_ORDR

 AuditTrail

30

Set to ignore:

MaxDuration
DurationDistribution
RoleName
TargetSize

Do not change:

ContinueOnError
TargetDevices

Cleanup JobRunEvent

HISTORY_ORDR

JobRunEvent

Cleanup JobSchedule

HISTORY_ORDR

JobSchedule

Cleanup AuditResult

HISTORY_ORDR

AuditResult

set to ignore

Set to ignore:

MaxDuration
DurationDistribution
RetentionTime
RoleName
TargetSize

Do not change:

ContinueOnError
TargetDevices

Cleanup ComplianceResult

HISTORY_ORDR

ComplianceResult

Cleanup Deploy

HISTORY_ORDR

Deploy

Cleanup PatchResult

HISTORY_ORDR

PatchResult

Cleanup SnapshotResult

HISTORY_ORDR

SnapshotResult

Cleanup AppServerCaches

CLEAN_ALL_AS

set to ignore

30

Set to ignore:

MaxDuration
DurationDistribution
ObjectType
RoleName
TargetSize

Do not change:

ContinueOnError
TargetDevices

Cleanup Database

CLEAN_DB

set to ignore

set to ignore

Set to ignore:

MaxDuration
DurationDistribution
RetentionTime
ObjectType
RoleName
TargetSize

Do not change:

ContinueOnError
TargetDevices

Cleanup Shared Objects

CLEAN_SHARED_OBJECTS

Cleanup FileServer

CLEAN_FS

Cleanup Retention

RETENTION

How to organize the individual cleanup jobs into Batch Jobs for a comprehensive cleanup plan

We will now organize the various cleanup jobs that we prepared into two Batch Jobs — one for the daily cleanup of historical data, and the other for weekly hard deletion of additional data that does not get deleted by the daily job. The daily cleanup job is a relatively complex Batch Job, with another two Batch Jobs nested within it (one within the other). The hierarchical organization of cleanup jobs within these Batch Jobs is as follows:

  • Daily cleanup of historical data:
    Batch Job named Daily Cleanup. This job contains the following jobs, set to run sequentially:
    1. Cleanup Retention
    2. Batch Job named Cleanup Historical, with the following jobs set to run in parallel:
      • Batch Job named Cleanup Results, with the following cleanups set to run sequentially:
        1. Cleanup AuditResult
        2. Cleanup SnapshotResult
        3. Cleanup ComplianceResult 
        4. Cleanup PatchResult
        5. Cleanup Deploy
      • Cleanup JobRunEvent
      • Cleanup AuditTrail
      • Cleanup JobSchedule
    3. Cleanup AppServerCaches
  • Weekly hard delete cleanup:
    Batch Job named Weekly Cleanup. This job contains the following jobs, set to run sequentially:
    1. Cleanup Retention
    2. Cleanup Database
    3. Cleanup Shared Objects
    4. Cleanup FileServer

The following instructions guide you through the creation of these Batch Jobs. For the daily cleanup job, we will start with the creation of the most nested, lowest-level Batch Job, and work our way upward and outward until we have created the final daily Batch Job.

 

Step

Example screen

1

To create the most nested Batch Job, which groups together the cleanup of job results, perform the following steps:

  1. Under the Jobs folder, right-click the BMC Maintenance folder and select New > Batch Job.
    The New Batch Job window and the Select Jobs to Add to Batch Job dialog box open.

  2. From the displayed tree, select the jobs for cleanup of job results and add them to the Batch Job. Use the Ctrl key to select multiple jobs. Then click OK.

    Click for the list of jobs:
    1. Cleanup AuditResult
    2. Cleanup SnapshotResult
    3. Cleanup ComplianceResult 
    4. Cleanup PatchResult
    5. Cleanup Deploy

    The jobs that you added to the Batch Job appear in a hierarchical list on the left.

  3. If necessary, use the Move Up moveup.png and Move Down movedown.png arrows to organize the jobs in the correct order.
  4. Select the New Batch Job node in the hierarchical list on the left.
    Several tabs of Batch Job settings appear on the right.
  5. On the Batch Job Options tab, enter Cleanup Results as the name of the job, and (optionally) enter a description for the job.
  6. Under Execution Options, ensure that Execute jobs sequentially is the selected option. Then click OK to save the job.

batch_jobselection.png

batchResults.png

2

To create the next Batch Job, which groups together the cleanup of historical data, perform the following steps:

  1. Right-click the BMC Maintenance folder and select New > Batch Job.

  2. Select the jobs for cleanup of historical data and add them to the Batch Job. Use the Ctrl key to select multiple jobs. Then click OK.

    Click for the list of jobs:
    • Cleanup Results (the Batch Job that you created in step 1)
    • Cleanup JobRunEvent
    • Cleanup AuditTrail
    • Cleanup JobSchedule
  3. Select the New Batch Job node in the hierarchical list on the left to display the tabs of Batch Job settings.
  4. On the Batch Job Options tab, enter Cleanup Historical as the name of the job, and (optionally) enter a description for the job.
  5. Under Execution Options, ensure that Execute jobs in parallel is the selected option. Then click OK to save the job.

batchHistorical.png

3

To create the final Batch Job for daily cleanup of historical data, perform the following steps:

  1. Right-click the BMC Maintenance folder and select New > Batch Job.

  2. Select the jobs included in the daily cleanup Batch Job and add them to the Batch Job. Use the Ctrl key to select multiple jobs. Then click OK.

    Click for the list of jobs:
    1. Cleanup Retention
    2. Cleanup Historical (the Batch Job that you created in step 2)
    3. Cleanup AppServerCaches
  3. If necessary, use the Move Upmoveup.png and Move Downmovedown.png arrows to organize the jobs in the correct order in the hierarchical list on the left.
  4. Select the New Batch Job node in the hierarchical list on the left to display the tabs of Batch Job settings.
  5. On the Batch Job Options tab, enter Cleanup Historical as the name of the job, and (optionally) enter a description for the job.
  6. Under Execution Options, ensure that Execute jobs sequentially is the selected option.
  7. To schedule the job, click the Schedules tab, and then click New Schedule addservers.jpg. Afterwards, in the Scheduling box, schedule a daily recurrence of the job, and click OK.
    In this example, we want to run the job 6 days a week (rather than all 7 days). Therefore, we select the Weekly option, and then select all days except for Sunday. We set a time late at night when the workload is normally low.
  8. Click OK to save the job.

batchDaily.png

dailySchedule.png

4

To create the weekly Batch Job for hard deletion of other types of data, perform the following steps:

  1. Right-click the BMC Maintenance folder and select New > Batch Job.

  2. Select the jobs included in the weekly cleanup Batch Job and add them to the Batch Job. Use the Ctrl key to select multiple jobs. Then click OK.

    Click for the list of jobs:
    1. Cleanup Retention
    2. Cleanup Database
    3. Cleanup Shared Objects
    4. Cleanup FileServer
  3. If necessary, use the Move Up moveup.png and Move Down movedown.png arrows to organize the jobs in the correct order in the hierarchical list on the left.
  4. Select the New Batch Job node in the hierarchical list on the left to display the tabs of Batch Job settings.
  5. On the Batch Job Options tab, enter Cleanup Historical as the name of the job, and (optionally) enter a description for the job.
  6. Under Execution Options, ensure that Execute jobs sequentially is the selected option.

  7. To schedule the job, click the Schedules tab, and then click New Schedule addservers.jpg. Afterwards, in the Scheduling box, schedule a weekly recurrence of the job (in this example, every Sunday evening), and click OK.

    Note

    If you use BMC Decision Support for Server Automation to create reports, be careful that the weekly cleanup job does not overlap the weekly ETL job for loading data to the reporting data warehouse. Typically, you first run the ETL job, and only after the ETL job has finished you run the database cleanup job.

    It is not recommended to run cleanup while running updates of database statistics (Oracle or SQL Server), so as not to impact the performance of the environment.

  8. Click OK to save the job.

batchWeekly.png

weeklySchedule.png

Where to go from here

If you set up this cleanup plan as a temporary means for catching up on the deletion of old data that accumulated in your database, you can follow the current plan for several weeks and then revert back to the basic plan described in Example-weekly-database-cleanup-plan.

For more complicated situations, where you have accumulated large amounts of BladeLogic data in the database, you might need to plan a period of downtime for your BMC Server Automation Application Server so that you can perform a one-time offline database cleanup, a more forceful cleanup mechanism.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*