Troubleshooting disk space problems

In normal use, the BMC Discovery datastore grows in size as new data is discovered. For more information about datastore growth, see The datastore is growing rapidly Is this normal? article in BMC Communities.

Monitoring disk usage

If the initial disk allocation is insufficient, the datastore could run out of disk space, at which point the system will shut down. Similarly, other disk partitions in the BMC Discoveryappliance could run out of disk space. The key to avoiding this is to monitor the system's disk space. There are several ways to accomplish this:

1. You can manually monitor disk usage though the Discovery UI. For a standalone appliance, go to the Administration > Appliance Configuration page. The Hardware > Storage section will show the usage statistics (total, used, free) for each primary Discovery partition. For a cluster, the Administration > Cluster Management page will show these statistics at a glance for each cluster member.
2. Configure Discovery to generate a baseline alert (and optionally send email notifications) when the size of the datastore passes the "Size Warning" threshold in Administration > Model Maintenance. This threshold has to be configured manually and adjusted as the datastore grows in size.
3. Configure Discovery to generate a baseline alert (and optionally send email notifications) when the datastore files have grown to the point where the available space in the datastore partition falls below the "Free disk space baseline threshold". For the datastore, this threshold is set to 2gb, but can be adjusted via tw_options. 
4. Configure Discovery to generate a baseline alert (and optionally send email notifications) when the appliance log files have grown to the point where the available space in the partition hosting these files falls below the "Free disk space baseline threshold". For the appliance logs, this threshold is set to 200mb, but can be adjusted via tw_options.
   For more information about baseline alerts, see Baseline configuration.
   For more information about the Disk Space Monitor, see Configuring disk space monitor.
5. Configure a cron job to send an email displaying the Discovery disk space usage. This provides a comprehensive, at-a-glance view of usage for all Discovery partitions.

See the following video (3:51) to know how to set up an email alert to monitor disk space on a Discovery appliance. 

https://youtu.be/DhTXW7uPRD4

For more information, see  How to avoid running out of disk space on a Discovery appliance  in BMC Communities.

Preventing disk space issues

It is normally possible to prevent disk space issues by performing some routine tasks:

1. Compact the datastore on a regular basis to reduce disk usage. See tw_ds_compact for more information.
   • If available disk space is too low, it may not be possible to run a compact. You should never move or delete any datastore files to reduce disk space usage.
   • The compaction can be scheduled to run on a regular basis using a cron job. For more information, see tw_cron_update.
2. If the /var/log directory is becoming full, see the BMC Communities article The var log directory is full. What files can be deleted to reduce space usage for suggestions regarding which files can be deleted to reduce space usage.   
3. If the /tmp directory is becoming full, see What files can be removed from tmp to provide free space for upgrades for suggestions regarding which files can be deleted to reduce space usage. 

If the appliance does run out of space 

The first step is always to determine which partition is full. This can be done by running "df -h" from the appliance command line. Typically, either /usr/tideway or the datastore partition is full.

The output of 'df -h' may not show any partition at 100% capacity, but whichever one reports the highest percentage usage is likely the one that is having the problem.

Also, check the output of 'df -h' for a mount point named "/mnt/addm/db_data". If this exists, it means that the datastore has been moved to this partition, and is no longer held in /usr/tideway.

If a Support case needs to be opened, always include the results of "df -h".

The common root causes of the appliance being out of space include:

• The datastore or datastore transaction files have grown beyond the limits of the partition.
• Large core dump files have been created due to a process failure.
• There are too many or too large reasoning transaction files in the persist directory.
• A local backup takes up too much space.

For more information, see this  BMC Communities post .

If the /usr/tideway partition is full          

To identify the biggest files in /usr/tideway, go to the appliance command line, su to root and run this command:

du /usr/tideway | sort -nr | head -n 30

This will show the largest files in /usr/tideway, in descending order. 

If the largest files are datastore files, then the directory “/usr/tideway/var/localdisk/tideway.db/data/datadir" will be near the top of the list. The datastore files themselves (typically these start with "p000") will also be large. See "If the partition where the datastore and/or datastore transaction log files reside is full (or nearly so)" below.

If the largest files are datastore transaction log files, then the directory "/usr/tideway/var/localdisk/tideway.db/logs" will be near the top of the list. The datastore log files themselves (typically these look like "log.000002301") may also be large. See "If the partition where the datastore and/or datastore transaction log files reside is full or nearly full" below.

If the largest files are not datastore-related, see this knowledge article for a list of files that can be safely removed so that the services can start.

If the partition where the datastore and/or datastore transaction log files reside is full or nearly full 

If there is some space left, you can try to compact the datastore. In this scenario, the "--smallest first" option is likely the only choice. For more information, see tw_ds_compact.

Otherwise, the only solution is to allocate a new, larger disk for the appliance VM, and move the datastore and/or datastore transaction log files to the new disk.

If the Discovery UI is available, moving these components can be done through the Discovery Configuration page. See:

• Managing disks and swap space
• Managing disks and swap space in a cluster

In most cases, the Discovery services have been shut down and the UI is not available. The components then have to be moved using the command line tw_disk_utils utility. For more information, see this  BMC Communities post .

See the following video (23:48) which explains how to resolve disk space problems in BMC Discovery. 

 https://youtu.be/9f7Ig2QhB2I