Infrastructure Management database and operating system issues

The following issues are related to the database or to the operating system:

Cannot load ianywhere error

SAP SQL Anywhere generates a cannot load ianywhere/ml/jdbcodbc/IIStatement File jdbcodbc\c\nativeobj.c at line #507--aborting execution error when the SAP SQL Anywhere server is already installed. 

This SAP SQL Anywhere related error is displayed when the SAP SQL Anywhere server is already installed. The Infrastructure Management installation is running, and the dbisqlg.exe file exists in the background processes. 

In the error message, click Yes to abort execution of the dbisqlg.exe file. To avoid this error, close the dbisqlg.exe process before you begin the installation.

Connection terminated abnormally

Connection to the database is terminated abnormally. 

Connections to the database are closed, for example, when the jserver process is restarted. This error message is displayed in the storm_ <hostName>db.log file located in the installationDirectory/pw/asatmp directory. 

No response is required. You can ignore this message.

Creation of a table fails

Creation of a table can fail when keywords that are reserved for the database are used for column headings.

You can use the register monitor script to validate whether the table is created after executing the script. When naming column headings, third-party developers might have used keywords that are reserved for the database. Ensure that no one uses the keywords that are reserved for the database. You can obtain a comprehensive list of reserved keywords from the database vendor. The following is a partial list of reserved keywords: 

• ASA Database 
• MESSAGE 
• FIRST 
• INDEX 
• Sybase and Oracle\ Databases: 
• SIZE 
• LOAD 
• LOAD_HIGH 
• LOAD_LOW 
• LOAD_AVG
  
  

Assertion failure

The database crashes after an assertion failure. 

For information about how to recover from an assertion failure, see  Recovering from an assertion failure .

Incorrect page count for table

Incorrect page count is displayed after deleting pages from a table in the database.

If an incorrect page count is displayed after you delete pages from a table in the database, perform the following tasks to correct this issue: 

1. Unload the table content. 
2. Delete the table. If you are unable to delete the table, rename the original table and create a new table by using the original table name. 
3. Reload the correct table content back to the table.

Reloading creates the table and adds the contents to the table.

Size of the Oracle data file

If you are using Infrastructure Management on Oracle, and if Oracle runs out of free space, the following error message is written to the TrueSight.log file: 

ORA-01653: unable to extend table PROACT.AUTODISCOVERY_ERRORS by 8 in tablespace NGP_TS 

Or

ORA-01654: unable to extend index AUTOPROACT._PATROL__SCA_0_STATS_5_N_1 by 1024 in tablespace PNET_TS_AUTO

Contact the Oracle administrator to create a new data file. As a best practice, you can monitor the Oracle tablespace using the Oracle KM. For information on configuring Oracle DB for monitoring, see  Step 5: Set up Infrastructure Management Self-Monitoring for SQLAnywhere and Oracle DBs.

Installation fails due to incorrect Oracle character set

The Infrastructure Management installation fails with the following error in the installation log file: failure Error: NLS_CHARACTERSET is set to [WE8MSWIN1252]. It needs to be set to [AL32UTF8]

To resolve this issue, run the installer again and change the Oracle character set to Unicode (AL32UTF8) in the Database Configuration Assistant, Step 9 of 11: Installation Parameters screen.

Reestablishing a lost connection to the database

Infrastructure Management automatically reconnects to the database if the database connection is lost temporarily. The database connection may be lost due to many factors, such as network latency and unresponsive database due to high system load.  

Behavior in 11.3.03

TrueSight Infrastructure Management tries to connect to the database continuously for 15 min, at 1-min intervals. The incoming data is not buffered during this time.

A failure notification email is sent every 2 min and a success notification email is sent once the database connection is reestablished.

If the database connection is not reestablished within 15 min, the server (both servers in a high-availability deployment) is shut down and an email notification is sent.

If required, configure the following parameters in the installedDirectory\pw\custom\conf\pronet.conf file:

Behavior in 11.3.01 and 11.3.02

The database reconnection mechanism will not work while the following operations are running:

• Auto Work Flow (Create/Update/Delete of Monitor Types/instances/devices/groups)
• Devices (Create/Update/Delete)
• Graphs (Create/Update/Delete)
• SLA (Create/Update/Delete)
• Groups (Create/Update/Delete)
• Partitioning
• Baselining
• Schema creation during installation
• Monitor wizard (Create/Update/Delete)
• CLI
• Integration Service (Add/Update/Delete)

If the database connection goes down for more than one minute and if the connection is not established, the Administrator is notified through an e-mail message that there is a problem connecting to the database.

To reconnect to the database

1. Check the e-mail message and the ProactiveNet.log for the error "Failed to establish DB connection after retries ."
2. Check the network connection and the database server status.
3. Restart the TrueSight Infrastructure Management server after restoring the database connection.

The following is an example of the e-mail message sent to the Administrator.

Hello,
The TSIM server running on "kali" server has not been able to establish connectivity with the database.
Please address the database connectivity problem and restart the TSIM server.
Please address the database connectivity problem and restart the TSIM server.
Database server details
•  Database Host: < _hostname:port number_
>
•  SID: < _SID_
>
•  Status: Unreachable
•  Status: Unreachable
- TSIM Server.

Exporting Oracle tables

The required files (custom_dump_dir_pnet.sql and pnet_exp_selected_tables.par) to export Oracle tables are available under the <Installation directory>/pw/pronto/sysdata/oracle_dump_utils directory.

To export the Oracle tables, perform the following steps:

1. On the Oracle Server, set oracle_sid, connect to the database by typing sqlplus at the command prompt.
2. Log in with sys user  
   Enter user-name: sys/<password>@<sid> as sysdba
3. Execute the sql script custom_dump_dir_pnet.sql and specify the location of the dump directory file and user (for example, proact) at the prompt.
4. To export the selected table of Infrastructure Management user, use the pnet_exp_selected_tables.par file and then from the command prompt, run the expdp utility by specifying the username, password, sid, and pnet_exp_selected_tables.par.
   expdp <Infrastructure Management_username>/<password>@<sid> PARFILE=<path\pnet_exp_selected_tables.par>
   Check the exp_selected_table_data.log file for logs whether export succeeded or not.
5. Collect the exported dump EXP_SELECTED_TABLE_DATA.DMP from the dump directory you specified in step 3 and upload it to the BMC FTP site (ftp.bmc.com) along with following details:
   
   1. Infrastructure Management user name
   2. Tablespace for Infrastructure Management user. To get this information, see the following example: 
      select default_tablespace, username  from dba_users where username = ‘<Infrastructure Management_username>’

Pronet_db.log file

The pronet_db.log file captures all the database activities if debug is enabled. Otherwise, it captures the SQL exceptions that occur during run time.

Maximum number of processes exceeded

When using the same database for multiple Infrastructure Management servers, you  get the following exceptions: Could not open connection or a Got minus one from a read call IO error. 

Exception stack trace 1:

Exception stack trace 2:

To resolve this issue,  you must increase the processes in multiples of hundred. The default value is 1000, you can increase it in multiples of hundreds like 1100,1200,1500 based on the traffic that you are monitoring.

To increase the number of processes allowed on a database server do the following: 

1. Log on to the database using the following command:

   sys/****@<SID> as sysdba alter system set processes =1500 scope=spfile

   sys/****@<SID> are your user credentials.

2. Restart the database. 
   Note: Restarting the database after executing the command increases the number of processes.

3. Restart the TrueSight Infrastructure Management server after restoring the database connection.

Unable to extend the Oracle System tablespaces SYSTEM or SYSAUX

ORA-01658: unable to create INITIAL extent for segment in tablespace SYSAUX

Solutions:

Recommended SYSTEM tablespace size is 2 GB and SYSAUX tablespace size is 3 GB to avoid this error.

Verify and fix one or all of the following in these tablespaces if required:

• The SYSTEM tablespace is not the default or temporary tablespace for the normal database users.
• Look for space consuming tables such as AUD$ in the SYSTEM tablespace and move it to a different tablespace.
• Query and check for objects consuming most of the space on these tablespaces and either move those outside for later use or delete it if you do not want those objects.
• Look for tables in SYSTEM and SYSAUX tablespaces that store historical query data, AWR snapshot, and audit information, which can be either deleted or truncated.
• If you cannot perform any of the above, add additional disk space to these tablespaces.

HPROF file is not generated on Linux

On Linux, when The Agent Controller, JServer, or Rate process crashes, the heap dump (HPROF) is not generated.

Possible cause:

The core file size is set to 0 (disabled).

Solution:

Run the following command to check if the core file size is set to 0 (disabled).
ulimit -c
If disabled, enable the core file by updating the /etc/security/limits.conf file:
<Non-root-User-Name> soft core unlimited
<Non-root-User-Name> hard core unlimited