Troubleshooting

BMC AMI Ops Insight and BMC AMI Ops User Interface are in different or same environment. Users might have shared DASD for BMC AMI Ops Insight and BMC AMI Ops User Interface.

Solution 

Use one of the following to resolve this issue.

The following points apply to both whether or not BMC AMI Ops Insight and BMC AMI Ops User Interface are on a shared DASD: 

• Make sure that the AMI_MGR_HOST property value in amipdt.properties of BMC AMI Ops Insight is a valid host name and not just localhost.
• If you running the product in SSL mode (https), import the truststore that is used in BMC AMI Ops Insight  into the Truststore that is used in BMC AMI Ops User Interface or the converse.   

When using the View Detailed Analysis option, you get an Analysis Failed message. The View Detailed Analysis option of the product uses the existing BMC AMI Ops Monitor products (BMC AMI Ops Monitor for Db2, CICS, z/OS, and IMS) for analysis. Because the Data Preparation address space queries the BMC AMI Ops Monitor Products, the CAS to which the Data Preparation address space connects must be part of the MV Plex that includes the BMC AMI Ops Monitor watched by BMC AMI Ops Insight.

Example

Solution

If a CAS was installed purely for testing the product, but not set up to interact with the rest of the MV Plex, either connect to a CAS that is already part of the MV Plex or update the CASDEF so that it can connect to the other CAS in the MV Plex. 

When you request Training, the request fails immediately, giving Process ended with no output error message. 

This happens when the owner of the Java 17 home directory (J17.0_64)  is not set to BPXROOT, or there is no APF authorization for sorting and other files at installPath/aoihome/tomcat9/default/lib.

Solution

1. Set the file owner of BMC AMI Ops Insight JAVA 17 home directory (J17.0_64) symbolic link to BPXROOT.

   You must run this job with a user ID having superuser permission (uid=0).

   The generic ingest calls BMCSORT from Java to sort SMF records.

   To prevent SEC6 abend, you must set the Java path file owner to BPXROOT according to IBM's requirement, because BMCSORT should run in supervisor mode.

   The file owner of $INSTALL_PATH/aoihome/java/J17.0_64 symbolic link must be BPXROOT.

   Symbolic Link Attributes                                                     Pathname : $INSTALL_PATH/aoihome/java/J17.0_64                 External link  . . : 0                                                 File size  . . . . : 21                                                File owner . . . . : BPXROOT(0)                                        Group owner  . . . : CC100538(102)                                     Last modified  . . : 2022-12-12 16:09:33                               Last changed . . . : 2022-12-12 16:09:33                               Last accessed  . . : 2022-12-12 16:09:33                               Created  . . . . . : 2022-12-12 16:09:33                               Link count . . . . : 1                                                 Device number  . . : 1BF                                               Inode number . . . : 667                                               Seclabel . . . . . :                                                   Symbolic link contents:                                                                    /usr/lpp/java/J17.0_64

2. Make sure all files under the USS directory &AMIINST/aoihome/tomcat9/default/lib must be APF-authorized. The AMIHLQ.BMCLINK and AMIHLQ.BMCLIB install data sets must also be APF authorized.

   Make sure that the APF-authorized and program-controlled extended attributes are ON:

   • For the pdrjvm17 & libsortjni64.so files in  $INSTALL_PATH/aoihome/tomcat9/default/lib
   • For the datasets specified in the STEPLIB parameter in the AMITCEN7 member

3. The MODELGEN_JAVA parameter in AMITCEN7 member must point to  $INSTALL_PATH/aoihome/java/J17.0_64.

    

When you log in to the application, the BMC AMI Ops Insight-related content is not visible

Solution

• Make sure that you have the relevant permissions (CONTROL) for the RACF profile BBM.AMOI.UI.ACCESS as specified in Security.
• If you have the relevant permissions to the RACF profile and the issue continues, make sure that the BMC AMI Ops User Interface has RACF access to BPX.SERVER as specified in RACF authorization for the BMC AMI Ops user interface started task.

When you log in to the application, you can view the administration windows, but you cannot create a new model.

A generic Ingest task is looping in LGBSORT, and no records are ingested. During sorting of input SMF files, if there are records with a record length greater than 32767, it creates an infinite loop. 

Solution

Apply PTF BQQ6434 to your test environment.

To verify whether the fix is applied, search for PTF BQQ6434 in the LGBSRT64 load module in your BMCLIB.

The following message is displayed in the user interface if the Tomcat is not running:

The following message is issued in the BMC AMI Manager log:

Solution

Restart Tomcat.

One of the following messages is also issued in the BMC AMI Manager log. This happens when BMC AMI Manager is trying to connect via SSL, but SSL is not configured properly.

Solution

Make sure that SSL is configured properly. For more information, see Enabling TLS authentication between Tomcat and BMC AMI Manager for more details.

The following message is displayed in the user interface. The data preparation address space entered in the amipdt.properties file is incorrect, or the address space is not available.

The following message is issued in the BMC AMI Manager log:

Solution

Verify that the data preparation address space entered in the amipdt.properties file is correct and that the address space is running.

The following error occurs when the training engine doesn't have enough memory allocated:

Solution

Set MODELGEN_MEMORY to a value equal to or greater than 64 MB.

The following errors are issued when the MODELGEN_MEMORY is set to a value that is too small:

Solution

Set MODELGEN_MEMORY to a larger value. 

• Default value is 40 GB.
• Minimum value is 64 MB.

We recommend using the default value if possible. Otherwise, use as large a value as feasible.

The following errors are issued if the system memory limit is lower than the MODELGEN_MEMORY setting (default value is 40 GB):

Solution

Increase the started task region size and USS MEMLIMIT.

The following errors are issued if the started task region size is too small:

Solution

Increase the started task region size.

An error similar to the following one is issued if MODELGEN_JAVA is set to an invalid value:

Solution

Set MODELGEN_JAVA to the correct JAVA_HOME value.

An error similar to the following one is issued if MODELGEN_PATH is set to an invalid value:

Solution

Set MODELGEN_PATH to the correct path for the DataTraining.jar file.

To see the details for resource allocation and use in your Docker container, use the Docker stats command.

The details for all containers allocated in the Docker instance are displayed.

The following table describes the dashboard container commands:

One of the following errors, which are issued to the Tomcat standard error file, might be the probable cause:

• Extended attributes APF and PRGCTL are not specified for libpredictjni.so and libpredictjni64.so. 
• The Java path symlink file owner is not BPXROOT.

Solution

To resolve this issue:

• Make sure you specify the APF and PRGCTL extended attributes for libpredictjni.so and libpredictjni64.so.
• Make sure that the Java path is defined with the owner BPXROOT (uid=0).

The following errors are issued to the Tomcat standard error file if the SMF data set name doesn't exist:

Solution

Enter the correct data set name.

All RACF permissions are configured, but you can't log in to the application, and the log displays the following message:

Unable to login Unable to verify password for user <username> Success: false, errno: 157, errno2: 151782063, errnoMsg: EDC5157I An internal error has occurred.

Solution

If you see the following INFO message in the job log, follow the steps that follow it:

• If the program's file system is mounted with the SETUID attribute, verify that <JAVA_HOME> value is set to 1.
• Verify that the APF and Program Control attributes of the <JAVA_HOME>/bin/classic/ libjvm.so program are honored. If not, set the <JAVA_HOME> directory to 1.

IBM Java JNI codes require the Program Control bit to invoke the RACF functions functions, so the Java file system must be mounted with SETUID. For more information, see the SAF/RACF-based security topic in the IBM documentation.

When the BMC AMI Ops Insight Tomcat server is running, the CPU consumption is very high. This happens when the HOME property in the OMVS segment of the Tomcat STC user ID is empty.

Solution

Make sure that the OMVS segment of the user ID, which is assigned to the Tomcat started task, is defined with a home directory. 

Set the HOME property of the OMVS segment as follows:

When processing RCA, you get the following errors:

• In Tomcat

  • Exception creating pipe:null

  • Failed to delete pipe nullclass java.io.IOException

• In BMC AMI Manager:

  • Error Occurred processing  rca","class java.io.IOException","Cannot run program \"rm\": error=124, EDC5124I Too many open files.

Solution

Upgrade to Java 17.0.9.

BMC AMI Ops Insight suddenly loses connection, resulting in the stopping of real-time data streaming. This causes a socket timeout exception in the Tomcat servlet log.

This is a connection timeout in the SMF record handler.          

When a socket timeout exception occurs, the product retries connecting three times, and if the connection is still not established, the streaming process stops, and the message PDRRH0405E is sent to operator (WTO) as follows:

PDRRH0405E
Realtime streaming has stopped for AMI-Manager <IP>:<port> due to connection timeout with SMF Record Handler. Restart SMF Record Handler started task to resume realtime streaming.                           

Solution                                                                                                                                  

Restart the SMF record handler to resume the real-time streaming process.                         

The USS file system runs out of storage space.

Solution

You can use one of the following options:

• Increase storage space.
• Check for redundant DB backups in ../aoidata/aoiinst/backups.
• Check for redundant logs and logs management configurations. 
• Customize the logs size and numbers using the following parameters: 

• 
  

  • For Tomcat, add and modify the following parameters in <HLQ>.XXSAMP(AMITCEN7):

    IJO="$IJO -DTCAT_SERVLET_LOG_SIZE=30MB
    IJO="$IJO -DTCAT_SERVLET_ROLL_MAX=30" 

  • For SMF Record Handler, modify the following parameters in ../aoihome/smfrh/bin/run_smf_handler.sh:

    export SMFHDL_LOG_SIZE=20MB
    export SMFHDL_ROLL_MAX=10

  • For BMC AMI Manager modify the following parameters in ../aoihome/oi_home/mgr/bin/run_manager.sh:

    export AMIMGR_LOG_SIZE=20MB
    export AMIMGR_ROLL_MAX=10

One of the following errors is displayed in the job log of BMC AMI Ops UI Discovery server, BMC AMI Ops UI server, or BMC AMI Manager started task, and the server starts.

• exception=java.net.ConnectException: EDC8128I Connection refused. (Connection refused) stacktrace=com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: EDC8128I Connection refused. 
• exception=java.net.ConnectException: EDC8128I Connection refused. (Connection refused) stacktrace=com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: EDC8128I Connection refused. 

For example, BMC AMI Ops UI Discovery job log displays ‘Started AmiOpsDiscoveryServiceApplication in 18.742 seconds (process running for 22.456)’. error message.

Solution

1. Make sure the ports are open and accepting connections. You can use the TSO NETSTATS command.
2. Make sure that the incoming connections to this host and port are not getting blocked in the TCPIP log. If an error message is displayed in the log for this host and port combination, then refer to the relevant IBM documentation to know more about the error code.
3. If you are running in TLS or AT-TLS mode and using RACF Keyrings, make sure the certificates in the keyring are valid.
   User IDs listed against the started task in SDSF have access to the Keyring.
4. To validate the settings, use one of the following options:

   1. If you are using TLS, see Configuring the system for SSL/TLS connections.

   2. If you are using AT-TLS, see Configuring the system for AT-TLS connections.

Job log started task shows a valid certificate not found error when a new or changed certificate is not present in the truststore.

Solution

1. To download the client certificate to the server from the browser, click View site information > Connection is secure on the left of the address bar.
2. FTP this .crt file to a USS location on the mainframe in ASCII format.

3. Add this certificate to the truststore defined in AMIHLQ.UBBSAMP(AMCMNEV) file.
   Alternatively, you can add the certificate to the truststore (PKCS12/JKS) by using the following Java keytool command:
   
   keytool -import -keystore <//trustStoreName//> -storepass <//password//> -file <//absolutePathTotheCertificate//> -alias <//certificateAliasName//> -trustcacerts -storetype <//trustStoretype//>