Troubleshooting

BMC AMI Ops Insight and BMC AMI Ops User Interface are in different or same environment. Users may or may not have shared DASD for BMC AMI Ops Insight and BMC AMI Ops User Interface.

Solution 

Use one of the following to resolve this issue:

The following points apply to both the cases mentioned here:

• Make sure that the AMI_MGR_HOST property value in amipdt.properties of BMC AMI Ops Insight is a valid host name and not just localhost.
• If you running the product in SSL mode (https), import the Truststore that is used in BMC AMI Ops Insight into the Truststore that is used in BMC AMI Ops User Interfaceor the converse.   

When using View Detailed Analysis option, you get the Analysis Failed message. The View Detailed Analysis option of the product uses the existing BMC AMI Ops Monitor products (BMC AMI Ops Monitor for Db2, CICS, z/OS, and IMS) for analysis. Because the Data Preparation address space queries the BMC AMI Ops Monitor Products, the CAS to which the Data Preparation address space connects must be part of the MV Plex that includes the BMC AMI Ops Monitor watched by BMC AMI Ops Insight.

Example

Solution

If a CAS was installed for purely testing the product, but not set up to interact with the rest of the MV Plex, either connect to a CAS that is already part of the MV Plex or update the CASDEF so that it can connect to the other CAS in the MV Plex. 

When you request for training, the request fails giving Process ended with no output error. This happens when the owner of JAVA symlink (<install_Path>/aoihome/java) is not set to BPXROOT or there is no APF authorization for sorting and other files at <install_path>/aoihome/tomcat9/default/lib.

Solution

1. Set the file owner of BMC AMI Ops Insight JAVA 17 home directory (J17.0_64) symbolic link to BPXROOT.
   The AMIUNTAR job for JAVA17 and JAVA1764 TAR files sets the file owner to BPXROOT. You must run this job with user ID having super user permission (uid=0). The generic ingest calls BMCSORT from JAVA to sort SMF records. Because BMCSORT should run in supervisor mode, to prevent SEC6 abend, you must set the JAVA path file owner to BPXROOT according to IBM's requirement.
2. All files under USS directory &AMIINST/aoihome/tomcat9/default/lib must be APF authorized. The AMIHLQ.BMCLINK and AMIHLQ.BMCLIB install data sets must also be APF authorized. 

When you log in to the application, the BMC AMI Ops Insight related content is not visible

Solution

You can use one of the following solutions:

• • Make sure you have the relevant permissions (CONTROL) for RACF profile BBM.AMOI.UI.ACCESS as specified in Security.
  • If the permissions to RACF profile are there and still the issue continues, make sure that the BMC AMI Ops UI has RACF access to BPX.SERVER as specified in RACF authorization for the BMC AMI Ops user interface started task .

When you log in to the application, you can view the administration screens, but you cannot create a new model.

Generic Ingest task is looping in LGBSORT, and no records are ingested. During sorting of input SMF files, if there are records with record length greater than 32767, it creates an infinite loop. 

Solution

Apply PTF BQQ6434 to your test environment.

To verify if the fix is applied or not, search for PTF BQQ6434 in LGBSRT64 load module in your BMCLIB.

The following message is displayed in the user interface if the Tomcat is not running. 

PDTAM0030E: Tomcat Rest interface not available

And, the following message is issued in the BMC AMI Manager log:

ERROR
com.bmc.ami.controllers.AppController - PDTAM0059E: Error occurred when
connecting to Tomcat Rest Interface: I/O error on GET request for
"http://<server>:<port>": Connection refused: connect; nested exception is
java.net.ConnectException: Connection refused: connect

Solution

Restart Tomcat.

One of the following messages is issued in the BMC AMI Manager log. This happens when BMC AMI Manager is trying to connect using SSL, but SSL is not configured properly.

ERROR
com.bmc.ami.controllers.AppController - PDTAM0059E: Error occurred when
connecting to Tomcat Rest Interface: I/O error on GET request for
"https://<server>:<port>": Unrecognized SSL message, plaintext connection?;
nested exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

or

ERROR
com.bmc.ami.controllers.AppController - PDTAM0059E: Error occurred when
connecting to Tomcat Rest Interface: I/O error on GET request for
"https://qac5:15546": sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target;
nested exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Solution

Make sure SSL is configured properly. See Enabling TLS authentication between Tomcat and BMC AMI Manager for more details.

The following message is displayed in the user interface. The data preparation address space entered in the amipdt.properties file is incorrect or the address space is not available.

PDTAM0042E: Exception occurred : ["Error Occurred
processing train","Request failed: Unknown AMI ID:
<address_space_name> Name Token failure: BMC AMI "

The following message is issued in the BMC AMI Manager log:

ERROR
com.bmc.ami.controllers.AppController - PDTAM0059E: Error occurred when
connecting to Tomcat Rest Interface: ["Error Occurred processing  train","Request failed: Unknown AMI
ID: <address_space_name> Name Token failure: BMC AMI "

Solution

Verify that the data preparation address space entered in the amipdt.properties file is correct and that the address space is running.

The following error occurs when the training engine does not have enough memory allocated.

JVMJ9GC017E -Xmx too small, must be at least 1 Mbytes
JVMJ9VM015W Initialization error for library j9gc29(2): Failed to initialize
Error: Could not create the Java Virtual Machine. Error: A fatal exception has
occurred. Program will exit”

Solution

Set MODELGEN_MEMORY to a value equal or greater than 64MB.

The following errors are issued when the MODELGEN_MEMORY is set to a value that is too small:

JVMDUMP039I Processing dump event systhrow, detail java/lang/OutOfMemoryError at
2020/03/24 13:59:07 - please wait. JVMDUMP032I JVM requested System dump using
'RGS.UIM.JVM.TDUMP.MVSXXXTC.D200324.T135907' in response to an event
JVMPORT022I Appending .X&DS to user-specified dump template to enable
multi-part dumps. IEATDUMP in progress with options
SDATA=(LPA,GRSQ,LSQA,NUC,PSA,RGN,SQA,SUM,SWA,TRT) IEATDUMP failure for
DSN='RGS.UIM.JVM.TDUMP.MVSXXXTC.D200324.T135907.X&DS' RC=0x00000008
RSN=0x00000022 
JVMPORT023W IEATDUMP failed because user-specified dump template
was too long. Retrying dump with default template. IEATDUMP in progress with
options SDATA=(LPA,GRSQ,LSQA,NUC,PSA,RGN,SQA,SUM,SWA,TRT) IEATDUMP success for
DSN='MVSXXX.JVM.MVSXXXTC.D200324.T135907.X&DS' 
JVMDUMP040I System dump written to dataset(s) using name template
MVSXXX.JVM.MVSXXXTC.D200324.T135907.X&DS 
JVMDUMP032I JVM requested Heap dump using
'/shrd/mvsxxx/trainingEngine/heapdump.20200324.135907.34079992.0002.phd' in
response to an event 
JVMDUMP010I Heap dump written to /shrd/mvsxxx/trainingEngine/heapdump.20200324.135907.34079992.0002.phd
JVMDUMP032I JVM requested Java dump using
'/shrd/mvsxxx/trainingEngine/javacore.20200324.135907.34079992.0003.txt' in
response to an event 
JVMDUMP010I Java dump written to /shrd/mvsxxx/trainingEngine/javacore.20200324.135907.34079992.0003.txt
JVMDUMP032I JVM requested Snap dump using
'/shrd/mvsxxx/trainingEngine/Snap.20200324.135907.34079992.0004.trc' in
response to an event 
JVMDUMP010I Snap dump written to
/shrd/mvsxxx/trainingEngine/Snap.20200324.135907.34079992.0004.trc 
JVMDUMP013I Processed dump event systhrow, detail java/lang/OutOfMemoryError. 
Exception in thread main java.lang.OutOfMemoryError: Java heap space tat
com.bmc.zso.ami.trainingEngine.DataSet$Model.<init>(DataSet.java:992) tat
com.bmc.zso.ami.trainingEngine.DataSet.add(DataSet.java:248) tat
com.bmc.zso.ami.trainingEngine.Driver.main(Driver.java:424) tat
com.bmc.zso.ami.trainingEngine.Main.main(Main.java:9

Solution

Set MODELGEN_MEMORY to a larger value. 

• Default value is 40 GB.
• Minimum value is 64 MB.

We recommend using the default value if possible. Otherwise use as large a value as feasible.

The following errors are issued if the system memory limit is lower than the MODELGEN_MEMORY setting (default value is 40 GB):

2020/03/12
07:46:29,716 ERROR íù íù com.bmc.zso.ami.pipe.AmiPipeManager  - error
message is:  JVMJ9VM015W Initialization error for library j9gc29(2):
Failed to instantiate heap.  40G requested Error: Could not create the
Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.

Solution

Increase started task region size and USS MEMLIMIT.

The following errors are issued if the started task region size is too small.

10.01.23 S0015688  OSZ0100E ERROR REPORT FOR YCSBWATC (ASID=X'08BF',CSCB=YCSBWATC)  508   
   508              TCB=008C3280 FMID=ZPDR110 RMID=PDR1100 PROD=AMIPRD                    
   508              LMOD=PDRPRDCT MEMBER=PDRWMPCR FUNCTION=PDRWMPCR                       
   508              RC=X'00000008' RSN=X'00000026' R1=X'00000000_00000100' EX_OFF=X'01AC'  

Solution

Increase the started task region size.

An error similar to the following is issued if MODELGEN_JAVA is set to an invalid value:

TrainingEngine Exception class java.io.IOException
Cannot run program \"\/java\/J17.0_64\/bin\/java\" (in directory
\"\/shrd\/mvsxxx\/trainingEngine\"):
\/java\/J17.0_64\/bin\/java\u0000: not found

Solution

Set MODELGEN_JAVA to the correct JAVA_HOME value.

An error similar to the following is issued if MODELGEN_PATH is set to an invalid value:

TrainingEngine Exception
class java.io.IOException Cannot run program /shrd/java/J17.0_64/bin/java (in
directory /mvsxxx/trainingEngine): EDC5129I No such file or directory.

Solution

Set MODELGEN_PATH to the correct path for the DataTraining.jar file.

To see the details for resource allocation and use in your Docker container, use the Docker stats command.

It shows the details for all containers allocated in the Docker instance.

The following table describes the dashboard container commands:

The following errors are issued to the Tomcat standard error file. One of the following might be the probable cause:

• Extended attributes APF and PRGCTL are not specified for libpredictjni.so and libpredictjni64.so. 
• The Java path symlink file owner is not BPXROOT.

BPXP029I OPEN ERROR FOR FILE PATH  451   
/shrd/mvsxxx/tomcat/bin/libpredictjni.so 
DEVICE ID 35 INODE 656.                  

IEA995I SYMPTOM DUMP OUTPUT  514                                       
SYSTEM COMPLETION CODE=EC6  REASON CODE=0594E04B                       
 TIME=13.26.46  SEQ=02731  CPU=0000  ASID=004F                         
 PSW AT TIME OF ERROR  070C4401   B9BFB48E  ILC 2  INTC 0D             
   NO ACTIVE MODULE FOUND - PRIMARY NOT EQUAL TO HOME                  
   NAME=UNKNOWN                                                        
   DATA AT PSW  39BFB488 - C07818F2  0A0DEBE9  D3600096                
   AR/GR 0: FFF00001/00000000_00000920   1: 00000000/00000000_04EC6000 
         2: 00000000/00000000_0594E04B   3: 00000000/00000000_063FBB10 
         4: 01FF000A/00000000_00000004   5: 00000000/00000000_00000004 
         6: 00000000/00000050_00000040   7: 0101006D/00000050_4E534B20 
         8: 0101006D/00000000_01D94518   9: 00000000/00000050_4E53531F 
         A: 00000000/00000050_4E533DA0   B: 00000000/00000000_063FBB10 
         C: 00000000/00000000_39BFB6E8   D: 00000000/00000050_4E534320 
         E: 00000002/00000000_39BFB473   F: FFFFFFFE/00000000_0594E04B 
 END OF SYMPTOM DUMP

Solution

To resolve this issue:

• When the extended attributes APF and PRGCTL are not specified for libpredictjni.so and libpredictjni64.so, make sure you correct the extended attributes for libpredictjni.so and libpredictjni64.so.
• When the Java path symlink file owner is not BPXROOT, make sure that the Java path is defined with owner is BPXROOT.

The following errors are issued to the Tomcat standard error file if the SMF data set name does not exist:

IKJ56228I DATA SET LPG.SMFTEST.QBGL.DATAX NOT IN CATALOG OR CATALOG CAN NOT BE ACCESSED
OSZ0062E DYNALLOC ERROR RC=04 ERROR=1708 INFO=0002 EERR=0000 EINFO=0000 ERSN=00000FD6
PDTPR0610E  Request ID (123456789012345678901234)  484                          
PDTPR0610E  Allocation error with SMF Input file                                
PDTPR0610E  Dataset: LPG.SMFTEST.QBGL.DATAX                                     
PDTPR0610E   RC 04  ERROR 1708  INFO 0002 ERSN 00000FD6                         
PDTPR0610E           EERR 0000 EINFO 0000                                       

Solution

Enter the correct dataset name.

All RACF permissions are configured, but you cannot log in to the application, and the log displays the following message:

Unable to login Unable to verify password for user <username> Success: false, errno: 157, errno2: 151782063, errnoMsg: EDC5157I An internal error has occurred.

Solution

If you see the following INFO message in the job log, follow the steps that follow it:

BPXP015I HFS PROGRAM <JAVA_HOME>/bin/classic/libjvm.so IS FROM A FILE SYSTEM MOUNTED WITH THE NOSETUID ATTRIBUTE.

• If the program's file system is mounted with the SETUID attribute, verify that <JAVA_HOME> value is set to 1.
• Verify that the APF and Program Control attributes of the <JAVA_HOME>/bin/classic/libjvm.so program are honored. If not, set the <JAVA_HOME> directory to 1.

IBM Java JNI codes require the Program Control bit to invoke the RACF functions functions, so the Java file system must be mounted with SETUID. For more information, see the SAF/RACF-based security topic in the IBM documentation.

When running BMC AMI Ops Insight Tomcat server, the CPU consumption is very high. This happens when the HOME property in the OMVS segment of the Tomcat STC user ID is empty.

Solution

Make sure that the OMVS segment of the user ID, which is assigned to the Tomcat started task is defined with a home directory. 

Set the HOME property of the OMVS segment to:

When processing RCA, you get the following errors:

• In Tomcat
  • Exception creating pipe:null
  • Failed to delete pipe nullclass java.io.IOException
    
    
• In BMC AMI Manager:
  • Error Occurred processing  rca","class java.io.IOException","Cannot run program \"rm\": error=124, EDC5124I Too many open files.

Solution

You must upgrade to Java 17.0.9.