Troubleshooting pods and containers in TrueSight Network Automation

#The following properties are read from inspectFaultHost.properties to make the
#Inspect fault host custom action return specified property values for fwsm or ace
#device.
#
#1. firewall.host1
#2. firewall.host2
#
#The above 2 properties take FWSM devices which participate in pair. Further
#host1 / host2 can allow multiple address names comma separated, e.g. FirewallA1,FirewallA2
#This is the case in large gold container where firewall host pairs are assigned in
#round robin.
#
#3. loadbalancer.host1
#4. loadbalancer.host2
#
#The above 2 properties take ACE devices which participate in pair. Further host1 / host2
#can allow multiple address names comma separated, e.g. LoadBalancerHost1,LoadBalancerHost2
#This is the case in large gold container where firewall host pairs are assigned in
#round robin.
#
#5. firewall.host1.adminFailoverGroup
#
#The above property controls the failover group returned by FWSM host (1/2/null). Combination
#of failover group and community1 / community2 active flag value will be used to determine
#adminActiveFlag property of firewall host.
#
#6. firewall.host1.faultCommunity1ActiveFlag
#
#The above property controls the faultCommunity1Activeflag for FWSM host1 to be returned by
#simulated custom action, for e.g. (Active / Standby)
#
#7. firewall.host1.faultCommunity2ActiveFlag
#
#The above property controls the faultCommunity2ActiveFlag for FWSM host1 to be returned by
#simulated custom action, for e.g. (Active / Standby)
#
#8. firewall.host2.adminFailoverGroup
#
#The above property controls the failover group returned by FWSM host (1/2/null). Combination
#of failover group and community1 / community2 active flag value will be used to determine
#adminActiveFlag property of firewall host.
#
#9. firewall.host2.faultCommunity1ActiveFlag
#
#The above property controls the faultCommunity1Activeflag for FWSM host2 to be returned by
#simulated custom action, for e.g. (Active / Standby)
#
#10. firewall.host2.faultCommunity2ActiveFlag
#
#The above property controls the faultCommunity2Activeflag for FWSM host2 to be returned by
#simulated custom action, for e.g. (Active / Standby)
#
#11. loadbalancer.host1.adminActiveFlag=true
#
#The above property controls the adminActiveFlag for ACE host1 to be returned by simulated
#custom action, for e.g. (true / false).
#
#12. loadbalancer.host2.adminActiveFlag=false
#
#The above property controls the adminActiveFlag for ACE host2 to be returned by simulated
#custom action, for e.g. (true / false).
#
#Suppose Pod has following ACE fault pairs LoadBalancerHost1, LoadBalancerHost2
#and following FWSM pairs FirewallHostA1 and FirewallHostA2 and we want make pair nodes look like
#below
# FirewallHostA1 = Admin
# FirewallHostA1 = fault comunity 1 active
# FirewallHostA2 = fault community 2 active
# LoadBalancerHost1 = Admin
# the 12 properties would look like below
firewall.host1=FirewallHostA1
firewall.host2=FirewallHostA2
loadbalancer.host1=LoadBalancerHost1
loadbalancer.host2=LoadBalancerHost2
firewall.host1.adminFailoverGroup=1
firewall.host1.faultCommunity1ActiveFlag=Active
firewall.host1.faultCommunity2ActiveFlag=Standby
firewall.host2.adminFailoverGroup=
firewall.host2.faultCommunity1ActiveFlag=Standby
firewall.host2.faultCommunity2ActiveFlag=Active
loadbalancer.host1.adminActiveFlag=true
loadbalancer.host2.adminActiveFlag=false 

When you are ready to move on to testing out container creation on real devices, one recommended way to test out the connectivity of network paths within a container is to connect to each in device in the container and verify that you can ping the VLAN interface address of each of the other devices that are supposed to be connected to it within the container.

Remember to use the view pages to inspect the state of your pod and container after you create them, to make sure that resources are in the state that you expect them to be in. You can also inspect the UI of the IPAM system to make sure that its resources are also in the state that you expect them to be in.

Back to top

Troubleshooting templates used during container configuration and unconfiguration

When you are troubleshooting or developing templates to be used during container configuration and unconfiguration, set the vdcCommitContainerActions property to false in the global.properties file. This property is set to true by default. Setting it to false prevents TrueSight Network Automation from committing any configuration changes to the devices involved. Any configuration changes you make can be rolled back by restarting the devices.

Back to top

Troubleshooting "Unable to reach CMDB. Please verify the credentials" error received during POD configuration

When creating a pod in TrueSight Network Automation, you might receive an Unable to reach CMDB. Please verify the credentials error. If this occurs, perform the following steps in order until you reach an action that resolves the problem:

1. Check if the Site has been created in the BMC.CORE:BMC_PhysicalLocation form on the BMC Remedy AR System server. The error message can be displayed if the record does not exist in this form.
   To create the record, please follow the steps as described in Creating a physical location for a pod.
2. If the Site has been created in the PhysicalLocation form and the problem persists, then verify Tomcat is running properly:
   1. Enter the URL http://AtriumWebServicesServer:portNumber in a browser.
      If Tomcat is running properly, an Apache screen is displayed.
   2. If the Apache screen is not displayed, restart Tomcat. If the problem still persists, check for errors in the Tomcat logs typically located in the AtriumWebServiceDirectory\BMC Software\Atrium Web Registry\shared\tomcat\logs directory.
3. If Tomcat is working, then verify the Atrium Web Services are working properly:
   1. Enter the Web Services URL http://AtriumWebServicesServer:portNumber/cmdbws/server/cmdbws in a browser.
      The result should be "Please enable REST support in WEB-INF/conf/axis2.xml and WEB-INF/web.xml".
   2. If nothing is returned, then it’s an issue with Atrium Web Services. Verifying the Atrium Web Services logs (typically located in the AtriumWebServiceDirectory\BMC Software\Atrium Web Registry\Logs folder) will help to narrow down the issue.
   3. If the page is loading fine, then settings in the TrueSight Network Automation console should be verified (Log on to TrueSight Network Automation and from Admin Tab > System Admin > System Parameters > External Integrations, verify the settings for "Enable CMDB Integration")
4. Verify the settings on the TrueSight Network Automation server for the BMC Atrium CMDB Integration:
   1. “Enable CMDB Integration” should be checked under the External Integration section. For BMC Cloud Lifecycle Management use cases, the "Enable CMDB Integration" parameter is used. The "Web Services Registry Integration" parameter should be kept disabled.
   2. Verify that the ‘Web Service Endpoint URL’ is correct. The format is http://AtriumWebServicesServer:portNumber/cmdbws/server/cmdbws.
   3. Enter the Username and Password. Ensure you can log in to the BMC Remedy AR System server with the same username and password. BMC recommends that you use "Demo" credentials for this step.
5. If the credentials are correct and the error is thrown, verify the cmdbws.properties file, typically located in the AtriumWebServiceDirectory\BMC Software\Atrium Web Registry\wsc\cmdbws folder.
   1. Verify that the hostname in the file is the hostname for the BMC Remedy AR System Server.
      The hostname should not be localhost, unless Atrium Web Services Registry Component is installed on the BMC Remedy AR System Server.
   2. If the hostname must be changed, after changing the hostname in the file restart the Tomcat service on the server.
      If the Atrium Web Service is running on the bundled Tomcat, you can use the Tomcat startup and shutdown scripts that are present in the AtriumWebServiceDirectory/shared/tomcat/bin folder for shutting down and starting Tomcat services.
6. If the BMC Remedy AR System Server hostname is correct in the cmdbws.properties file then check if the webapps folder (typically located at AtriumWebServiceDirectory\BMC Software\Atrium Web Registry\wsc\webapps) contains two folders, namely atriumws7604 and cmdbws. These folders should have three subfolders named axis2-web, META-INF, and WEB-INF. If these folders are missing, copy them from an available working environment.
7. If this is a Linux environment then make sure that the following environment variables are set before starting BMC Atrium Web Services. (These commands can also be set in the bmcAtriumSetenv.sh file. After editing the file, restart the Tomcat services.)
   
   • ATRIUMCORE_HOME=/opt/bmc/AtriumWebRegistry
   • export ATRIUMCORE_HOME
8. Sometimes, there can be communication issues between the TrueSight Network Automation Web services and the TrueSight Network Automation database. If the issue persists even after following the above steps, restart TrueSight Network Automation Web services.
   • For Windows environments, TrueSight Network Automation Web services are listed as BCA-Networks Web Server under services.msc and can be started or stopped from there.
   • For Linux environments, the following command can be used to stop or start TrueSight Network Automation Web services:
     service enatomcat stop / start
     The enatomcat services are listed in the /etc/init.d directory.
9. If the problem persists after completing this procedure, contact BMC support.

Back to top

Gathering PCM-specific information for BMC Support

If you need to gather detailed information about PCM behavior in order for BMC Support staff to troubleshoot issues, you can bump up the logging in the com.bmc.bcan.engine.network.pcm package in the logging.properties file and restart TrueSight Network Automation. This generates a lot of diagnostic details in the BCA-Networks.log.* log files while you execute subsequent PCM operations, which can then be sent to BMC Support for analysis.

Back to top