Troubleshooting FTS indexing issues

1

Determine the type of indexing activity:

• Ongoing indexing
• Manual full reindex
• Manual form reindex or field change

Determine the cause of indexing activity. It can be a user activity or there can be an automated process that causes indexing. Depending on the cause of indexing, different FTS operations can occur. You can view some of these operations in the ft_pending table.

• Determine if the problem is with ongoing indexing.
  You can identify this by checking if the recently added records are returned in a search result. You can also check if the size of the ft_pending table increases.
• If you perform a manual full reindex, the ft_pending table contains a record with Operation 5 or -5 for each form to be indexed.
• If you perform a manual form reindex, the ft_pending table contains a record with Operation 6 or -6 for each form to be indexed.
  This can be caused by a Data Wizard job, using the FTS Management console, or other manual methods.
  Manual form reindexing also occurs if you reindex a knowledge source by using the Manage Knowledge Sources console.
• If you have a scheduled scan time job for indexing the Join forms and Vendor forms, entries are created in the ft_schedule table. No entries are created in the ft_pending table.

2

Use the FTS Management console to check if the server is currently performing a reindex activity

Perform the following steps to check if any server is performing reindex:

1. Select Application > Remedy Management Console > AR System Server Group Console > FTS Management.
2. In the FTS Configuration panel, check if the Re-Indexing In Progress value is set to Yes for any server.

If the Re-Indexing In Progress value is set to Yes for any server, check if the form reindex is in-progress. Identify if the majority of indexing jobs are individual records or the entire form. Form reindexing takes a longer time to complete.

3

Enable logs and reproduce the problem.
If you cannot reproduce the problem, wait until the problem occurs on its own.

Perform the following steps:

• Enable SQL logs and full text index logs on each of the affected indexing servers.
  For more information, see Enabling-server-side-AR-System-logs.
• Optionally, enable the arfts.log file in debug mode. This file logs the indexing requests made by the AR System Server. Log the request for 15 to 20 minutes or until the problem is observed. If there are specific steps to reproduce the problem, perform those steps.

• The arftindx.log grows very quickly. Provide sufficient Max log size to contain 15 to 30 minutes of recording time. You can set the Max log size to 500 MB or more depending on the volume of data.
  

• Analyze Java heap stack if you encounter any of the following problems:

  • Java heap size
  • AR System Server stops responding
  • FTS plug-in server stops responding

  Use a Java Virtual Machine (JVM) or other appropriate tools to obtain a heap dump or thread dump as necessary. Collect the dump files when you observe the issue.
  For more information about collecting a heap or thread dump, see the following knowledge articles:

  • Collecting a Heap or Thread dump of the ARServer
  • Collecting a Heap or Thread dump of the FTS Plugin Server

4

Disable logging

Use steps described in the Enabling-server-side-AR-System-logs topic to disable the logging. Disable the logs from each affected indexing server without fail.

5

Collect the logs

Copy logs to another location where you can view them. Ensure that the logs are not overwritten. The log file size can be large and it might vary depending on volume.

Review the symptoms and error messages. Use the table to troubleshoot the issue.

6

Analyze the logs

You can review the logs yourself to try to identify error messages or behaviors. Use the table to help identify possible symptoms and solutions.

7

Creating a BMC Support Case

Collect and send logs and detailed information when creating a case with BMC Support:

1. Provide the following information as part of your case:
   • Name of the user who performed the search
   • The search criteria used
   • Time when the problem occurred
   • Any error messages received
     
     
2. Perform the following steps:
   1. Run the Log Zipper utility on each affected indexer server and select Zip Logs.
   2. If you provide the Log Zipper file from multiple servers, rename the file to include the server name.
3. Attach the zip file to your case. You can attach the zip file upto 2 GB. You can also upload the files on FTP.
   For more information, see How to use BMC's Managed File Transfer (MFT) to send log files to BMC Support to comply with GDPR policies.

ARERR 8760

On screen

• Check the Server-Plugin-Alias parameter in the ar.cfg file or in the Centralized Configuration.

• FTS-plug-in-configuration

Slow Indexing

The arftindx.log file on the FTS Management console

• Set the 390602 threads to four or more.
• Check the arftindx.log file and determine if the large join forms are indexed.
• Check the arftindx.log file and ensure whether the index is set to a table field or to an attachment field.
  Since indexing can take a very long time, consider removing some indexes.
• Check the AR System Server parameters for FTS indexer servers.

• FTS Best Practice Configuration

Data is not indexed

The arftindx.log file and the ft_pending table in the browser

• Ensure that the field is indexed in Developer Studio.
• Ensure that the record or a form in indexed in the arftindx.log file.
• Ensure that the record or a form in indexed in the arftindx.log file is not showing any error
• If the arftindx.log file shows an error, the index might be corrupt. You can reindex the form.
• Ensure that you are not out of disk space for the Collection directory.
• If it is taking too long to index other data, review the knowledge article Best Practices.
• Slow disk I/O can cause indexer to take too long to write to the flat file Collection directory.

• How to FTS reindex a specific form using the Driver utility

AR System Server memory grows very high during the indexing process

Operating system tools

FTS searcher plug-in potential memory leak.

Addressed in 1802 (9.1.04.02) hot fix package:  SR_9104P2_20181128_CU_ALL.zip

You get an error that FTS is either not licensed or the FTS service is not yet ready

The arftindx.log file

• Ensure that only indexer servers are ranked for FTS in AR System Server Group Operation Ranking form.
• Use the AR System Administration console and ensure that FTS indexing is not disabled.
• Use the FTS Configuration console to confirm that FTS is configured correctly.
• Enable and review the FTS indexer logs to  confirm that the FTS service is ready and is functioning as expected.
• Review the Shared Settings in the Centralized Configuration and ensure that Operating-Mode is set to 0 or is not present.

• Remedy - Server - FTS is either not licensed or the FTS Service is not ready yet