Troubleshooting FTS indexing issues

• You do not get any results when you perform an FTS search.
• Data is not indexed.
• Reindexing is incomplete.
• You get incorrect results from the Global searches.
• You receive error messages while performing an FTS or Global search.

Enable FTS indexing audit to capture the FTS indexing status for a form and all the action types. To enable and record audit of all bulk reindexing operations with their result, perform the following steps:

1. Log in to Mid Tier.
2. Enable the FTS-Indexing_Audit parameter to audit your indexing operations.
3. Trigger FTS indexing and view your FTS indexing status on the FTS Indexing Audit form which provides the following details:
   • Schema Id
   • Indexing Status—Lists the following FTS indexing status: 
     • In Progress
     • Completed
     • Errored
   • Action Type—Lists the following FTS operation types:
     • SCHEMA_DELETE_INDEX
     • SCHEMA_REINDEX
     • INDEX_FROM_UPDATED_SINCE
     • FIELD_REINDEX
     • FIELD_DELETE_INDEX
   • Records To Be Indexed—Number of records pending to be indexed after you start indexing.
   • Records Actually Indexed—Number of indexed records after you start indexing.
   • Started—The date when you start FTS indexing.
   • Finished—The date when the FTS indexing is complete.

• You do not get any results because the index is not set correctly.
• The size of the ft_pending table is increasing. 
• You cannot add new records and the existing records are not processed.
• You get error messages while performing an FTS reindex.
• You observe a performance issue during the indexing process.
• Reindexing does not work.
• Scheduled/Scan time indexing does not work.

Perform the following steps to troubleshoot the FTS indexing issues:

1. Determine the type of indexing activity:
   • Ongoing indexing
   • Manual full reindex

   • Manual form reindex or field change

     Determine the cause of indexing activity:

     It can be a user activity or an automated process that causes indexing. Depending on the cause of indexing, different FTS operations can occur. You can view some of these operations in the ft_pending table.

     • Determine if the problem is with ongoing indexing.
       You can identify this by checking if the recently added records are returned in a search result. You can also check if the size of the ft_pending table increases.
     • If you perform a manual full reindex, the ft_pending table contains a record with Operation 5 or -5 for each form to be indexed.
     • If you perform a manual form reindex, the ft_pending table contains a record with Operation 6 or -6 for each form to be indexed.
       This can be caused by a Data Wizard job, using the FTS Management console, or other manual methods.
       Manual form reindexing also occurs if you reindex a knowledge source by using the Manage Knowledge Sources console.
     • If you have a scheduled scan time job for indexing the Join forms and Vendor forms, entries are created in the ft_schedule table. No entries are created in the ft_pending table.

2. Use the FTS Management console to verify the server is currently performing a reindex activity.

   Perform the following steps to check if any server is performing reindex

   1. Select Application > AR System Management Console > AR System Server Group Console > FTS Management.
   2. In the FTS Configuration panel, check if the Re-Indexing In Progress value is set to Yes for any server.
   3. If the Re-Indexing In Progress value is set to Yes for any server, check if the form reindex is in-progress. Identify if the majority of indexing jobs are individual records or the entire form.
      Form reindexing takes a longer time to complete.

3. Enable logs and reproduce the problem.
   If you cannot reproduce the problem, wait until the problem occurs again.

   Perform the following steps to enable logs:

   • Enable SQL logs and full text index logs on each of the affected indexing servers.
     For more information, see Enabling-server-side-AR-System-logs.

   • Optionally, enable the arfts.log file in debug mode. This file logs the indexing requests made by the AR System server. Log the request for 15 to 20 minutes or until the problem is observed. If there are specific steps to reproduce the problem, perform those steps.

   • The arftindx.log grows very quickly. Provide sufficient Max log size to contain 15 to 30 minutes of recording time. You can set the Max log size to 500 MB or more depending on the volume of data.
     

   • Analyze Java heap stack if you encounter any of the following problems:

     • Java heap size
     • AR System server stops responding
     • FTS plug-in server stops responding

     Use a Java Virtual Machine (JVM) or other appropriate tools to obtain a heap dump or thread dump as necessary. Collect the dump files when you observe the issue.
     For more information about collecting a heap or thread dump, see the following knowledge articles:

     • How to collect Java heap dumps (and thread dumps) for AR Server troubleshooting

     • How to enable JMX to FTS plugin

4. Disable logging.
   To disable logging, see the steps described in the Enabling-server-side-AR-System-logs.
   Make sure that you disable the logs from each affected indexing server.
5. Collect the logs.
   Copy logs to another location where you can view them. Make sure that the logs are not overwritten. The log file size can be large and it might vary depending on volume. Review the symptoms and error messages. 
6. Analyze the logs.
   Review the logs to try to identify error messages or behaviors.

7. Create a BMC Support case.

   Perform the following steps to collect and send logs with detailed information when creating a case with BMC Support:

   1. Provide the following information as part of your case:
      • Name of the user who performed the search
      • The search criteria used
      • Time when the problem occurred
      • Any error messages received
   2. Run the Support Assistant tool. 
   3. Attach the zip file to your case. You can attach the zip file upto 2 GB. You can also upload the files on FTP.

• Set the 390602 threads to four or more.
• Check the arftindx.log file and determine if the large join forms are indexed.
• Check the arftindx.log file and make sure that the index is set to a table field or to an attachment field.
  Since indexing can take a very long time, consider removing some indexes.
• Check the AR System server parameters for FTS indexer servers.

For more information, see FTS Best Practice Configuration for 9.1.x.

• Make sure that the field is indexed in Developer Studio.
• Make sure that the record or a form in indexed in the arftindx.log file.
• Make sure that the record or a form in indexed in the arftindx.log file is not showing any error
• If the arftindx.log file shows an error, the index might be corrupt. You can reindex the form.
• Ensure that you are not out of disk space for the Collection directory.

• If it is taking too long to index other data, review the knowledge article FTS Best Practice Configuration for 9.1.x.

• Slow disk I/O can cause indexer to take too long to write to the flat file Collection directory.

For more information, see How to FTS reindex a specific form using the Driver utility.

AR System memory grows very high during the indexing process

Check FTS searcher plug-in potential memory leak.

You get an error that FTS is either not licensed or the FTS service is not yet ready

• Make sure that only indexer servers are ranked for FTS in AR System Server Group Operation Ranking form.
• Use the AR System Administration console and make sure that FTS indexing is not disabled.
• Use the FTS Configuration console to confirm that FTS is configured correctly.
• Enable and review the FTS indexer logs to confirm that the FTS service is ready and is functioning as expected.
• Review the Shared Settings in the Centralized Configuration and make sure that Operating-Mode is set to 0 or is not present.

For more information, see FTS is either not licensed or the FTS Service is not ready yet.