Troubleshooting FTS searching issues

1

Determine if the search is performed on the indexer server or searcher server (user-facing server)

• For searches performed on the indexer server, enable the FTS debug logging.
• For searches performed on the searcher server, enable the FTS Search plug-in log by using the steps described in the following knowledge article:
  How do I enable Debug FTS Plugin logs in Remedy AR System

2

Enable API and SQL logging on the server performing the search

See the steps described in the Enabling-server-side-AR-System-logs topic.

For load-balanced environments, enable the API and SQL log from all user-facing servers in the server group.    

Perform the following steps to enable debug searcher logging from the indexer server:

• Remove previous log entries from the ..\db directory to reduce the log file size.
• Rename log files to make it easier to identify.
• Ensure that the log files include the time when the issue was reproduced.
  Starting with version 18.05, you can enable logs across servers and server groups. For more information, see Managing-the-server-group-logs. 

3

Reproduce the issue.

If you do not have specific steps to reproduce the issue, enable the logs, wait for the problem to occur, and then perform these steps.

Provide the following information to BMC Support.

• Name of the user who performed the search.
• The search criteria used.
• Time when the problem occurred.

Important:

Analyze Java heap stack if you encounter any of the following problems:

• Java heap size
• AR System Server stops responding
• FTS plug-in server stops responding

Use the Java Virtual Machine (JVM) or other appropriate tools to obtain a heap dump or thread dump as necessary. Collect the dump files when you observe the problem. 
For more information about collecting a heap or thread dump, see the following knowledge articles:

• Collecting a Heap or Thread dump of the ARServer
• Collecting a Heap or Thread dump of the FTS Plugin Server 

4

Disable logging

• Allow three to five minutes for log buffering before running log zipper.
• Use steps described in the Enabling-server-side-AR-System-logs topic to disable the logging. 

5

Collect the logs

• Copy logs to another location where you can view them.
  Ensure that the logs are not overwritten.
• Review the symptoms and error messages.
  Use the table to troubleshoot the issue.

6

Analyze the logs

You can review the logs, identify error messages or behaviors. Use the table to identify possible symptoms and solutions.

7

Create a BMC Support case

Collect and send logs and detailed information when creating a case with BMC Support:

1. Provide the following information as part of your case:
   • Name of the user who performed the search.
   • The search criteria used.
   • Time when the problem occurred.
   • Any error messages received.
   • Include the pluginsvr_config.xml file located in the ..\pluginsvr\fts\secondary directory.
   • If you have more than one indexer, include all.
2. Perform the following steps:
   1. Run the Log Zipper utility on each affected indexer server and select Zip Logs. 
   2. If you provide the Log Zipper file from multiple servers, rename the file to include the server name.
3. Attach the zip file to your case.
   You can attach a zip file upto 2 GB. You can also upload the files on FTP.
   For more information, see How to use BMC's Managed File   Transfer (MFT) to send log files to BMC Support to comply with GDPR policies.

ARERR 8760

On Screen

• Check the Server-Plugin-Alias parameter in the ar.cfg file or in Centralized Configuration.

• FTS-plug-in-configuration

Slow response

Search results take too long to return

Browser

1. Check the numcorethreads in the plug-in configuration.
2. Check the API or SQL logs to determine if the FTS portion or the database portion of the search is slow.
3. Ensure that the search string is not long by verifying the arfts-searcher.log file.
4. Check the FTS Best Practice knowledge article to successfully set up the AR System Server and the FTS plug-in server.

• FTS Memory Leak
• FTS Best Practice Configuration

ARERR 657 or ARNOTE 657

The arerror.log file on screen

Check the arftindx.log file for related messages.

• Enable FTS Indexer logs

ARS:ARF.FTS (8753)
Error while searching

On screen

Check the configuration of the FTS secondary plug-in.

• FTS-plug-in-configuration

ARWarn 682

ARERR 682

Full text search service not available; a database search was performed

The arerror.log file and the arsql.log file on the screen

• Check if a reindex is being performed. Obtain the arftindx.log file from each indexer.
• Query the ft_pending table for operationType 5 or 7.
• Obtain the arerror.log and arsql.log files from the searcher server.
  • If you perform search on an indexer server and failover occurs, the search does not move to the next rank searcher server.
  • If you perform a search on a non-indexer server (searcher server), all the indexer servers are busy.
• If indexing is progressing, wait for it to complete.
• If Indexing does not work, contact BMC Support and provide the necessary log files.

• OperationType
• Knowledge article ft_pending table contents

Data is missing from results

Browser

OS Tools

SW00552847

Addressed in 18.02 hot fix package:  

SR_9104P2_20181128_CU_ALL.zip

FTS search is not working if field data contains dash or underscore like (- or _).

Browser

SW00551883

Addressed in 18.02 hot fix package: 

SR_9104P2_20181128_CU_ALL.zip

arftsplugin-searcher.log

Increase heap (See knowledge article FTS Best Practices)

Common Error Messages:

• java.lang.OutOfMemoryError: GC overhead limit exceeded
• java.lang.OutOfMemoryError: Java heap space

Cannot insert duplicate key row in object 'dbo.ft_pending'

API/SQL logs

SW00548960

Addressed in 18.02 hot fix package: 

SR_9104P2_20181128_CU_ALL.zip

ERROR (678): Full text indexing is disabled -- turn on full text indexing and retry your operation;

The arfts-searcher-plugin.log file on screen

Check the FTS tab from Server Information form and confirm FTS searching is not set to disabled.

• FTS-tab-configuration-options

ARSYS.ARF.FTS]
Fri Nov 30 12:40:13 2018 390602 :r:0 : The specified plug-in does not exist. (ARERR 8755)

The arerror.log file

• Check the Server-Plugin-Alias parameter in the ar.cfg file to validate Server Name and Port.
• Check plug-in configuration from the Plug-in Server Configuration.

• Troubleshooting-FTS-plug-in-issues
• Setting-plugin-server-configuration-options