Default language.

Troubleshooting performance issues with Email Engine

If there are performance issues with , you might not receive email notifications at all or you might not receive them in a timely manner.

To resolve performance issues with , follow the guidelines described in this topic. Alternatively, create a BMC Support case.

Best Practice

We recommend that you refer to BEST FAQ on AR System Email Engine Issuesfor resolutions to the most common questions on .


Related topic

Contacting Support

Issue symptoms

  • You do not receive email notifications.
  • A large number of outgoing email messages are queued up and are not processed (or processed very slowly) from the AR System Email Messages form.

See also Resolutions for common issues .

Issue scope

  • One or more users do not receive email notifications in a timely manner.
  • One or more s are affected in a Server Group.

Diagnosing and reporting an issue

Perform the following steps to troubleshoot outgoing issues with :

Task

Action

Steps

Reference

1.

Identify whether the problem is with outgoing or incoming emails.

  • For the outgoing email messages, check whether you do not receive notifications at all or you don't receive them in a timely manner. See Task 2.
  • For the incoming email messages, check whether the incoming messages take a long time to create new incidents and approve requests.

2.

Identify if the outgoing email messages queue in the AR System Email Messages form.

  • Follow one of these steps to search the AR System Email Messages form for outgoing messages with a Send status of Yes:
    • Use the following advanced query:
      'Message Type*' = "Outgoing" AND 'Send Message' = "Yes"
    • Consider adding a date range to your query, for example:
      'Message Type*' = "Outgoing" AND 'Send Message' = "Yes" AND 'Create Date:' >= ($DATE$ - 86400)
  • Check the configuration of the outgoing mailbox.
  • Consider applying the most current hotfix.
  • If there a large number of records returned by the query, see Resolutions for common issues.

3.

If you do not notice any queuing in the AR System Email Messages form, check the application workflow.

  • Check any custom workflow that involves email notifications.
  • Enable API logs, Filter logs, SQL logs, and Escalation logs to track the expected customer workflow actions, as required.

4.

Check the configuration of  .

Check if the following settings are correct in the AR System Email Mailbox Configuration form and EmailDaemon.properties file:

  • Polling Interval (In the Mailbox Configuration form)
  • Polling Is Minutes (T or F)(* Confirm)
  • NumberOfSenderThreads
  • OutgoingMessagesQueueSize
  • SendEmailSetSize
  • RPC

To review all the properties or settings in the  EmailDaemon.properties  file, see Updating-the-Email-Engine-settings-by-using-the-Centralized-Configuration.

5.

Review logs.

Analyze the  logs, and check for SEVERE errors for outgoing mailboxes.

The following are the locations for the  logs:

  • (Windows)
    Location: emailEngineInstallDir\AREmail\Logs
    Name: email.log
  • (UNIX)
    Location: emailEngineInstallDir/logs
    Name: emaild.log and/or emaild.sh_log

6.

Enable logs.

Enable FINEST logging for .

7.

Enable  logging.

To enable the server-side API logs, Filter logs, and SQL logs in  :

  1. Open AR System Administration Console.
  2. Select System > General > Server Information.
    The Server Information page is displayed.
  3. On the Log Files tab, select the API Log check box, Filter Log check box, and SQL Log check box.
  4. Click Apply.
  5. Click OK.
    The server-side logs in  are enabled.

Note: If the  connects to a Load Balancer, you must enable logs on all s in the group.

To enable logs on multiple servers in a server group, select the Server Group Logging option in the Server Group Administration Console.

8.

Reproduce the issue or wait for it to occur.

Check if you can reproduce the performance issue. Otherwise, wait until the issue occurs on its own.


9.

Collect and analyze the  logs and  logs.

Copy the log files to another location and then review them so that they do not get overwritten. The log files might be huge in size depending upon the volume.

Collect the  logs including the .bak file, if it exists.

The following are the locations for the  logs:

  • (Windows) <ARSystem Install Dir>\AREmail\Logs\email.log
  • (UNIX) <ARSystem Install Dir>/AREmail/Logs/email.log

Collect API logs, Filter logs, and SQL logs from each  in the server group.

10.

Use  Log Analyzer.

  • Try to identify long queries from  by using the API logs and SQL logs from the  that owns the email operations.
  • Check the API logs for API calls that show the API call GetListEntryWithFields (GLEWF) or GetListEntry (GLE) that are from . Then use the corresponding TID and RPC ID to find the SQL and ending API call.
    • <API > <TID: 0000000589> <RPC ID: 0000191683> <Queue: Prv:390634> <Client-RPC: 390634 > <USER: Remedy Application Service > <Overlay-Group: 1 > /* Wed Jan 22 2020 13:40:36.8220 */ +GLE ARGetListEntry -- schema AR System Email Messages from E-mail Engine (protocol 25) at IP address 127.0.0.1 using RPC // :q:0.0s
    • <API > <TID: 0000000589> <RPC ID: 0000191683> <Queue: Prv:390634> <Client-RPC: 390634 > <USER: Remedy Application Service > <Overlay-Group: 1 > /* Wed Jan 22 2020 13:40:40.6590 */ -GLE OK

Note: If you run  on a private RPC socket, for example, Queue: Prv:390636, it can help you to identify the corresponding API or SQL calls in the logs.

11.

Find a solution.


12.

Create a BMC Support case

  • Collect and send logs and detailed information when you create a case with BMC Support.
  • Run the Log Zipper utility.
  • Attach a ZIP file to your case. You can attach a ZIP file of size up to 2 GB. You can also upload the file on FTP.

Resolutions for common issues

After you determine a specific symptom or an error message, use the following table to identify the solution:

Symptom

Action

Reference

There are too many records on the AR System Email Messaging forms.


Run SQL scripts to get form counts.

There are too many records on the AR System Email Messaging forms.

Follow the data retention policies and create a custom escalation to remove the older records or enable the archiving feature.

You get the following message in the  log:

Slow processing of emails

Use the Throughput Analyzer.

There are outdated  files or libraries in the  installation directory.


Obtain the latest recommended hotfix or contact BMC Support.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*