Troubleshooting performance issues with Email Engine

Identify whether the problem is with outgoing or incoming emails.

• For the outgoing email messages, check whether you do not receive notifications at all or you don't receive them in a timely manner. See Task 2.
• For the incoming email messages, check whether the incoming messages take a long time to create new incidents and approve requests.

See Troubleshooting incoming issues with Email Engine.

If you do not notice any queuing in the AR System Email Messages form, check the application workflow.

• Check any custom workflow that involves email notifications.

• Enable API logs, Filter logs, SQL logs, and Escalation logs to track the expected customer workflow actions, as required.

Check the configuration of Email Engine.

Check if the following settings are correct in the AR System Email Mailbox Configuration form and EmailDaemon.properties file:

• Polling Interval (In the Mailbox Configuration form)
• Polling Is Minutes (T or F)(* Confirm)
• NumberOfSenderThreads
• OutgoingMessagesQueueSize
• SendEmailSetSize
• RPC

Analyze the Email Engine logs, and check for SEVERE errors for outgoing mailboxes.

The following are the locations for the Email Engine logs:

• (Windows)
  Location: emailEngineInstallDir\AREmail\Logs
  Name: email.log
• (UNIX)
  Location: emailEngineInstallDir/logs
  Name: emaild.log and/or emaild.sh_log

If SEVERE errors exist, see Troubleshooting outgoing issues with Email Engine.

Enable FINEST logging for Email Engine.

To enable the server-side API logs, Filter logs, and SQL logs in AR System:

1. Open AR System Administration Console.
2. Select System > General > Server Information.
   The Server Information page is displayed.
3. On the Log Files tab, select the API Log check box, Filter Log check box, and SQL Log check box.
4. Click Apply.
5. Click OK.
   The server-side logs in AR System are enabled.

Note: If the Email Engine connects to a Load Balancer, you must enable logs on all AR System servers in the group.

To enable logs on multiple servers in a server group, select the Server Group Logging option in the Server Group Administration Console.

Copy the log files to another location and then review them so that they do not get overwritten. The log files might be huge in size depending upon the volume.

Collect the Email Engine logs including the .bak file, if it exists.

The following are the locations for the Email Engine logs:

• (Windows)<ARSystem Install Dir>\AREmail\Logs\email.log
• (UNIX)<ARSystem Install Dir>/AREmail/Logs/email.log

Collect API logs, Filter logs, and SQL logs from each AR System server in the server group.

• Try to identify long queries from Email Engine by using the API logs and SQL logs from the AR System server that owns the email operations.
• Check the API logs for API calls that show the API call GetListEntryWithFields (GLEWF) or GetListEntry (GLE) that are from Email Engine. Then use the corresponding TID and RPC ID to find the SQL and ending API call.
  • <API > <TID: 0000000589> <RPC ID: 0000191683> <Queue: Prv:390634> <Client-RPC: 390634 > <USER: Remedy Application Service > <Overlay-Group: 1 > /* Wed Jan 22 2020 13:40:36.8220 */ +GLE ARGetListEntry -- schema AR System Email Messages from E-mail Engine (protocol 25) at IP address 127.0.0.1 using RPC // :q:0.0s
  • <API > <TID: 0000000589> <RPC ID: 0000191683> <Queue: Prv:390634> <Client-RPC: 390634 > <USER: Remedy Application Service > <Overlay-Group: 1 > /* Wed Jan 22 2020 13:40:40.6590 */ -GLE OK

Note: If you run Email Engine on a private RPC socket, for example, Queue: Prv:390636, it can help you to identify the corresponding API or SQL calls in the logs.

• See the following topics:
  • Running AR System Log Analyzer
  • Viewing AR System Log Analyzer output
  • Analyzing AR System Log Analyzer output
• If the SELECT statement generated by the GLE call, which is made by Email Engine, takes considerable amount of time to run, and if your Email Engine runs on a version earlier than 20.02, see Outgoing emails encounter slow performance where GLE API call has query taking significant amount of time to execute.

• If you still notice issues after following the preceding steps, or if you cannot complete any of these steps, run the Log Zipper utility. You need to run the Log Zipper utility while submitting a case on the performance of outgoing emails. You must run the Log Zipper utility on each server on which you have enabled the logs.
• Rename the zipped file to contain the server name.

• Collect and send logs and detailed information when you create a case with BMC Support.
• Run the Log Zipper utility.
• Attach a ZIP file to your case. You can attach a ZIP file of size up to 2 GB. You can also upload the file on FTP.

• See Collecting diagnostics in a zip file.
  Note:  You must click  Zip Logs  (not  Zip All Logs) to collect the log files.
• See How to use BMC's Managed File Transfer (MFT) to send log files to BMC Support to comply with GDPR policies?
  

There are too many records on the AR System Email Messaging forms.

There are too many records on the AR System Email Messaging forms.

• Archiving overview
• Managing the archiving process

You get the following message in the Email Engine log:

Slow processing of emails


There are outdated Email Engine files or libraries in the Email Engine installation directory.

• KA 000164912
• BMC Support