Troubleshooting performance issues with Email Engine

If there are performance issues with Remedy Email Engine, you might not receive email notifications at all or you might not receive them in a timely manner.

To resolve performance issues with Email Engine, follow the guidelines described in this topic. Alternatively, create a BMC Support case.

Best Practice

We recommend that you refer to BEST FAQ on AR System Email Engine Issuesfor resolutions to the most common questions on Email Engine.

Issue symptoms

You do not receive email notifications.
A large number of outgoing email messages are queued up and are not processed (or processed very slowly) from the AR System Email Messages form.

Issue scope

One or more users do not receive email notifications in a timely manner.
One or more Email Engines are affected in a Server Group.

Diagnosing and reporting an issue

Perform the following steps to troubleshoot outgoing issues with Email Engine:

Task	Action	Steps	Reference
1.	Identify whether the problem is with outgoing or incoming emails.	For the outgoing email messages, check whether you do not receive notifications at all or you don't receive them in a timely manner. See Task 2. For the incoming email messages, check whether the incoming messages take a long time to create new incidents and approve requests.	See Troubleshooting-incoming-issues-with-Email-Engine.
2.	Identify if the outgoing email messages queue in the AR System Email Messages form.	Follow one of these steps to search the AR System Email Messages form for outgoing messages with a Send status of Yes: Use the following advanced query: 'Message Type' = "Outgoing" AND 'Send Message' = "Yes" Consider adding a date range to your query, for example: 'Message Type' = "Outgoing" AND 'Send Message' = "Yes" AND 'Create Date:' >= ($DATE$ - 86400) Check the configuration of the outgoing mailbox. Consider applying the most current hotfix. If there a large number of records returned by the query, see Resolutions for common issues.	See Troubleshooting-outgoing-issues-with-Email-Engine.
3.	If you do not notice any queuing in the AR System Email Messages form, check the application workflow.	Check any custom workflow that involves email notifications. Enable API logs, Filter logs, SQL logs, and Escalation logs to track the expected customer workflow actions, as required.	See Extending the Notification Engine escalation pools
4.	Check the configuration of Email Engine.	Check if the following settings are correct in the AR System Email Mailbox Configuration form and EmailDaemon.properties file: Polling Interval (In the Mailbox Configuration form) Polling Is Minutes (T or F)(* Confirm) NumberOfSenderThreads OutgoingMessagesQueueSize SendEmailSetSize RPC	To review all the properties or settings in the EmailDaemon.properties file, see Settings in the EmailDaemon.properties file.
5.	Review logs.	Analyze the Email Engine logs, and check for SEVERE errors for outgoing mailboxes. The following are the locations for the Email Engine logs: (Windows) Location: emailEngineInstallDir\AREmail\Logs Name: email.log (UNIX) Location: emailEngineInstallDir/logs Name: emaild.log and/or emaild.sh_log	If SEVERE errors exist, see Troubleshooting-outgoing-issues-with-Email-Engine.
6.	Enable logs.	Enable FINEST logging for Email Engine.	See How to enable Email Engine Logs in 9.x and later?
7.	Enable AR System logging.	To enable the server-side API logs, Filter logs, and SQL logs in AR System: Open AR System Administration Console. Select System > General > Server Information. The Server Information page is displayed. On the Log Files tab, select the API Log check box, Filter Log check box, and SQL Log check box. Click Apply. Click OK. The server-side logs in AR System are enabled. Note: If the Email Engine connects to a Load Balancer, you must enable logs on all AR System servers in the group. To enable logs on multiple servers in a server group, select the Server Group Logging option in the Server Group Administration Console.	See Managing-logs-for-server-group.
8.	Reproduce the issue or wait for it to occur.	Check if you can reproduce the performance issue. Otherwise, wait until the issue occurs on its own.
9.	Collect and analyze the Email Engine logs and AR System logs.	Copy the log files to another location and then review them so that they do not get overwritten. The log files might be huge in size depending upon the volume. Collect the Email Engine logs including the .bak file, if it exists. The following are the locations for the Email Engine logs: (Windows)<ARSystem Install Dir>\AREmail\Logs\email.log (UNIX)<ARSystem Install Dir>/AREmail/Logs/email.log Collect API logs, Filter logs, and SQL logs from each AR System server in the server group.	See How can I measure the throughput of my Email Engine?
10.	Use AR System Log Analyzer.	Try to identify long queries from Email Engine by using the API logs and SQL logs from the AR System server that owns the email operations. Check the API logs for API calls that show the API call GetListEntryWithFields (GLEWF) or GetListEntry (GLE) that are from Email Engine. Then use the corresponding TID and RPC ID to find the SQL and ending API call. <API > <TID: 0000000589> <RPC ID: 0000191683> <Queue: Prv:390634> <Client-RPC: 390634 > <USER: Remedy Application Service > <Overlay-Group: 1 > /* Wed Jan 22 2020 13:40:36.8220 / +GLE ARGetListEntry -- schema AR System Email Messages from E-mail Engine (protocol 25) at IP address 127.0.0.1 using RPC // :q:0.0s <API > <TID: 0000000589> <RPC ID: 0000191683> <Queue: Prv:390634> <Client-RPC: 390634 > <USER: Remedy Application Service > <Overlay-Group: 1 > / Wed Jan 22 2020 13:40:40.6590 / -GLE OK Note:* If you run Email Engine on a private RPC socket, for example, Queue: Prv:390636, it can help you to identify the corresponding API or SQL calls in the logs.	See the following topics: Running-AR-System-Log-Analyzer Viewing-AR-System-Log-Analyzer-output Analyzing-AR-System-Log-Analyzer-output If the SELECT statement generated by the GLE call, which is made by Email Engine, takes considerable amount of time to run, and if your Email Engine runs on a version earlier than 20.02, see Outgoing emails encounter slow performance where GLE API call has query taking significant amount of time to execute.
11.	Find a solution.	See Resolutions for common issues.
12.	Run the Log Zipper utility.	If you still notice issues after following the preceding steps, or if you cannot complete any of these steps, run the Log Zipper utility. You need to run the Log Zipper utility while submitting a case on the performance of outgoing emails. You must run the Log Zipper utility on each server on which you have enabled the logs. Rename the zipped file to contain the server name.	See Log Zipper utility.
13.	Create a BMC Support case	Collect and send logs and detailed information when you create a case with BMC Support. Run the Log Zipper utility. Attach a ZIP file to your case. You can attach a ZIP file of size up to 2 GB. You can also upload the file on FTP.	See Collecting-diagnostics-in-a-zip-file. Note: You must click Zip Logs (not Zip All Logs) to collect the log files. See How to use BMC's Managed File Transfer (MFT) to send log files to BMC Support to comply with GDPR policies?

Resolutions for common issues

After you determine a specific symptom or an error message, use the following table to identify the solution:

Symptom	Action	Reference
There are too many records on the AR System Email Messaging forms.	Run SQL scripts to get form counts.	Email Engine is performing poorly
There are too many records on the AR System Email Messaging forms.	Follow the data retention policies and create a custom escalation to remove the older records or enable the archiving feature.	Archiving-overview Managing-the-archiving-process
You get the following message in the Email Engine log: Slow processing of emails	Use the Throughput Analyzer.	How can I measure the throughput of my Email Engine?
There are outdated Email Engine files or libraries in the Email Engine installation directory.	Obtain the latest recommended hotfix or contact BMC Support.	KA 000164912 BMC Support