Configuring proactive problem management settings

As an administrator or a user with Problem config permission, you can configure the proactive problem management settings to specify additional stop words for a language and number of jobs. You can also specify the default number of clusters to be displayed in the heat map, and exclude fields from the data set before running a job.

You can also configure the natural language processing settings for resolution insights. For more information about the algorithm, see How incidents are clustered and resolution insights are derived in BMC Helix ITSM Insights.

Users who do not have Problem config permission cannot modify the proactive problem management settings.

Stop words and regular expression

You can configure the machine learning settings to specify additional stop words for a language. You can include stop words, such as location name, site names, company name, and their patterns that appear in the incident text but might not be helpful for clustering. You can use regular expressions to define stop word patterns, such as a combination of words and sentences, which the algorithm can either remove or extract based on your preference while clustering.

In version 23.3.04 and later, new jobs only support stop word files in YAML (yet another markup language) format. However, older jobs created in previous release versions still support stop word files in TXT format.

You can download the sample .YAML stop word file and include the following details in it:

List of stop words
Prefix and postfix notations by using wildcards
Patterns of stop words by using regular expressions based on your use case.

The following template-based examples show how you can define stop word patterns using regular expressions in YAML file:

Tip

If incidents contain template-based details, we recommend using a template-based stop word file that includes regular expressions for removing or extracting stop words, as shown in the examples.
However, for incidents that contain simple stop words without any template, such as other, then, and if, you may define the words in the stop_word section of the YAML stop word file for extraction or removal.

Example 1: Using regular expression to remove words and sentences from getting clustered

Using regular expression to remove words and sentences from getting clustered

While generating clusters in the Proactive problem management dashboard, you can define patterns by using regular expressions to remove words and sentences from incident details.
This example displays how you can remove words and sentences from the template-based incident.

Template-based Incident details

Reported by: John Smith
Address: 123 Main Street, New York, NY 10001
Email: joe@example.com
Phone: (555) 123-4567
Date of Birth: 07/15/1988
Social Security Number: 123-45-6789
Problem Summary: User unable to connect to the corporate VPN using IP address 192.168.1.101. The VPN access page https://vpn.example.com shows a timeout error after entering credentials.

Template-based Stop word file in YAML

The following stop word file is used to remove the irrelevant details from the incident details while generating clusters:

# Regex section contains regular expressions used for matching patterns in text.
# These can be used for tasks like text extraction and removal.
regex:
  removal:
    # Match email addresses
    - '\b[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}\b'
    # Match URLs
    - '\b((https?|ftp):\/\/[^\s\/$.?#].[^\s]*)\b'
    # Match phone numbers (US format)
    - '\b(?:\+1)?\s?$?\d{3}$?[-.\s]?\d{3}[-.\s]?\d{4}\b'
    # Match phone number (Indian format)
    - '\b(?:91[-.\s]?)?\d{5}[-.\s]?\d{5}\b'
    # Match IP addresses
    - '\b((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b'
    # Match dates of birth (MM/DD/YYYY)
    - '\b(0[1-9]|1[0-2])\/(0[1-9]|[12][0-9]|3[01])\/\d{4}\b'
    - '\b\d{3}-\d{2}-\d{4}\b'

# Wildcards section contains patterns with wildcard characters for flexible matching.
# These are used in scenarios where exact matching is not required, allowing for variability.
wildcards:
  # Match any word that ends with .com
  - '%.com'
# Note: All values in each section must be in between single quotes
# (unless any of the stop words, regex, or wildcards contain single quotes; enclose such strings within double quotes)

Output

The email addresses, URLs, phone numbers (US and Indian format), IP addresses, and date of birth details are removed from incident details before generating clusters.

Example 2: Using regular expression to extract specific word and sentences pattern for clustering

Using regular expression to extract specific word and sentences pattern for clustering

While generating clusters in the Proactive problem management dashboard, you can define patterns by using regular expressions to extract certain words and sentences from incident details.
This example displays how you can extract the details after "My requests" in the DESCRIPTION section of the template-based incident.

Template-based incident details

Customer Info:
ID: m756871
Name: ABC (ABC-DEMO.COM)
Email: ABC@xyz.com
Business: CACA
Business Group: CACA Asia Pacific
Enterprise: Agricultural Supply Chain
Phone: 011-2689 6767
Manager: QWERRTY
Region: Asia Pacific
Country:
City:

Form Name: TCE - Requests

DESCRIPTION
Create a personalized description to help you locate this ticket in “My Requests”:: Matrix Execution required by 12/6

REQUEST INFORMATION
What do you need help with today?: I need to make a request related to Master Data

Stop word file in YAML

# Stopwords section contains common words that should be excluded from text processing.
# These words are typically considered insignificant for the purpose of analysis.
stop_words:
- 'Pacific'
- 'Impact'

# Regex section contains regular expressions used for matching patterns in text.

# These can be used for tasks like validation, searching, or text extraction.
regex:
removal:

extraction:

- '(?<=::).*$'

# Wildcards section contains patterns with wildcard characters for flexible matching.
# These are used in scenarios where exact matching is not required, allowing for variability.
wildcards:
  # Match any words that starts with ERR
  - 'Parameter%'
  # Match any word that has prod in between
  - '%prod%'
  # Match any word that ends with .com
  - '%.com'

Output

The Matrix Execution required by 12/6 value from the incident is used for clustering.

When a problem coordinator selects a specific language while defining a job, the default stop words and additional stop words, and their patterns uploaded here or at the time of job definition is used during job processing.

The following video illustrates how you can configure the resolution insights functionality in BMC Helix ITSM Insights:

https://youtu.be/tmLvSHCBX_w

To configure additional stop words

In BMC Helix ITSM Insights, click the icon.
The Settings page is displayed.
Navigate to Settings > Proactive problem management > Configure machine learning.
The Configure Machine Learning page is displayed.
Select a language from the list. The language selected is used for processing incident data and stop words.

Important
For this release, Russian and Korean languages are not supported.
For the supported languages, BMC Helix ITSM Insights has a default stop words library.
Stop words are commonly used words that add little meaning and will be filtered while processing. Example of stop words: a, an, the, and, it.
Select a language and click Update Stop Words.
In the Update Stop Words window, download the sample stop word file for reference.
Review the sample stop word file to understand the specified format and create the stop word file for uploading.

Define stop words and their patterns by using regular expressions in a .YAML file, and validate the file by using any YAML validator.
Important
While creating a stop word file, you must adhere to the YAML format mentioned in the examples. See Stop word file examples for creating stop word patterns by using regular expressions.
You can validate your regular expression and YAML file in https://regex101.com/ and https://yamlchecker.com/ respectively.

Upload the .YAML file that contains additional stop words for the selected language.
Proactive problem management has a built-in library of a standard set of stop words for every supported language. The algorithm refers to the library and the .YAML file while processing incident information.

Important

You can continue using the stop word file that was uploaded in .TXT format in the previous releases. However, in version 23.3.04 and later, you must use a YAML file to define stop words and their patterns.
Every time you upload a new stop word file, it overrides the old file. The last updated YAML file is used for creating clusters.
While generating cluster labels in the dashboard from the relevant incidents, the algorithm compares the incident description words with the stop word file. Therefore, cluster labels do not contain words mentioned in the stop word file.

View the default stop words of English locale

from
subject
re
edu
use
need
able
bmc
com
abc
however
ourselves
alone
us
would
already
most
off
back
and
none
because
where
first
it
nevertheless
too
each
whereupon
wherein
as
this
whatever
always
serious
then
‘ve
not
he
them
n't
even
thru
anyway
above
eight
'm
or
besides
hereby
than
during
being
never
therein
has
does
hereupon
whereafter
is
becomes
ca
get
seemed
nowhere
nobody
rather
whenever
yourselves
few
may
elsewhere
but
my
again
will
're
more
once
'm
otherwise
anything
various
had
together
within
via
are
fifty
afterwards
mine
how
many
thereupon
should
himself
everything
against
sixty
perhaps
although
's
along
except
his
whether
anywhere
must
one
their
s'
for
no
someone
upon
meanwhile
might
here
namely
indeed
under
n‘t
almost
least
forty
we
everyone
toward
before
if
show
about
please
another
through
've
to
unless
side
also
move
any
can
just
two
throughout
three
could
me
across
whence
else
five
amongst
mostly
wherever
four
hence
front
anyone
herself
whereby
somehow
whose
ever
go
was
themselves
since
when
noone
's
with
same
did
n’t
per
yourself
am
become
beside
well
why
they
'll
regarding
ours
her
give
our
made
‘re
thereby
an
much
full
ten
take
're
out
such
therefore
over
still
seem
'd
former
latter
next
everywhere
‘d
quite
becoming
amount
sometime
twenty
doing
last
name
’ll
whom
these
every
latterly
make
seeming
among
part
that
twelve
either
of
myself
say
hundred
thereafter
those
at
you
down
several
herein
been
beyond
nine
him
towards
what
onto
do
both
sometimes
thence
moreover
the
due
beforehand
empty
fifteen
thus
anyhow
have
whereas
eleven
done
yours
she
your
whoever
who
only
own
somewhere
formerly
between
others
neither
below
up
its
hers
be
whither
were
into
yet
became
‘m
less
until
all
see
used
seems
now
enough
often
call
without
on
a
something
further
put
though
after
while
bottom
six
nothing
hereafter
behind
which
very
so
top
using
whole
i
there
really
in
’d
nor
third
cannot
around
itself
by
keep
other
’ve
‘ll
some

View the use of % in stop words

The following table describes the usage of % in stop words:

Incident summary	Stop word	Description
ITSMInsights is running low on memory	ITSM%	Removes the stop word ITSM and the characters following it. In this case, ITSMInsights is removed from the resulting cluster label.
ITSMInsights is running low on memory	%Ins%	Removes the stop word Ins and the characters preceding and following it. In this case, ITSMInsights is removed from the resulting cluster label.
ITSMInsights is running low on memory	%Insights	Removes the stop word Insights and the characters preceding it. In this case, ITSMInsights is removed from the resulting cluster label.

Click Save.
The stop words file is uploaded.

To configure the maximum number of jobs

In BMC Helix ITSM Insights, click the icon.
The Settings page is displayed.
Navigate to Settings > Proactive problem management > Settings
Specify the maximum number of jobs per user.
Click Save.

To configure the default number of clusters in the heat map

Configure the number of clusters that you want to display in the heat map by default. You can specify any value between 5 and 50. You can use the Max number of clusters field in the heat map to increase or decrease the number of clusters.

Navigate to > Proactive problem management > Settings.
In # of clusters shown on heatmap, specify the number of clusters you want to display in the heat map by default.
Click Save.

To configure the default minimum cluster quality value in heatmap

You can configure the default minimum value of the Cluster quality slider that appears in the Proactive problem management heatmap. As a result of this configuration, you can view only those clusters that have a score higher than or equal to the value that you set.

Navigate to > Proactive problem management > Settings.

In Minimum cluster quality, specify the default minimum cluster quality value to be displayed in the slider.

Click Save.

To exclude fields from a data set

As an administrator or a user with Problem config permission, you can exclude fields in a data set. The excluded fields will be unavailable for the users to include when creating and running jobs in the Proactive problem management workspace.

Important

If users have created jobs by using some fields before they were excluded, those jobs remain unaffected. However, when users try to edit those jobs, Proactive problem management displays the list of fields that are deleted from the job data set and prompts the users to save or run the job again to apply the changes.

Only the fields present in the exclusion list can be excluded from the data set. Submitter is the only system field that can be excluded from the data set.

Navigate to > Proactive problem management > Configure data set.
The Configure data set page is displayed.

From the list, select the fields that you want to exclude.
Click Save.

To configure natural language processing for resolution insights

The resolution insights algorithm has a set of predefined indicator words to derive knowledge sources and resolution details from incidents.
You can improve the results delivered by the algorithm by providing additional indicator words and other details based on your line of business.

Navigate to > Natural language configuration > Configuration.

In the Natural language for resolution insights area, enter the details in the fields as described in the following table:

Field	Description
Indicator words for knowledge sources	Enter the indicator words for the algorithm to identify the knowledge sources. The set of characters after the indicator word is considered a knowledge source by the algorithm. You can append to the list of standard words based on your specific workflow requirements. The words that you add appear in the textbox. Click to view the standard indicator words for knowledge sources KBA KB KB- KCS KCS- KCS-Reference- Important The standard indicator words for knowledge sources appear based on the language that you select.
Resolution indicator word(s)	Enter the indicator words for the algorithm to identify the resolution. The set of characters after the indicator word is considered a resolution by the algorithm. You can append to the list of standard words based on your specific workflow requirements. The words that you add appear in the textbox. Click to view the standard resolution indicator words Action taken Solution Resolution Steps Taken to resolve the issue Steps Taken to resolve issue Steps taken to troubleshoot and resolve the problem Resolution Provided Important The standard resolution indicator words appear based on the language that you select. You can add resolution indicator words of a different language by either pasting those words in this field or by typing them using the native keyboard.

Field

Description

Indicator words for knowledge sources

Enter the indicator words for the algorithm to identify the knowledge sources.
The set of characters after the indicator word is considered a knowledge source by the algorithm.
You can append to the list of standard words based on your specific workflow requirements. The words that you add appear in the textbox.

Click to view the standard indicator words for knowledge sources

KBA
KB
KB-
KCS
KCS-
KCS-Reference-

Important

The standard indicator words for knowledge sources appear based on the language that you select.

Resolution indicator word(s)

Enter the indicator words for the algorithm to identify the resolution.
The set of characters after the indicator word is considered a resolution by the algorithm.
You can append to the list of standard words based on your specific workflow requirements. The words that you add appear in the textbox.

Click to view the standard resolution indicator words

Action taken
Solution
Resolution
Steps Taken to resolve the issue
Steps Taken to resolve issue
Steps taken to troubleshoot and resolve the problem
Resolution Provided

Important

The standard resolution indicator words appear based on the language that you select.
You can add resolution indicator words of a different language by either pasting those words in this field or by typing them using the native keyboard.

In the Phrases section, select a language and attach the .csv file containing proper nouns and verbs, and also the words (noisy phrase) to be ignored.

Click to view the format of the .csv file

Noisy phrases are words such as Hello, Hi, Greetings, etc that are not processed by the algorithm.
Proper nouns and verbs are processed on priority by the algorithm.
The uploaded file appears as a row in the Phrases section.
Important
- Only .csv file is supported for uploading noisy phrases, proper nouns, and verbs.
- The maximum size for the .csv file that you can upload is 1 MB.
(Optional) Click Replace the file to update the existing .csv file for a language.
Click Save.

For information about configuring job settings for resolution insights, see Configuring-one-time-job-settings-for-proactive-problem-management and Configuring the recurrent job settings for proactive problem management.

Configuring proactive problem management settings

Stop words and regular expression

Using regular expression to remove words and sentences from getting clustered

Using regular expression to extract specific word and sentences pattern for clustering

To configure the maximum number of jobs

To configure the default number of clusters in the heat map

To configure the default minimum cluster quality value in heatmap

To exclude fields from a data set

To configure natural language processing for resolution insights

On this page