Unsupported content

 

This product is in "End of Version Support". However, the documentation is available for your convenience. You will not be able to leave comments.

Before Service Pack 5, BMC Remedy Full Text Search (FTS) required that one server in a server group act as the FTS server. With version 7.6.04 Service Pack 5, you can now install more than one FTS server in a server group.  Each of these servers acts as an independent FTS server, providing service failover.

The following topics are provided:

Overview of how FTS works in a server group

FTS within BMC Remedy Action Request (AR) System is made up of the following primary components:

  • FTS code that manages the FTS functionality
  • FTS plug-in — a Java plug-in, which is the core index engine
  • Index

As events occur within AR System that cause data to be indexed, the AR System server sends that data, along with appropriate instructions, to the FTS plug-in, which then adds, deletes, or updates data within the index. Other events could include a request to search the index. AR System server sends the search request to the FTS plug-in, which performs the search and returns the results to AR System server. The AR System server then deals with the data accordingly.

When you configure a server group for FTS, ensure that the following conditions are met:

  • Only a primary FTS indexing server(s) performs all indexing operations.
  • A primary FTS indexing server is AR System server that also has the FTS collection and conf directories located on a local disk.
  • A primary FTS indexing server hosts all instances of the FTS plug-ins.

For FTS high-availability configuration, you can configure two or more servers as FTS indexing servers.

How failover occurs in high-availability configuration

FTS requires that at least one of the servers in a server group act as a primary FTS indexing server. To provide failover, you can have two or more primary FTS indexing servers in the server group where each server acts as an independent FTS server and indexing FTS in their own Collection directory. Each FTS indexing server (known as primary FTS indexing server) manages its own separate local replica of the full text index data. When an indexing action takes place, each FTS indexing server indexes it independently. For example, if you create a form and then index a field on that form, each FTS indexing server indexes that field. Because you can have multiple FTS servers, each with its own current copy of the FTS data, you can fail over to a second server.

An AR System server is designated as a primary FTS indexing server by having a ranking record for FTS in the AR System Server Group Operation Ranking form. Each FTS server defined in the ranking form acts as an indexing server and provides FTS search services to other servers in the server group. Each defined FTS server will always index, regardless of its ranking order. However, the server that is ranked 2 contains redundant FTS data and must be used for failover. This server is not intended to be a user-facing server.

Servers in the server group that are not ranked for FTS are search-only servers. The search-only servers are user facing servers and they connect to one of the FTS indexing servers to search the FTS data.

Note

The servers that are not designated as indexing server should not be listed in the AR System Server Group Operation Ranking form.

 

The following figure shows the FTS plug-ins and FTS high-availability architecture in a server group.

BMC Remedy Full Text Search (FTS) plug-in and high-availability architecture

Primary FTS indexing servers always connect to their own local indexes. The other servers in the group can connect to any of the indexing servers. BMC recommends that all non-Primary FTS servers initially connect to the Primary FTS server which is ranked 1. The Server-Plugin-Alias parameter in the ar.cfg [or ar.conf] files of the servers specify the initial connection. This initial connection is known as the home connection.

While servicing an FTS request, if an FTS indexing server experiences a connection error, the request attempts to connect to another FTS indexing server (as specified in the AR System Server Group Operation Ranking form) until a server is found or there are no more to try.

For example, consider a scenario where you have two primary FTS indexing servers that are ranked 1 and 2 in the AR System Server Group Operation Ranking form. Your Server-Plugin-Alias parameter points to the server that is ranked 1. If this server experiences a connection error, the connection request goes to the server that is ranked 2. If for some reason even the server that is ranked 2 is down, the request then is returned as an error since no primary FTS indexing servers are available.

Important

  • A slight load on the database can occur if you are performing re-indexing and your database server is running on a low memory.
  • Avoid performing a full re-index at the same time on two FTS index servers running in your production environment, as a failover will occur if the FTS indexing server is busy while performing a re-index.
  • A slight latency in displaying the search results can occur when the failover happens as it is required to establish a connection to the next ranked server for FTS.

Configuring FTS in a server group

FTS uses one plug-in as the writer (primary) and another plug-in as the reader (secondary). The reader and writer plug-ins are installed on the FTS indexing server. In a server group, only one writer instance must be running on the designated FTS indexing server. The FTS writer (primary) serves as the reader and writer for the FTS indexing server, as well as a writer for all servers.

The writer (primary) is connected to the FTS indexing server. The reader (secondary) serves as the reader for all the other servers in the group, so you must configure all other servers to connect to the reader instance running on the FTS indexing server. The reader instance runs on a separate port on the FTS indexing server.

The events that cause data to be written to the index result in data being put into the AR System database as a queue of items to index. Only a primary FTS indexing server processes index requests from this queue. However, any instance of AR System server can send a search request to its corresponding FTS plug-in. This ensures index integrity. To further ensure integrity of the system, the FTS plug-in design is such that any launched instance defaults to a read-only state until the primary FTS indexing server specifically initializes the primary plug-in instance for writing.

Note

  • The FTS indexing server communicates with the writer plug-in for all search and indexing requests. There is no search failover for the writer plug-in running on the FTS indexing server.
  • The secondary reader plug-in serves the search requests from other servers and does not serve as a backup to the primary writer plug-in.

Each primary FTS indexing server has its own virtual queue of data to index. When AR System queues data for indexing in parallel to AR Database changes in data, it queues separately for each primary FTS Indexing server as designated by the Server Group Ranking form for FTS

FTS is configured after all servers in the group have been installed and configured to run within a server group. Each primary FTS indexing server should have its respective FTS collection directory and FTS configuration directories located on the same computer.

To set up FTS in a server group

  1. Rank the FTS servers in the AR System Server Group Operation Ranking form. (See page 33 of the BMC Remedy IT Service Management Suite 7.6.04 Installing and Configuring Server Groups White Paper.)

    Note

    You should use the FTS indexing server, which is ranked 1 in the AR System Server Group Operation Ranking form, for searching and the other FTS indexing server, which is ranked 2, as the failover server.

  2. To configure FTS, use the AR System Administration FTS Configuration form, shown in the following figure.
    To access this form, click AR System Administration Console > General > FTS Configuration.

AR System Administration FTS Configuration form

AR System Administration FTS Configuration form fields

Field nameDescription
FTS AgentUse the FTS Agent check box to enable or disable the FTS plug-in processes in the armonitor.cfg/conf file.

If you select this check box, the FTS plug-in processes are automatically uncommented in the armonitor file. If you clear this check box and save the changes, the FTS processes are automatically commented in the armonitor file. If you are using FTS in a server group, the Primary and Secondary FTS plug-in processes are commented/uncommented based on your selection.

Using the FTS Agent check box does not require manual intervention to uncomment the FTS plug-in processes in the armonitor file.

You must restart the AR System server for the changes to take effect.
Server ConfigurationSelect one of the options and perform the associated actions:
    • Single Server — (Not a server group)
      • Enter values in the FTS Port1 and Max JVM Memory1 fields for the FTS Reader and Writer instances.
      • If you select this option, the fields under Reader are unavailable because on a single server only one FTS instance serves as both reader and writer.
    • Server Group-Primary
      • Enter values in the FTS Port1 and Max JVM Memory1 fields for FTS Writer.
      • Enter values in the FTS Port2 and Max JVM Memory2 fields for FTS Reader.
      • If you select the Server Group-Primary option, the Primary Server Name field under Reader is unavailable because the primary server name is not required for configuring FTS on the primary server.
    • Server Group-Secondary
      • Enter values in the Primary Server Name and FTS Port 2 fields for FTS Reader. 
      • If you select the Server Group-Secondary option, the Max JVM Memory2 field under Reader and the FTS Port1 and Max JVM Memory1 fields under Writer are unavailable because from the secondary server you must connect only to the reader instance running on the primary server.
FTS Port1Enter the FTS port for primary server. The port range is 9988–9998.

During installation, the installer picks the available port from the port range starting from the lower port number. For more information about port numbers, see page 59 of the BMC Remedy AR System 7.6.04 Installation Guide.
Max JVM Memory1Enter the JVM heap size (in megabytes) for primary server.
Primary Server NameEnter the name of the primary server.
FTS Port2Enter the FTS port for secondary server. The port range is 99779982.

For more information about port numbers, see page 59 of the BMC Remedy AR System 7.6.04 Installation Guide.
Max JVM Memory2Enter the JVM heap size (in megabytes) for secondary server.
FTS Collection DirectoryEnter the full path of the FTS Collection directory.
FTS Configuration DirectoryEnter the full path of the FTS Configuration directory.

Notes

  • You must log on to each server and perform FTS configuration on each server using the FTS Configuration form.
  • You must restart the AR System server for the changes made to this form to take effect.
  • Ensure that the Collection and Configuration directories are available on a local drive of each indexing server. 
  • Log on to each server and ensure that the paths of the Collection and Configuration directories are identical on all the servers in the server group. For example, if the indexing servers in a server group are running on Windows and the paths of these directories are C:\data1\BMCData\BMCARSystem\FTS\Collection and C:\data1\BMCData\BMCARSystem\FTS\Configuration respectively, ensure that all reader servers have the same path set in their ar.cfg/conf file with respect to the location of the Collection directory on the indexing servers. The failover fails if the directory paths are different. All indexing servers and reader servers in a server group refer to this path in the ar.cfg/conf file. The indexing servers also refer to this path for the FTS plug-ins in the pluginsvr_conf.xml file.

To upgrade with FTS installed

When upgrading, if you plan to re-index all of your FTS indexing servers, complete the preceding procedure.

If you are upgrading and do not want to re-index the FTS indexing servers, use the following procedure:

  1. Complete any re-indexing before starting the upgrade.
  2. Stop all user activity on the servers and do not start any new re-indexing.
  3. Ensure that any remaining ft_pending table entries are completed before continuing (They should complete quickly).
    For more information about the ft_pending table, see page 297 of the BMC Remedy Action Request System 7.6.04 Configuration Guide.
  4. Disable indexing for all servers in the server group.
  5. (Optional) Designate additional servers for FTS in the AR System Server Group Operation Ranking form. (You can do this after the upgrade, but then you must restart the AR System server again.)
  6. Perform the upgrade and enable indexing for FTS on the FTS indexing servers as follows:
    1. Click the FTS tab on the Server Information form.
    2. Clear the Disable Full Text Searching and Disable FTS Indexer check boxes.
    3. Click Apply.

This ensures that Full Text Search and the FTS Indexer are enabled after the upgrade.

Note

Other than the initial deployment where all FTS index servers might be indexed at the same time, you should re-index only one server at a time to allow the other servers to handle failover requests.
Was this page helpful? Yes No Submitting... Thank you

21 Comments

  1.  
    1.  
  2.  

     

     

    1.  
  3.  
    1.  
  4.  

     

     

    1.  
  5.  
    1.  
  6.  
    1.  
  7.  
    1.  
  8. Hi :)

    I have a couple of questions.

     

    I°) Questions:
    1°) how does it work?:

    I dont understand. Here it's said:
    "When an indexing action takes place, each FTS indexing server indexes it independently. For example, if you create a form and then index a field on that form, each FTS indexing server indexes that field. Because you can have multiple FTS servers, each with its own current copy of the FTS data, you can fail over to a second server."

    Though later:
    "If you use FTS in a server group, only one server in the group can index data at a time.
    In a server group, the server that owns the full text indexing operation processes all pending indexing tasks regardless of their server of origin. (The other servers have read-only access to the index files.)"

    So in the first part I understand that all indexing servers index a new record, on the second part only the "rank 1" indexes? It seems the second one is the "old" behavior.

     

    2°) Setting plugins:
    In any cases, there can be only one record (entry?) in "AR System Administration :FTS Config", right?

    I'm taking the server group example you gave (ARS 1 to ARS 4).

    Does That mean that on ARS 1 and ARS 2:
    -> I define them both as "Server Group Primary", so it'll enable both plugins (writer and reader),
    -> In the ranking form, I declare those two server as FTS indexers (let's say ARS1 as rank 1 and ARS2 as rank 2),
    -> So both of them will FTS index real time on their own FTS collection directory,
    -> In ar.cfg I have "Full-Text-Mode: Server-Group-Primary",
    -> In ar.cfg I have only one record for "writer" plugin "Server-Plugin-alias: ARSYS.ARF.FTS ARSYS.ARF.FTS ARS1:9998" (for ARS1, for ARS2 it's ARS2), I have no record for the "reader" plugin (9977),
    -> In armonitor.cfg both "/fts/primary" and "/fts/secondary" plugin are enabled,

     

    For ARS 3 and 4:
    -> I define both those servers as "Server Group-Secondary",
    --> I set the "Primary Server Name" from "Reader" to the ARS1 for example,
    -> They are not in the ranking form, so they don't "FTS index" but send their search by default to "ARS1" (and anyway the "writer" fts plugin won't be enabled in armonitor.conf ?),
    -> In ar.cfg I have "Full-Text-Mode: Server-Group-Secondary",
    -> In ar.cfg I have only one record for the "reader" plugin "Server-Plugin-alias: ARSYS.ARF.FTS ARSYS.ARF.FTS ARS1:9977" (since ARS1 I defined ARS1 as rank1 in the ranking form), I have no record for the "writer" plugin (9998),
    --> Though in armonitor.cfg both "/fts/primary" and "/fts/secondary" plugin are disabled (at least in 8.1),
    --> Is that the normal behavior? I would have expect the "Server Group-Secondary" to have the "/fts/secondary" plugin activated in armonitor.cfg?

     

    All ARS servers:
    -> need to have the same paths for FTS (Server Information->FTS),

     

    At the end:
    -> someone creates a new incident:
    --> ARS1 and ARS2 will index it on their own FTS data collection,

    -> a people connects to ARS2 and searches something:
    --> the query goes to ARS2 since ARS2 is a "Server Group-Primary",
    --> if ARS2 plugin is down, the query will go to ARS1,

    -> a people connects to ARS3 and searches something:
    --> the query goes to ARS1 (always) as long as ARS1 answers (since it's defined in "Server-Plugin-Alias to connect to ARS1),
    --> if ARS1 plugin is down, the query will go to ARS2 (I guess it uses ranking),

     

     

  9. Hi Laurent,

    I see that there are non-FTS HA and FTS HA topics here.  It is a little confusing, I will review this and see that it gets clarified.

    To answer #1:   Each Collection Directory must be owned by only a single FTS indexing Server.  In a non-HA environment, there is only 1 server.  In an HA environment, there are two servers, each a primary FTS indexing server, but ranked #1 and #2 respectively in the Server Group Ranking form. #2 Primary will cover for #1 when #1 is not available and #3, #4 (and more user facing if you have them) need search services. #1, when back online will eventually be detected and fail-back will happen over time and as appropriate.

    To Answer #2: Each Server has a FTS Config.  You must go to each server to config it using the FTS Config tool in the Administrators console.  When using this config tool on a Primary (#1 & #2 in our diagram) You will be prompted for the information for both the Primary FTS Plugin's port and memory, as well as the secondary Port and memory. 

     

    For #3:  Both Indexers need to use the same port for the secondary instances in an HA deployment so make sure the port you chose is available on both servers.  You are also correct in that #3 and #4 are secondary (meaning they request searches from a primary/indexer) and these do not run FTS Plugins, rather they point to the #1 ranked primary (the server name) and the port.  This is why it asks for a server name (the #1 ranked server as well as the port for the secondary on that #1 ranked primary) and port but not the memory.  You can see in armonitor after running this that a primary has two armonitor entries for two separate FTS Plugin instances that are not commented out, and on a secondary, these two entries in armonitor are commented out.  They are left there so that if you reconfigure a secondary to a primary, the FTS config tool has the needed entries from install that just need to be run.  

    Hope this helps with the FTS configuration and I will look into having the doc clarified.  Appreciate your input.

     

  10. Hi David :)
    Thanks for the clarifications :)

    For Answer #1:
    But both ARS1 and ARS2 (primaries) are indexing FTS (in their own collection folder of course) at the same time, right, since they are primaries?

    Answer 2 #:
    Ok :)

    Answer 3#:
    I see... I tought that secondaries needed the "/fts/secondary" plugin to speak to ARS1/ARS2 "reader", so something like the "reader" would be also the "query" plugin... So for secondaries no need for those since they really are the "writer" and "reader", and not "query".
    Thanks a lot :)

     

    Other questions now ;)
    1°) who listens to who (for primaries server):
    Let's say that ARS1 is rank1 and so a primary FTS indexer. ARS2 is a primary as well, but rank 2. ARS3 and ARS4 are secondary.

    If I configured the secondaries to connect to ARS1 (primary server name), they will first try to connect to ARS1 for their queries, ok. If ARS1 isn't available, then the query will fall back to ARS2, ok.

    Now, for ARS1 if there is a query, it'll use his own collection, ok.

    And what for ARS2 if there is a query, it'll use his own collection, right? Since he's a primary you don't set the "Primary server name", it's the "local" server name.

    Now let's say the FTS plugin on ARS2 are down (writer / reader). Will ARS2 try to send its query to ARS1?

     

    2°) reindexing?:
    Let's say that ARS1 is rank1 and so a primary FTS indexer. ARS2 is a primary as well, but rank 2. ARS3 and ARS4 are secondary.
    So ARS1 and ARS2 update their own indexes (collection) at the same time, but ARS1 answers other ARS servers queries (all but the ARS2 since it's a primary so ARS2 queries his own collection).

    Let's say ARS1 is down for a day.
    Like you said ARS2 takes the lead to "answer" other ARS server requests and it continues to index like before. When ARS1 goes back online, how does ARS1 know what records he needs to reindex since for example he "missed" 100 incidents since it was down (the records created when he was down, for a day)? Do we need to do a full reindex on ARS1?

     

    3°) weird case scenario:
    Let's say that ARS1 is rank1 and so a primary FTS indexer. ARS2 is a primary as well, but rank 2. ARS3 and ARS4 are secondary.
    I configured the secondaries to connect to ARS1 (primary server name), they will first try to connect to ARS1 for their queries.

    Now when will the fail over fire actually since there are now two plugins on ARS1? One for writer (indexing), one reader (to get and answer queries)?

    Let's say that ARS1 "writer" plugin doesn't work anymore. So that means ARS1 can't update its FTS collection index anymore, but the "reader" plugin is still up.
    What happens in this case? ARS1 can continue to handle queries, but his collection isn't up to date anymore, so search results on his collection wouldn't be perfect. Will he continue to handle the queries coming through its "reader" plugin or the "reader" plugin would detect that the "writer" plugin is down and so will "refuse" (?) the query so it would be failovered to ARS2?

     

     

  11. #1 - Yes both at the same time and both independently to their own local collection directory.

    #3 - Secondaries do not use local FTS Plugins.  They connect directly to the Server : Port specified during the FTS Config right from ARS - this will be stored in the Plugin Alias in ar.cfg.

    Other #1 - All indexers use their own collection/ local index always.  Indexer own a collection and they do not fail-over.  They are designed to be run on Admin/Integration server for Rank #1 Primary, and for Rank #2 - this is designed to be a redundant system (for fail-over) and not a user facing server.  It can be, but with the caveat that if the plugins are down, there is no fail-over by design of HA concepts.

    #2  the design of the FTS HA queuing mechanism is such that each index request is simultaneously and independently queued for each Ranked server for FTS.  There is a 1-1 relationship between the servers that are configured as Primary, and those that are Ranked.  Only configured primary servers should be ranked for FTS.

    #3-reading will still happen, monitor should bring up the failed plugin, should that even happen, shortly after it goes down and any FTS indexing activity would be caught up via the queue for that server.

    1. #1 / #3:
      Ok ^_^

      #1 bis:
      I see. I was asking this because some customers don't have enough money or hardware to create a "real" server group with 2 font ARS servers and one "admin" one (not front), so they just do something like 2 front ARS servers, so both of them would be "primaries" in this scenario.
      Thanks (smile)

       

      #2 bis:
      I see, thanks (smile)

       

      #3 bis:
      Ok, fair enough.

       

      Thanks David for the help, really appreciate it (smile)

  12.  

     

     

     

  13.  
© Copyright 2013 - 2017 BMC Software, Inc.
Legal notices