BMC Remedy Action Request System 8.

Configuring full text

search

Author: Confluence Admin

Date: 21-Oct-2013 05:02
URL: https://docs.bmc.com/docs/display/ars81/Configuring+full+text+search

Page 1 of 20
 BMC Remedy Action Request System 8.1

Table of Contents
1 FTS tab configuration options __________________________________________________________ 4
2 FTS Configuration form in the AR System Administration Console _____________________________ 7
3 Creating index files in a different directory from the default __________________________________ 11
3.1 To create index files in a different directory ___________________________________________ 11
4 Scheduling scans for updates _________________________________________________________ 12
4.1 To schedule a scan for updates ___________________________________________________ 12
5 Configuring the Ignore Words List ______________________________________________________ 13
6 Displaying FTS weight in a results list ___________________________________________________ 14
6.1 To display FTS weight in a results list _______________________________________________ 14
7 Providing a shortcut for specifying expanded FTS qualifications ______________________________ 15
8 Configuring FTS for localization _______________________________________________________ 16
9 Advanced FTS configuration files ______________________________________________________ 18
9.1 To use third-party configuration files ________________________________________________ 18
10 Adding FTS licenses ________________________________________________________________ 20
10.1 To add an FTS license __________________________________________________________ 20

Page 2 of 20
 BMC Remedy Action Request System 8.1

This section contains information about configuring FTS in BMC Remedy AR System.

FTS tab configuration options

FTS Configuration form in the AR System Administration Console
Creating index files in a different directory from the default
Scheduling scans for updates
Configuring the Ignore Words List
Displaying FTS weight in a results list
Providing a shortcut for specifying expanded FTS qualifications
Configuring FTS for localization
Advanced FTS configuration files
Adding FTS licenses

Page 3 of 20
 BMC Remedy Action Request System 8.1

1 FTS tab configuration options

This section outlines the configuration options for FTS from the FTS tab of the AR System Administration:
Server Information form.

Full Text Search configuration options

(Click the image to expand it.)

FTS tab fields

Field name Description

Disable Full Controls whether the server uses the full text search engine for searching. When disabled,
Text the server uses the search capability of the underlying database. This setting does not
Searching apply to multi-form searching.

FTS The location in the file system where search engine index files are stored.
Collection
Directory
Note

If you change the directory in this field, update the pluginsvr_config.xml file
with the correct directory path. This file is in
<ARSystemInstallDir>\pluginsvr\fts.

Page 4 of 20
 BMC Remedy Action Request System 8.1

FTS The location in the file system where search engine configuration files are stored.
Configuration
Directory

Full Text The maximum number of search results returned from the search engine. The default
Search value is 10,000. To limit the number of search results (because of constraints on the
Threshold computer where the search engine is running), reduce the threshold. If you change this
option, you must restart the AR System server for the change to take effect.

Indexing Defines the number of minutes the server waits between periodic attempts to index
Failure entries that failed to index for an unexpected reason in a prior attempt. The default value
Recovery is 60 minutes.
Interval

Temporary During the processing of search results, the server creates a temporary table if the
Table number of FTS matches reaches this value. If the number of FTS matches is less than
Threshold this value, the server uses the SQL IN operator for a query on an existing table. The
default value is 200.

Complex During the processing of search results, the server combines results from subqueries to
Search arrive at the final result set. If the number of rows created during processing exceeds this
Threshold value, the server returns an error message indicating that the search is too complex. The
default value is 1,000,000.

Indexer Number of seconds before a signal is sent to the server that owns the full text indexing
Server Group operation (if no signal is pending). When a server is not the owner of the full text indexing
Signal Delay operation and generates indexing work, that server signals the server that is the owner of
indexing. To reduce the frequency of signals sent, the signaling is conducted on a delayed
basis. When indexing is generated, the server checks whether a signal is pending. If so,
the server does not need to take any action. If a signal is not pending, the server creates
a pending signal to be sent in the specified amount of time. If the full text signal delay
configuration value is changed, the duration of any pending delay interval does not
change. The change takes effect the next time a delay interval is started. Values are

Default--10 seconds
Minimum--1 second
Maximum--None

Case Defines whether full text searching is case sensitive or insensitive. This setting affects all
fields indexed for full text search and affects how the indexes are built. Therefore,
changes to this setting trigger an automatic re-index. The default setting is case
insensitive.

Page 5 of 20
 BMC Remedy Action Request System 8.1

Search Defines how the server modifies qualifications received from the client. The choices are
Options
Force Leading & Trailing Wildcards
Ignore Leading & Force Trailing Wildcards
Ignore Leading Wildcards
Remove Leading & Trailing Wildcards
Query Unchanged (default)

Reindex Initiates the re-indexing of all fields selected for full text indexing. This check box is
disabled if a re-index is in progress. The dialog box must be redisplayed for the check box
to become active following completion of the re-indexing.

Disable Full Controls whether the server processes pending indexing tasks. When disabled, indexing
Text Indexer tasks are still recorded for processing at a later time when indexing is enabled.

Use FTS in Controls whether the server uses full text searches when executing queries during
Workflow workflow that have full text indexed fields in the qualification. By default, this option is
selected. If you clear this check box, the server uses the search capability of the
database.

Ignore Words Defines which words are ignored during indexing.

List

Title Field Defines the relevancy weight for the Title field of a form in a multiple-form search. (See
Weight Configuring forms for multiple-form FTS.) A value of 1 is the baseline and does not
increase the relevancy weight. If you leave the field empty, the weight is 1. To increase
the weight, enter any number from 1.1 to 2. To decrease the weight, enter any number
from 0.9 to 0.1.

Environment Defines the relevancy weight for the Environment field of a form in a multiple-form search.
Field Weight (See Configuring forms for multiple-form FTS.) A value of 1 is the baseline and does not
increase the relevancy weight. If you leave the field empty, the weight is 1. To increase
the weight, enter any number from 1.1 to 2. To decrease the weight, enter any number
from 0.9 to 0.1.

Keywords Defines the relevancy weight for the Keywords field of a form in a multiple-form search.
Field Weight (See Configuring forms for multiple-form FTS.) A value of 1 is the baseline and does not
increase the relevancy weight. If you leave the field empty, the weight is 1. To increase
the weight, enter any number from 1.1 to 2. To decrease the weight, enter any number
from 0.9 to 0.1.

Page 6 of 20
 BMC Remedy Action Request System 8.1

2 FTS Configuration form in the AR System

Administration Console
You can use the AR System Administration FTS Configuration form to configure Full Text Search (FTS). You
can access this form from the BMC Remedy AR System Administration Console.

You must configure FTS manually using the AR System Administration FTS Configuration form in the
following scenarios:

If you select the Install option to install FTS version 8.1.00

If you select the Upgrade option to upgrade an existing version of FTS to version 8.1.00.

Note

After completing the changes to this form, continue with the FTS configuration. See
Configuring full text search.

AR System Administration FTS Configuration form

Page 7 of 20
 BMC Remedy Action Request System 8.1

AR System Administration FTS Configuration form fields

Field name Description

FTS Agent Use the FTS Agent check box to enable or disable the FTS plug-in processes in the
armonitor.cfg/conf file.

If you select this check box, the FTS plug-in processes are automatically uncommented in
the armonitor file. If you clear this check box and save the changes, the FTS processes
are automatically commented in the armonitor file.

If you are using FTS in a server group, the Primary and Secondary FTS plug-in processes
are commented/uncommented based on your selection.

Using the FTS Agent check box does not require manual intervention to uncomment the
FTS plug-in processes in the armonitor file.

You must restart the AR System server for the changes to take effect.

Page 8 of 20
 BMC Remedy Action Request System 8.1

Server Select any one of the options and perform the following:
Configuration
Single Server (Not a server group)
Enter values in the FTS Port1 and Max JVM Memory1 fields for the FTS
Reader and Writer instances.
If you select this option, the fields under Reader are unavailable because on
a single server only one FTS instance serves as both reader and writer.
Server Group-Primary
Enter values in the FTS Port1 and Max JVM Memory1 fields for FTS
Writer.
Enter values in the FTS Port2 and Max JVM Memory2 fields for FTS
Reader.
If you select the Server Group-Primary option, the Primary Server Name
field under Reader is unavailable because primary server name is not
required for configuring FTS on the primary server.
Server Group-Secondary
Enter values in the Primary Server Name and FTS Port 2 fields for FTS
Reader.
If you select the Server Group-Secondary option, the Max JVM Memory2
field under Reader and the FTS Port1 and Max JVM Memory1 fields under
Writer are unavailable because from the secondary server you must
connect only to the reader instance running on the primary server.

For more information, see Configuring full text search for a server group.

FTS Port1 Enter the FTS port for primary server. The port range is 9988 - 9998.

During installation, the installer picks the available port from the port range starting from
the lower port number. For more information about port numbers, see Understanding port
numbers.

Max JVM Enter the JVM heap size (in megabytes) for primary server.
Memory1

Primary Enter the name of the primary server.

Server Name

FTS Port2 Enter the FTS port for secondary server. The port range is 9977 - 9982.

For more information about port numbers, see Understanding port numbers.

Max JVM Enter the JVM heap size (in megabytes) for secondary server.
Memory2

FTS Enter the full path of the FTS Collection directory.

Collection
Directory

Page 9 of 20
 BMC Remedy Action Request System 8.1

FTS Enter the full path of the FTS Configuration directory.

Configuration
Directory

Note

You must restart the BMC Remedy AR System server for the changes made to this form to take
effect.

Page 10 of 20
 BMC Remedy Action Request System 8.1

3 Creating index files in a different directory from

the default
By default, index files are stored in <ARSystemInstallDir>\ftsconfiguration\collection.

3.1 To create index files in a different directory

1. Update the FTS Collection Directory field on the FTS tab of the AR System Administration: Server
Information form.
See FTS tab configuration options.
Alternatively, you can update the Full-Text-Collection-Directory option of the ar.conf file.
See ar.cfg or ar.conf.
2. Update the pluginsvr_config.xml file with the correct directory path.
The pluginsvr_config.xml file is in <ARSystemInstallDir>pluginsvr\fts.

Page 11 of 20
 BMC Remedy Action Request System 8.1

4 Scheduling scans for updates

Updates to entries for join, vendor, and view form types are not always generated in the same manner as
regular forms. To ensure accuracy in full text searches, schedule scans for updates in these types of forms.
When the forms are scanned, the AR System server indexes only the entries that have been added and
changed.

The scans work as follows:

For join forms, the server can detect which fields on the join form represent last-modified timestamps
on base forms. Using those timestamps, the server scans for updates at the scheduled times.
For vendor and view forms that contain a core field with field ID 6 (equivalent of a last-modified
timestamp), the server scans for updates at the scheduled times.
If the form does not contain a field with ID 6 (vendor or view forms) or any last-modified timestamps
(join forms), the form cannot be scanned for updates only, and the server re-indexes all indexed fields
on the form each time the form is scheduled to perform a scan.
Deleted entries (or entries that disappear because join key fields are changed in base forms) are not
detected, and the entries are represented in the index until you complete a field re-index. For more
information, see Re-indexing considerations.

Use caution when scheduling scan intervals. Do not overload the system with a large number of form scans
at small intervals, especially those that perform a complete reindex because of the unavailability of last
modified timestamps.

Note

If using the BMC Remedy AR System interface is the only way your organization updates the
database table associated to a view form, you do not need to schedule scanning for that view form.

4.1 To schedule a scan for updates

1. In BMC Remedy Developer Studio, open the form.
2. Select the Definitions tab.
3. Expand the Other Definitions panel and the Full Text Search panel.
4. Select the scan times for updates to fields that have been indexed for FTS.
5. Save the form.

Page 12 of 20
 BMC Remedy Action Request System 8.1

5 Configuring the Ignore Words List

You can configure the FTS engine to ignore frequently used words (such as and, the, because, and so on)
or words that you do not want indexed. Adding entries to the Ignore Words List saves space in the FTS
index and speeds up text searches. The FTS option comes with a default set of ignored words that you can
modify as needed.

Accrue searches that contain words from the Ignore Words List do not find any matching BMC Remedy AR
System requests for those words. However, the accrue search retrieves requests for the other search terms.
For restrictions on FTS, see Limitations of FTS.

Note

The Ignore Words List is different for each supported language.

For information about adding words to the Ignore Words List, see FTS tab configuration options.

Page 13 of 20
 BMC Remedy Action Request System 8.1

6 Displaying FTS weight in a results list

In the Results List Fields panel on the Definitions tab, you can configure results lists to display the FTS
weight for all requests retrieved.

6.1 To display FTS weight in a results list

1. In BMC Remedy Developer Studio, open the form for which you want to define the results list format.
2. Select the Definitions tab.
3. Expand the Other Definitions panel and then the Result List Fields panel.
4. Click Add.
5. In the Field Selector dialog box, select WEIGHT, and click OK.
The WEIGHT field displays the weighted value of retrieved BMC Remedy AR System requests when
you create a results list in the browser. If sorted by descending weight, the requests are listed in
order, based on a relevance factor calculated by the search engine.
6. If necessary, specify a Separator and Width.
7. Save the form.

Page 14 of 20
 BMC Remedy Action Request System 8.1

7 Providing a shortcut for specifying expanded

FTS qualifications
To provide a shortcut method for specifying expanded FTS qualifications, add a form search field (a
display-only field with field ID 178) to a form. Then, users can use that field to search all full text indexed
fields on the form with an expanded OR search. For example, if the Description and Worklog fields on a form
are indexed for FTS and the user performs a QBE search by supplying the search term firewall in a form
search field, the qualification generated on the server is:

'Description' LIKE "firewall" OR 'Worklog' LIKE "firewall"

On a form where a single attachment field is the only field indexed for FTS, you can use this feature to
provide a QBE search for the attachment field. Otherwise, only the advanced search bar method is available
for searching attachments.

Important

Use caution when labeling the form search field so that users do not get the impression that using
this field searches all fields on the form. The feature searches only fields indexed for FTS.

Note

This feature is available only from version 7.0.00 or later clients. For environments with pre-7.0.00
clients, hide this field for those clients by using client-side workflow when $VERSION$ < " 7"
(the space in front of the 7 is intentional). If the field is visible and used in pre-7.0.00 clients, the
qualification is not sent to the server (unbeknownst to users), potentially resulting in an unqualified
query. Also, if users on a system without a full text search server license enter a qualification in the
form search field, the AR System server returns an error.

Page 15 of 20
 BMC Remedy Action Request System 8.1

8 Configuring FTS for localization

The <ARSystemInstallDir>\ftsconfiguration directory contains the files described in the following table to
enable you to configure FTS for localization.

Configuration files for FTS

Configuration Description
file

stopwords_ Specifies stop words used for eliminating common words. Each stop word is a separate
<locale>.txt line item in the text file. You create a file for each locale or language. You can update this
file through the Ignore Words List field on the AR System Administration: Server
Information form.

rootwords_ Lists words and their corresponding root word. You create a file for each locale or
<locale>.txt language. By default, FTS uses stemmers particular to the installed locale. A stemmer
takes words (such as fast, faster, fastest) and converts them to stem words at indexing
time so that using a word, such as fast, finds all references to it, such as faster and fastest
.

The rootwords_<locale>.txt file overrides how the FTS or third-party stemmers define
root words. If a word is found in the root words list, then the root word is used, and the
stemmer is not run on the original word. (For information about using a third-party
stemmer, see Advanced FTS configuration files.)

Each line in the rootwords_<locale>.txt file contains space-separated words with the
first word being the root word and the others being words that map to the root word, for
example:

run running runs

thesaurus_ Contains synonyms used to perform thesaurus expansion during indexing. You create a
<locale>.txt file for each locale or language. If the thesaurus.txt file is present, any terms it finds in
the thesaurus are expanded within the index to contain its synonym values at the same
word location.

Each line in the text file contains space-separated words that are synonyms. For example:

quick fast speedy

Page 16 of 20
 BMC Remedy Action Request System 8.1

Note

If you modify any of the FTS configuration files, you must restart the server for the changes to take
effect.

The FTSLocaleConfig.xml file (in <ARSystemInstallDir>\ftsconfiguration) contains pointers to the

configuration files. For example:

<config>
<locale locale="en">
<stopwordfile>enStopword.txt</stopwordfile>
<rootwordfile>enRootword.txt</rootwordfile>
<thesaurusfile>enThesaurus.txt</thesaurusfile>
<indexAnalyzer> </indexAnalyzer>
<searchAnalyzer> </searchAnalyzer>
<stemmer>English</stemmer>
</locale>
<locale locale="de">
.
.
.

You can include as many "locale" elements as you need in the FTSLocaleConfig.xml file. By default, AR
System is installed with two-letter locales defined, but you can also include country codes (for example,
<locale locale="en_US"> ).

If any element is blank or missing in any locale section of the FTSLocaleConfig.xml file, FTS does not use
that item in the analysis process.

For advanced configuration, you can enter the name of an index analyzer, search analyzer, or stemmer file
in the FTSLocaleConfig.xml file. For more information, see Advanced FTS configuration files.

Page 17 of 20
 BMC Remedy Action Request System 8.1

9 Advanced FTS configuration files

The following table lists the advanced files referenced in the FTSLocaleConfig.xml file.

Advanced configuration files

File Description

indexAnalyzer Enables you to define external Lucene analyzers for the indexing process. For more
information, see http://lucene.apache.org/java/docs/index.html.

searchAnalyzer Enables you to define external Lucene analyzers for the searching process. For more
information, see http://lucene.apache.org/java/docs/index.html.

stemmer The FTSAnalyzer uses the Snowball stemmers from the Snowball project for performing
stemming functionality. This configuration enables you to define which stemmer to use
for a particular language, or it enables you to define a custom stemmer with the
Snowball project tools. For information about the Snowball project, see
http://snowball.tartarus.org/.

If the default FTS functionality is not producing the results you expect, you can reference third-party index
analyzers, search analyzers, and stemmers.

You might want to process the data differently when indexing versus searching.

An index analyzer expands all words in the database. For example, if a user is searching for a word
like computer, other words like system and machine are included in the search.
A search analyzer does not expand the words being searched, which improves the performance. If a
user is searching for computer, only that word is searched for.

9.1 To use third-party configuration files

1. Configure a third-party configuration jar file (for example, customAnalyzer.jar ).
This jar file can contain one or more analyzers (such as indexAnalyzer, searchAnalyzer, and
stemmer). Each analyzer should have a specific name (for example,
org.myorg.lucene.analysis.EsparantoAnalyzer).
2. Insert the analyzer names in the FTSLocaleConfig.xml file. For example:

<indexAnalyzer>org.myorg.lucene.analysis.EsparantoAnalyzer</indexAnalyzer>
<searchAnalyzer>org.myorg.lucene.analysis.EsparantoAnalyzer</searchAnalyzer>
<stemmer>Esparanto</stemmer>

Page 18 of 20
 BMC Remedy Action Request System 8.1

3. Make sure that the Java can find the jar file that you created in step 1:
1. Place the jar file in the fts plug-in directory (by default, C:\Program Files\BMC
Software\ARSystem\pluginsvr\fts ).
2. To add the jar to the class path, edit the pathelement option of the pluginsvr_config.xml
file in the fts directory. For example:

<pluginsvr_config>
<port>9998</port>
.
.
.
<plugins>
plugin>
<name>ARSYS.ARF.FTS</name>
<pathelement type="location">C:/Program Files/BMC
Software/ARSystem/pluginsvr/fts/ftsplugin _VerNum_.jar</pathelement>
<pathelement type="location">C:/Program Files/BMC
Software/ARSystem/pluginsvr/fts/tika\-0.3\-standalone.jar</pathelement>
<pathelement type="location">C:/Program Files/BMC
Software/ARSystem/pluginsvr/fts/ *customAnalyzer.jar*
</pathelement>
.
.
.
</plugin>
</plugins>
</pluginsvr_config>

Page 19 of 20
 BMC Remedy Action Request System 8.1

10 Adding FTS licenses

FTS is integrated throughout the BMC Remedy IT Service Management Suite. You must define a license
before installing FTS.

10.1 To add an FTS license

1. From the IT Home page, open the AR System Administration Console.
2. Choose System > General > Add or Remove Licenses.
3. Click Add New.
4. Define the following settings:
License Type BMC Remedy Full Text Search
Number of Licenses 1
5. Click Save.
6. Restart the BMC Remedy AR System server.

Page 20 of 20