Ask The Expert Full Text Search

Pega Product Support: Ask the Expert - Full
Text Search
Posted by Marissa Rogers Apr 1, 2016
Meet Rajiv
Rajiv is a Senior Manager in engineering handling teams in the Data Engine area of the Pega platform.
Since joining Pega in mid-2009, he has worked with teams in the areas of full text search, core engine, and
SQL generation, BIX, AES, PDC, Performance and Data Layer. He is passionate about full text search and
manages the team which works on enhancing and maintaining the full text search feature of product which
includes the legacy Apache Lucene and the newer Elastic Search.
Message from Rajiv

I’m pleased to join the Product Support Community “Ask the Expert” event and welcome all of your full text
search related questions. Over the course of the month I will try and answer questions related to full text search
and indexing including configuration (landing page), failover, replication, full / incremental indexing and search
which includes both Apache Lucene (pre 7.1.7) and Elastic Search (7.1.7 and beyond). For questions that fall
outside my area of expertise, I may direct you to other resources.
Ask the Expert Rules
Generated by Pega Mesh on 2016-07-01-05:00

1
Pega Product Support: Ask the Expert - Full Text Search
• Follow the Product Support Community's Community Rules of Engagement

• This is not a Live Chat - Rajiv will reply to your questions over the course of this month long event
• Questions should be clearly and succinctly expressed
• Questions should be of interest to many others in the audience
• Have fun!
Do You Have a Question for Rajiv? Leave a Comment below!

1668 Views Tags: performance, database, aes, bix, lucene, data_layer, pdc, elastic search, ask the expert,
full_text_search
Brendan Horan in response to Rajiv Nistala on page 2

May 2, 2016 10:16 AM
Sincere thanks for all of the insight provided throughout this wonderful session Rajiv Nistala. And thanks to
all our participants for your great questions. It wouldn't be much of an Ask the Expert session if no one asked
anything
Stay tuned for our next Ask the Expert session on the Pega 7 UI. We will be providing more details once we
move to our new home on the PDN.
Thanks.
B.
Rajiv Nistala
May 1, 2016 4:34 AM
Thank you Marissa Rogers for setting up this Ask the Expert on full text search. And thanks to each and every
one who participated in this giving me and Pega, feedback and ideas on improving the product further. We will
continuously improve your experience with our products.
As this event comes to a close, please don't hesitate to contact the Pega Support Community or myself with
your full text search related questions and feedback.
Ravi Kumar Pisupati in response to Rajiv Nistala on page 2

Apr 28, 2016 12:11 PM
Sure Rajiv. Thank you!
Rajiv Nistala in response to Ravi Kumar Pisupati on page 3

Apr 28, 2016 1:54 AM
Hi Ravi,

2
I suggest you read the upgrade guide for 7.1.9 (page 103) to understand what it takes to move from Lucene
(which is what we were using in 6.2) to Elastic Search in 7.1.7 or beyond - https://pdn.pega.com/documents/
pega-719-platform-upgrade-guide
-Rajiv
Rajiv Nistala in response to Markus Marbach on page 4

Apr 28, 2016 1:50 AM
Issue: I have noticed inconsistent search behaviour where updates of a work object status
or even creations on one node are not reflected in search results on the other nodes. I see
discrepancies of the number of documents indexed between the two index hosting nodes.
Shouldn't they be the same and in sync? How can I get them re-sync'd?
Elastic Search should take care of replication automatically. How are you checking that the count of documents
in each node is different?
You may want to check that all nodes can communicate with each other over port range 9300~9399
Question: How can I re-sync that index from one of the others without a full Re-Index and
which one would it be sync'd from?
The check button should show the same message irrespective of whichever node you click it from for all nodes
in a single cluster. If it is showing corrupted, a full re-index is required. Please use the re-index button on the
search landing page.
Issue: It appears that this call is also made when doing a full re-index presumably because
the re-indexing process has to open the work object for indexing. Do I have to build some
logic in the OpenDefaults to exclude the external call when opened by the re-indexer? How
can I tell that this is part of a re-indexing task?
That is strange. We do an open of the work object from within the engine and so I wouldn't expect
OpenDefaults to be called. That said, how did you figure out that re-indexing is indeed calling it?
Question: Is there a way to suppress the firing of expressions when using
pxRetrieveSearchData?
By expressions you mean declare expressions?

You can use the pzDisableDeclarativesOnCurrentThread function to disable or enable declarative. Use it with
caution.
Ravi Kumar Pisupati

Apr 27, 2016 5:40 PM
Hi Rajiv,
We are running our current application using V6.2 and we are in upgrade process now. We are going to
upgrade the application into V7.1.9 and the corresponding CPMHC also we are upgrading it. When we go into

3
V7.1.9, what are all the precautionary steps to be taken care from Elastic search mechanism side? Moreover,
our application is a complex and has huge data in the work table (TB's).
Thanks,
Ravi Kumar.
Markus Marbach
Apr 26, 2016 8:03 PM
Hi Rajiv,
Thanks for all your insights of everything search related.
I have a few questions/issues I'm battling with at one of my sites. We are running Pega 7.1.7.
1. We have three nodes in the Pega cluster. The 01 & 02 nodes contain Search Index Files. The node 03 does
not contain a search index file.
Issue: I have noticed inconsistent search behaviour where updates of a work object status or even
creations on one node are not reflected in search results on the other nodes. I see discrepancies of the number
of documents indexed between the two index hosting nodes. Shouldn't they be the same and in sync? How
can I get them re-sync'd?
2. The third non-index hosting node shows a corrupt index when clicking on the Check button.
Question: How can I re-sync that index from one of the others without a full Re-Index and which one would it
be sync'd from?
3. We have some logic in the Work-.OpenDefaults activity to call an external system refreshing some data
upon opening a work object.
Issue: It appears that this call is also made when doing a full re-index presumably because the re-indexing
process has to open the work object for indexing. Do I have to build some logic in the OpenDefaults to exclude
the external call when opened by the re-indexer? How can I tell that this is part of a re-indexing task?
4. We are using pxRetrieveSearchData for fetching work object data from the search index rather than the
work tables. This works great except that expression get triggered when receiving the result set. We're re-using
some of the same classes and pages for the result page as for the normal cases where we have expressions
and onchange rules defined.
Question: Is there a way to suppress the firing of expressions when using pxRetrieveSearchData?
thanks in advance for your feedback.

4
Rajiv Nistala in response to Scott Holz on page 5

Apr 25, 2016 10:15 AM
Hi Scott,
This question doesn't seem related to "Full Text Search". I suggest you raise a separate discussion in Pega
Product Support community and I am sure the right folks will engage and reply.
-Rajiv
Scott Holz
Apr 25, 2016 8:58 AM
One of my partners is demoing Pega Knowledge Management. I provided them with version 7.21, which is the
only version available for download. They have a 7.1x version of Pega platform installed (7.17 I think). Will KM
7.21 run correctly or do they need to upgrade to 7.2?
Scott Holz
Rajiv Nistala in response to Halim Yahia on page 5

Apr 24, 2016 12:38 PM
When we were using only Apache Lucene for full text search (till 7.1.6 version of the platform), the search
indices could only be maintained by one node and all other nodes needed to connect to it to get the search
results. With the introduction of Elastic Search, the indexes can be distributed to multiple nodes to provide
failover and high availability. Thus the dynamic settings have "distributed" word in them. It is up to the customer
to leverage this feature by configuring multiple nodes in the same cluster to host the index nodes or not via the
search landing page.
Halim Yahia in response to Rajiv Nistala on page 5

Apr 22, 2016 8:34 AM
Thank you, that helps. Now it works.
It was indexing/distributed/index_enabled that was false.
One question more : we use a single node. So, normally we do not use distibuted index. Why then we "should"
put distributed settings to true ? Thank for your clarification.
Rajiv Nistala in response to Halim Yahia on page 6

Apr 22, 2016 6:57 AM
Hi Halim,

5
There could be a number of reasons for this. You should try these
1) Go to the search landing page (Designer Studio -> System -> Settings -> Search) and click on the check
button next to each of the 3 indexes. If any of them has a message in red, re-index it.
2) Check the following instances of Data-Admin-System-Settings - This step should be done only after doing
(1) above doesn't fix the issue.
indexing/distributed/enabled
indexing/distributed/index_enabled
indexing/distributed/search_enabled
All the above should be set to true.
3) On the search landing page, in the index host node section, check if the node id(s) listed there are valid
node(s). If not, please correct them and click submit.
4) Please check if this issue is occurring in your instance - https://pdn.pega.com/support-articles/re-index-
button-grayed-out-search-landing-page
Also, do the check the logs for any errors that are happening. You can usually see further details by enabling
"debug" logging on "com.pega.pegarules.search.internal.es" package.
-Rajiv
Halim Yahia
Apr 22, 2016 5:57 AM
Hi Rajiv,
we are facing an issue after upgrade from PRPC 6.2 to Pega 7.1.8. The problem we have is after index end,
we can do search only on some of work objects. The system can find some of them, for example D-20130, but
not D-20000.
We have a lot of objects to index: arround 360000.
We tried to index several times, without success.

Filesystem is OK.
Have you any idea about this issue ?
Thank you.
Rajiv Nistala in response to Leonid Yakovlev on page 7

Apr 21, 2016 7:52 AM
Sorry, my bad. Your initial value was correct. I looked at a different piece of code and got confused.

6
I suggest you raise an SR for this issue.
Leonid Yakovlev in response to Rajiv Nistala on page 7

Apr 21, 2016 7:30 AM
Hi Rajiv, now it doesn't index it at all. I see in log:
WARN - Failed to index instance with key [ClassName P-4528 0 000000]

com.pega.pegarules.pub.database.BadInputException: This external handle is not valid: ClassName P-4528 0
000000
From: (BE4AB6CF108565211D15BCC38B34BF951)
at
com.pega.pegarules.data.internal.access.InstanceName.getKeysFromExternalHandle(InstanceName.java:526)
at com.pega.pegarules.data.internal.access.Opener.openData(Opener.java:505)

Apr 21, 2016 6:22 AM
The pxWorkKey should be
ClassName key1 key2
Instead of
ClassName key1!key2
Please try with this change (space instead of exclamation) and see if that helps.

Apr 21, 2016 6:02 AM
Hi Rajiv,
1) Yes, after full re-indexing pxInsHandle has correct value in results, multiple results displayed correctly (they
have different pxInsHandle) - everything working correctly for instance created before full re-indexing
2) In pr_sys_queue_ftsindexer table we create instances manually with Declare Trigger and there - pxWorkKey
is set to correct value. There is no pxInsHandle value there. Blob of queue record also, doesn't contain
"ClassName null". It contacins "ClassName key1!key2" - which we set with Declare Trigger.

Apr 21, 2016 5:36 AM
Hi Leonid,

7
HFIX-25630 is supposed to address the issue with external classes (PRPC classes mapped to tables which
don't contain pzInsKey and pxObjClass). In this scenario, instead of getting all results that qualify, we were
getting only 1 result in search.
Questions:
1) If the instance is not indexed via incremental indexing agent (FTSIncrementalIndexer) but by using the re-
index button on the search landing page, are you able to search for the results?
2) Are you noticing "ClassName null" in the pr_sys_queue_ftsindexer table (queue table used by
FTSIncrementalndexer agent)? Or in the search results?
-Rajiv
Leonid Yakovlev
Apr 21, 2016 4:16 AM
Hi Rajiv, I observed the problem (Pega 7.1.8 with HFIX-25630) that instances indexed with
FTSIncrementalIndexer will be returned in the result list with pxInsHandle="ClassName null" (at the same
time, pzInsKey of the instance is correct and doesn't have null in it). As a result - instead of 2 added into
index entries result returns only one (both matches search criteria) - the both have same pxInsHandle and,
therefore duplicates removed from result and displayed only the first record from result. Should the hotfix,
which we installed address this problem? I think it stopped working after committing this hotfix, but I'm not sure
if it worked before we committed it (however, we made a restart afterwards - it didn't help). Thanks.
Elastic Search log shows the following records:

Running count (unique): 6, Hit count: 7
Results count: 6

Apr 18, 2016 5:07 AM
At the moment we are thinking about simulation of "Re-Index" button click from Search
Settings LP. We are planning to call activity pzLPFTSManagerRunBuildIndex by Agent and
pass all necessary parameters (including index type and class for re-indexing). Is this the
best way to do this? Do you have any better ideas to achieve the same?
Yes, running this activity via an agent could be one of the options.
Alternatively, from 7.1.8 and beyond you can use the command line utility called FullTextIndexer. This will spin
up a new instance and doesn't overload the existing cluster nodes. You can find the details in this PDN article
https://pdn.pega.com/batch-creation-and-updating-elasticsearch-index-files-command-line/batch-creation-and-
updating
Leonid Yakovlev

8
Apr 17, 2016 3:32 AM

Hi Rajiv,
What is the best way to refresh search index for single Data class completely? We are planning to make
this operation periodically (several times per year). We expect this to happen when one of data-sets used in
database view is completely refreshed (this action will be triggered outside).
On our current data-set rebuilding of entire search index (for single class) from Search Settings LP takes
around 10-15 minutes, so we expect reasonable downtime. Refresh of external data-set and rebuilding index
will happen off working hours (e.g. over night), of course.
At the moment we are thinking about simulation of "Re-Index" button click from Search Settings LP. We
are planning to call activity pzLPFTSManagerRunBuildIndex by Agent and pass all necessary parameters
(including index type and class for re-indexing). Is this the best way to do this? Do you have any better ideas to
achieve the same?
Rajiv Nistala in response to Aakarsha K on page 9

Apr 15, 2016 1:00 AM
Hi Aakarsha,
This doesn't seem related to Full Text Search. Also, I noticed that there is already a discussion created for this
issue - https://mesh.pega.com/thread/59014. You should expect responses from community members there.
-Rajiv
Aakarsha K
Apr 14, 2016 11:04 PM
Hi
I am unable to install Pega 7 personal edition & getting following error in log file .Please help
Targets:Initialization,Database Server Installation,Data Load,Configuration,PRPC Launching

Buildfile: F:\115008_PE72\.\pegaGuiInstallerTemp\build.xml
Initialization:
[echo] Beginning installation of PRPC 7.2 Personal Edition -- this process should take about 5-10 minutes ...
[mkdir] Created dir: F:\PRPCPersonalEdition
[mkdir] Created dir: F:\PRPCPersonalEdition\temp
[unzip] Expanding: F:\115008_PE72\pegaGuiInstallerTemp\PersonalEdition.zip into F:\PRPCPersonalEdition
[echo] Configuring Tomcat (Setting Java Home and Catalina Home)...
[replace] Replaced 0 occurrences in 0 files.

9
Database Server Installation:

[exec] The files belonging to this database system will be owned by user "KARTHEEK".
[exec] This user must also own the server process.
[exec] The database cluster will be initialized with locale "English_United States.1252".
[exec] The default database encoding has accordingly been set to "WIN1252".
[exec] The default text search configuration will be set to "english".
[exec] Data page checksums are disabled.
[exec] creating directory F:/PRPCPersonalEdition/pgsql/data ... ok

[exec] creating subdirectories ... ok
[exec] selecting default max_connections ... 100
[exec] selecting default shared_buffers ... 128MB
[exec] selecting dynamic shared memory implementation ... windows
[exec] creating configuration files ... ok
[exec] creating template1 database in F:/PRPCPersonalEdition/pgsql/data/base/1 ... ok
[exec] initializing pg_authid ... ok
[exec] child process was terminated by exception 0xC000001D
[exec] initdb: removing data directory "F:/PRPCPersonalEdition/pgsql/data"
[exec] initializing dependencies ...
[exec] Result: 1
[echo] Waiting for Postgres to start...

Unable to connect to database -- will retry in 1 second...
BUILD FAILED
F:\115008_PE72\pegaGuiInstallerTemp\build.xml:156: Unable to connect to database within 15 seconds.

10
Total time: 45 seconds
Rajiv Nistala in response to David Kircheis on page 11

Apr 13, 2016 7:03 AM
Hi David,
Currently there is no OOTB mechanism to extend Pega search to index / search external content outside of the
tables mapped by the classes present in the Pega platform. This means that we can index and search content
stored in external tables mapped to PRPC classes but this does rule out ECM repositories. While the platform
does allow for storage of attachments in ECM repositories, the current index / search code can only refer to
attachments stored in the Pega database.
I suggest you raise an enhancement request to get this on the radar of product management, based on the
urgency of your need.
-Rajiv
Rajiv Nistala in response to Alexey Muskancev on page 11

Apr 13, 2016 6:52 AM
Hi Alexey,
This question doesn't look related to full text search.

Is the test information limited to only a single table specific to your application? Then truncate should be fine.
If it spans multiple tables which involves tables where the platform stores instances as well, then you must be
careful to delete only those instances that were created for your test rather than truncate.
-Rajiv
David Kircheis
Apr 12, 2016 12:49 PM
Hi,
Is it possible to extend Pega search to index/search external content in connected databases or ECM
repositories (via CMIS)?
Thanks in advance for your thoughts on this!
Alexey Muskancev

11
Apr 12, 2016 7:26 AM

Hello Rajiv,
I want to insert about 60 Gb newest test information to database (like client_id,name, address, etc).
Is it possible to clean old database data (like truncate table), except system tables that need for pega working?
(so after truncate pega must works!)
I need to delete only all information needs for test team, and after deleting will insert new testing data . (so
after truncate pega must works!)
Is it real? Where i can find description about this?
Thanks.
using pega 6.1 / oracle 11g
Rajiv Nistala in response to Ionut Hamin on page 12

Apr 11, 2016 1:54 AM
So it looks like you are trying to do partial rule resolution. To understand better, what is the source of your page
lists? If it is coming from the database, the report definitions from Pega 7 onwards have the ability to do rule
resolution on the rules retrieved (without circumstancing). That might solve the problem directly.
Also, since this is not full text search related, I suggest you move these comments to a separate discussion on
Pega Product Support space so that experts in obj-filter and obj-sort can help you.
Ionut Hamin in response to Rajiv Nistala on page 12

Apr 11, 2016 12:48 AM
Thank you, you are right , it is not about to full text search but I am sure that you can help me with your
expertize or suggest me something .
Yes, the page list contains rule instances and I want to keep only the last created version(s) in my list, so
remove all the others. Let's say that I am having an instance created in 3 versions 01.01.23, 01.01.25,
(a)-01.01.43, and another with two, 01.01.23,(a)-01.01.25, I want to keep only the two instances (a) whithout
inserting in the obj filter the instance version number I want to keep (avoid in the future the risk that if someone
else is creating another major version the obj filter will not work anymore correctly).There is something in the
obj filter I can use or I need to have another approach?
Thank you in advance for you kindly kelp!
Ionut
Rajiv Nistala in response to Ionut Hamin on page 13

Apr 10, 2016 12:12 PM
Hi Ionut,

12
This doesn't seem related to full text search. That said, could please let me know what you mean by only
results from major versions. Does your page list contain rule instances and you want to remove rules which are
not of a specific major version (or versions)?
-Rajiv
Ionut Hamin
Apr 8, 2016 11:14 PM
Hi Rajiv,
I want to filter from a pagelist only the results from the major versions. What should be the best approach?
Thank you!
Rajiv Nistala in response to Chiranjeet Mishra on page 13

Apr 7, 2016 10:41 AM
Looks like you raised a specific discussion for this - https://mesh.pega.com/thread/58663. I'll let people answer
it there than here as this doesn't seem related to full text search. Please do correct me if my understanding is
wrong.
Chiranjeet Mishra
Apr 7, 2016 10:11 AM
Hi Rajiv,
Can we use Obj-Sort method on a pagelist which we have declared ourselves in Pages and Classes but not
created a PageList type property? If not, how can we sort the results in such case?

Apr 7, 2016 9:50 AM
If you delete the instance from DB using the standard obj-delete or obj-delete-by-handle mechanism, the
platform will take care of deleting it from the index.
There is currently no exposed API to delete a single instance.

1) If the entry does not exist in the database, you might want re-index from the landing page, all entries of that
class to which this instance belongs.
2) You can however create an entry in the pr_sys_queue_ftsindexer table (System-
Queue-FTSIncrementalIndexer class) (refer to a previous comment in this thread) with pxIsDelete property set
to true. Note that in this case, if that instance is updated again via the platform obj-save methods, the entry will
be added back to the index.
Leonid Yakovlev

13
Apr 7, 2016 9:40 AM

Hi Rajiv, how to remove an entry from search index (by providing key)?
Rajiv Nistala in response to Richard Jackson2 on page 14

Apr 7, 2016 9:27 AM
Full Index is a costly operation both in terms of time and CPU / Memory resources. We thus automatically
trigger this on a fresh install but don't do this on upgrades. Fresh install does not have any work but just the
OOTB rule and data instances. In the Pega 7 platform, this is triggered by a start up agent which calls the
activity pzInstallation which calls pzSearchInstall activity.
Note that we trigger this only for Rule and Data. Work has to be manually enabled.
If you do want to make this run automatically for every system startup, you could write your own start up agent
which the necessary pieces of the pzSearchInstall activity as necessary.
I would strictly advise you NOT to do this. With 7.1.7, with the introduction of Elastic Search in the platform,
multiple nodes can be used as search nodes and provide fail over if one of the nodes goes down. Thus, if
for every restart we do indexing, the overhead is multiplied by the number of search nodes configured on the
search landing page because of the replication that needs to happen.
Rajiv Nistala in response to Nipun Khokhar on page 14

Apr 7, 2016 9:20 AM
If you see "Limited Results" in the search screen, then the search settings are incorrect on the search landing
page because of which instead of using the full text feature (Apache Lucene). If you use tracer to check what
happens when you search, you should find a failure.
Please take a look at this help document to understand how to configure the search landing page correctly
-https://pdn.pega.com/sites/pdn.pega.com/files/help_v71/procomhelpmain.htm#basics/v6portal/landingpages/
system/systemsear…
Note that sometimes the wrong Host Node ID is set up in the landing page and at other times the Search Soap
URL points to a different node than what is identified by Host Node ID. Usually correcting one or both of these
fixes the issue.
Richard Jackson2
Apr 6, 2016 3:23 PM
Hi Rajiv,
What triggers Pega to automatically build the index after installation or update? Is there a way to reset this, so
that a full index happens automatically on the next system startup?
Thanks.

14
Nipun Khokhar
Apr 6, 2016 2:56 PM
Hi Rajiv -
We are using PRPC 6.3 SP2. We are having issues using the 'Search'. It gives the below result while
searching rules.
Search Results [ Limited Results ]

Not able to do a content search on rules. I have also tried reindexing rules via the SMA.
Jagannadham Parupudi in response to Rajeesh Shenoy on page 15

Apr 6, 2016 8:19 AM
Hi Rajeesh,
Currently there is no OOTB support for touch tones on each dailpad entry. Its the string of digits in "phone #"
textbox which will be sent when you press on Send DTMF button. If customer requirement is to send tone on
each digit press, they need to customize the existing functionality by detecting key pressed and sending it to
sendDTMF function.
-Jagan
Rajiv Nistala in response to Rajeesh Shenoy on page 15

Apr 6, 2016 2:02 AM
Hi Rajesh,
This is not my area of expertise. I suggest you raise a separate Pega Product Support discussion thread so
that the right people could respond on this.
-Rajiv
Rajeesh Shenoy
Apr 5, 2016 5:52 PM
Hi Rajiv
Does Pega provide DTMF feature in Pega Call Toolbar. My Use Case is to use touch tones from dial pad.
While pega have Send DTMF ,its not sending any touch tones on each dialpad entry.

15
Rajiv Nistala in response to Elliott Davis on page 16

Apr 5, 2016 5:43 AM
Hi Elliott,
The FindWork is an older section which uses the FindWorkGadget which is non-auto-generated. I believe the
older ones (not Responsive) used iFrames.
pyFindWorkGadget is a newer auto-generated search gadget that you can use in your end user portals. It runs
shows search results in an auto-generated overlay: pyWorkSearch.
-Rajiv
Elliott Davis in response to Rajiv Nistala on page 25

Apr 4, 2016 4:47 PM
Thank you for the clear and definite answers, Rajiv.

Apr 4, 2016 4:46 PM
In the absence of detailed documentation, you have been an excellent resource. Thank you!
Elliott Davis
Apr 4, 2016 4:45 PM
Hi Rajiv.
Thank you for answering my many questions already. I also appreciate your patience about the fact that my
questions are rather similar. And I'm afraid I have another variation on the same theme:
What is the difference, if any, between search behavior when using these two in a portal:
- FindWork with Responsive mode
- pyFindWorkGadget
Cheers,
Elliott

Apr 4, 2016 4:42 PM
Thank you, Rajiv. I will certainly use Responsive mode.

16
Rajiv Nistala in response to Rajani Kanth Anupoju on page 17

Apr 4, 2016 11:20 AM
Hi Rajani,
I sincerely apologize if my answers have offended you. Since I didn't understand your use case clearly, I
wanted to know more details.
If you are using "Responsive" FindWork, you can customize your filters. Please refer to this mesh discussion -
https://mesh.pega.com/message/233107#233107
-Rajiv
Rajani Kanth Anupoju in response to Rajiv Nistala on page 22

Apr 4, 2016 11:12 AM
Thank Rajiv for your EXCELLENT answer.
Probably you didn't read my question correctly with all your senses. I said SIMILAR to BIX and NOT same.
It's better off staying away from you for any further questions.
Rett Hean in response to Rajiv Nistala on page 17

Apr 4, 2016 11:12 AM
Thank you Rajiv, that was helpful.
Rajiv Nistala in response to prasanth kumar on page 18

Apr 4, 2016 11:12 AM
Unfortunately, I have no expertise on gmail connectivity. I already see a lot of discussion on that thread so I am
sure folks will definitely help you out on that front.
Marissa Rogers in response to prasanth kumar on page 18

Apr 4, 2016 11:09 AM
Hi prasanth kumar - Your question is not related to the topic in this Ask the Expert session. Please continue to
discuss your question in your thread.
Rajiv Nistala in response to Rett Hean on page 18

Apr 4, 2016 11:09 AM
What is the suggested method to perform content searches for multiple words as an exact
phrase? Is the same as google using quotes to surround the phrase?

17
Yes
Strangely when I searched in my local development environment with click here to close this window without
quotes, I got a lot more results than when when I put quotes around it. I randomly clicked on some of the rules
(when searching with quotes), opened the rule XML for each of them and then searched for the exact text and
found it in each one of them.
I think the root issue is which rule comes back as the first result. Now this is typically left to how Lucene (or
Elastic Search which uses Lucene underneath) ranks the results. Note that with "Contains" search, the rank is
same so it is some random order. If we don't use wildcard searches, then the ranking is usually based on the
frequency of terms. Please refer to the Apache Lucene website for details on how the default ranking works.
And does the suggested method change when using Lucene vs. Elastic?
No
prasanth kumar
Apr 4, 2016 10:51 AM
i ahve some problem with gmail connectivity
i have posted on mesh plz solve this
gmail connectivity
Rett Hean
Apr 4, 2016 9:29 AM
What is the suggested method to perform content searches for multiple words as an exact phrase? Is the same
as google using quotes to surround the phrase?
In my experiences, I either get no results, or completely irrelevant results. It's almost as if it is searching for the
actual quote sometimes, it's hard to tell.
Here is an example, where it acutally kind of works, but ultimately did not give me what I was looking for.
I know there is code somewhere in Pega which is behind the UI displaying, "Click here to close this window",
but I want to find the source rule.
Using no quotes, I get some hits, but none of them are from the rule I have actually opened (in this case it is
DenyExternalState @baseclass)

18
Using quotes, I got more hits, but again, none of them are from the rule I have actually opened.

19
Thoughts? And does the suggested method change when using Lucene vs. Elastic?

Apr 4, 2016 8:04 AM

20
Hi Leonid,
There is no document for this. The values for the properties will depend on the option you choose. If you go via
Rule-Declare-Trigger, then Pega platform would set properties for some of them. You will need to set only the
following
pxObjClass (should be already present if you create a clipboard page of that class)
pyAgentName = Pega-SearchEngine:FTSIncrementalIndexer
pyItemStatus = Scheduled
pyMaxAttempts = 1
pyMinimumDateTimeForProcessing = <current timestamp>
pxWorkKey = <pzInsKey of the instance that changed>
pxWorkClass = <class name of the external data class>
pxIsDelete = <true if the entry should be deleted from index, false otherwise>
pyAccessGroup = <provide the access group context for this>
pyRetainOnFailure = true
pyRetainOnSuccess = false
pyUseApplications = true
pySelectId = <current date time> + Uppercase(Node ID)
pyItemId = Uppercase(pxObjClass) + " " + pySelectId
-Rajiv

Apr 4, 2016 5:55 AM
Rajiv, what is a minimum valid set of properties for System-Queue-FTSIncrementalIndexer instance? Is there
any document for this?

Apr 4, 2016 5:26 AM
Hi Rajiv, thank you again. You correctly understood scenario and both solutions seems absolutely valid for me.
Thanks a lot!

Apr 4, 2016 5:23 AM
Hi Leonid,
So if I understand correctly, when the work object is saved, it also creates an entry in a different table (maybe
you can elaborate how this happens in your database) and a view on top of this and other tables forms the
external table for you. I am assuming you are having a Pega class which maps to this view and thus you are
able to index the entries when you do a full re-index via the search landing page.

21
There are two ways to handle this
1) Write a rule-declare-trigger which an add entries to the pr_sys_queue_indexer table (via Pega class System-
Queue-FTSIncrementalIndexer) for that view class
2) Write a database trigger on the table which holds the actual data class to create an entry in
pr_sys_queue_indexer table (via the Pega class System-Queue-FTSIncrementalIndexer) for that view class
This will allow for incremental indexing to happen when agent runs by picking up the changed instances from
queue table. From 7.1.8 onwards, this pr_sys_queue_ftsindexer table does not need a BLOB so even option 2
above is viable.

Apr 4, 2016 5:09 AM
Hi Rajiv, thank you for the explanation, its very helpful. Yes, the new entries in the DB view appear when we
create a workobject in Pega (view combines data from a static external source with data coming from Pega).
However, inside Pega there is no visible link between external class on DB view and class of workobjects,
instances of which we create. So, when we create a workobject it becomes available in the view. A user is
given a search functionality running against external class only (not the class of workobjects).
Rajiv Nistala написал(а):
Note that for this work

a) The associated index needs to be enabled on the search landing page
b) The class definition does not have "Exclude from full text search" checked
These criteria are satisfied for the external class.
How can we inform incremental index to consider new entries created in external class (as a result of created
workobjects)? Will Incremental index automatically check all indexed classes when a new workobject is
committed to a database? Do we need a link between workobject class and this external Data- class? Does
this require restart after initial configuration? Or do we need to call Indexer API manually whenever we create/
update a workobject?
Rajiv Nistala in response to Rajani Kanth Anupoju on page 23

Apr 4, 2016 5:03 AM
1) Is it possible to override or extend the FTSIncrementalIndexer ?
I wanted to use entries in " pr_sys_queue_ftsindexer" to update my work objects information

to my other database(similar to what BIX does). Is it possible?
Dumb question from my side - why can't use you use BIX when what you are trying to do is similar to BIX?

22
Can I disable FTSIncrementalIndexer agent & build my logic to process entries in "
pr_sys_queue_ftsindexer"?
Kindly elaborate your use case.

2) Within the workobject I don't wanted to index all properties. How to restrict ?
I wanted to do a search based on specific property (Eg: CreatedBy, Comments, etc).
I will redirect here to read the following help topic on PDN - https://pdn.pega.com/sites/pdn.pega.com/files/
help_v72/procomhelpmain.htm#concepts/concepts2/understandingsearch.htm
This clearly outlines which properties are filterable and which are returnable. You can customize further for
embedded properties. Please read the details here - https://pdn.pega.com/sites/pdn.pega.com/files/help_v72/
procomhelpmain.htm#data-/data-customproperties-/data-customproper…
Rajani Kanth Anupoju

Apr 4, 2016 4:26 AM
Hi Rajiv,
I have couple of questions

1) Is it possible to override or extend the FTSIncrementalIndexer ?
I wanted to use entries in " pr_sys_queue_ftsindexer" to update my work objects information to my other
database(similar to what BIX does). Is it possible?
Can I disable FTSIncrementalIndexer agent & build my logic to process entries in " pr_sys_queue_ftsindexer"?
2) Within the workobject I don't wanted to index all properties. How to restrict ?
I wanted to do a search based on specific property (Eg: CreatedBy, Comments, etc).

Apr 4, 2016 3:27 AM
How FTSIncrementalIndexer determines which classes (and which new records) it should
incrementally index?
Whenever a rule, data or work instance gets inserted, updated or deleted in the DB, Pega creates an entry
in the pr_sys_queue_ftsindexer table for the FTSIncrementalIndexer to consume. The agent runs every 60
seconds and drains the queue, figures out what needs to be indexed, opens the BLOB for each of the entries
and updates the appropriate index accordingly.
Is there any place in Pega, where I can see a list of classes considered for incremental
indexing?

23
The following instances of classes are indexed

a) All rule classes
b) All data classes which have a ruleform associated with them in the class definition (exception: Data-Social-
Tag)
c) All work classes
Note that for this work
a) The associated index needs to be enabled on the search landing page
b) The class definition does not have "Exclude from full text search" checked
We are planning to run a search against external class (Data-), which is a database view
underneath (Pega 7.1.8 + SFAi 7.14). Should it work?
Yes, provided the Data- class has a rule form associated with it in the class definition
At the moment, I observe the issue that the index can be built only once (after configuring
RuleForm and clicking re-index button in search settings). Afterwards, if new entries appear
in DB view, incremental index ignores them: I was waiting hours after adding a new record
(checked in DB Client), but the indexer didn't collect it. New records become visible in
search result only after forced re-indexing again. The only instances, which appears in the
table pr_sys_queue_ftsindexer are for Data-Admin-Connect-EmailListener class. I switched
on the full log (ALL) for ElasticSearch class, but there are no exceptions or errors there.
Do I need to make a restart (and cache clear or search index clear) after configuring search
for the first time for new Data- class?
How are you creating entries in the external table? If it is via Pega platform, then the queue table should have
an entry. If the external table is getting entries outside of Pega platform, then there is no way to know about this
change in the platform and thus pr_sys_queue_ftsindexer table will not have any entry. Please clarify how new
entries are appearing in the DB view.
Leonid Yakovlev
Apr 3, 2016 1:57 PM
How FTSIncrementalIndexer determines which classes (and which new records) it should incrementally index?
Is there any place in Pega, where I can see a list of classes considered for incremental indexing?
We are planning to run a search against external class (Data-), which is a database view underneath (Pega
7.1.8 + SFAi 7.14). Should it work?
At the moment, I observe the issue that the index can be built only once (after configuring RuleForm and
clicking re-index button in search settings). Afterwards, if new entries appear in DB view, incremental index
ignores them: I was waiting hours after adding a new record (checked in DB Client), but the indexer didn't
collect it. New records become visible in search result only after forced re-indexing again. The only instances,
which appears in the table pr_sys_queue_ftsindexer are for Data-Admin-Connect-EmailListener class. I
switched on the full log (ALL) for ElasticSearch class, but there are no exceptions or errors there.
Do I need to make a restart (and cache clear or search index clear) after configuring search for the first time for
new Data- class?

24

Apr 2, 2016 8:53 AM
The list view here is only used for presentation and does not comply to a standard list view which fetches
results from a database.
Please read the following help topics to understand search and how to retrieve results from the search index.
You should Responsive for this work.
https://pdn.pega.com/sites/pdn.pega.com/files/help_v72/procomhelpmain.htm#concepts/concepts2/
understandingsearch.htm
https://pdn.pega.com/sites/pdn.pega.com/files/help_v72/procomhelpmain.htm#data-/data-customproperties-/
data-customproper…
https://pdn.pega.com/release-note/method-specifying-custom-search-properties-has-changed

Apr 2, 2016 8:51 AM
So, questions about the initial results:
• If we could describe the search behavior with a SELECT statement, what would be the
ORDER BY logic?
• Can we specify the ORDER BY logic?
• What would be the GROUP BY logic?
• Can we specify the GROUP BY logic?
Unfortunately with both V6Search and Responsive search, we cannot specify either the group by or order by of
the results.
The default sorting is based on the relevancy as returned by Lucene (till 7.1.6) and Elastic Search (from 7.1.7
onwards). In Responsive search, you should be able to sort the columns by clicking on the header. It is not
possible with V6Search.
Questions about Show All Results window:
Disregarding the back-end details, in terms of the behavior I see as an end user, the only
differences between initial results and the Show All Results window are obvious differences:
number of results returned
presentation
Aside from those obvious differences, are there any other differences in WHERE, ORDER
BY or GROUP BY logic?
Can I specify any of those?
Same apply for group by and order by for all results. Note that sorting should work when using Responsive by
clicking the header.

25
Apr 2, 2016 8:43 AM

Hi Elliott,
Thanks for posting this. Unfortunately, the documentation is not detailed and I will pass the feedback regarding
the same.
You may want to look at this PDN link - https://pdn.pega.com/how-add-full-text-search-work-objects-and-text-

attachments-composite-user-portal for more details about each of the options other than Responsive. Note that
this is also not elaborate enough.
In general to understand search, this is useful - https://pdn.pega.com/sites/pdn.pega.com/files/help_v72/

procomhelpmain.htm#concepts/concepts2/understandingsearch.htm
Specifying what can be saved into the search index is specified here - https://pdn.pega.com/sites/
pdn.pega.com/files/help_v72/procomhelpmain.htm#data-/data-customproperties-/data-customproper…
And the changes in 7.2 in specifying what can seached / indexed is here - https://pdn.pega.com/release-note/
method-specifying-custom-search-properties-has-changed
Responsive and V6Search are very similar in functionality. Advanced is what was the default for version and 5
till sometime in version 6. Later we switched the default to V6Search.
For customers and applications who wanted to customize search, V6Search was not that flexible even though
the default behavior was the same. Responsive was developed to fill that gap.
Responsive uses the pxRetrieveSearchData activity which leverages a report definition under the hood. The
report definition captures the metadata on what filters need to be applied to the search query and which fields
to return in the results provided they make it to the index via the custom properties definition (please refer
above).
pxRetrieveSearchData activity is evolving and doesn't have all the pieces yet in place in terms of the features
we can provide with full text search. With each release we are trying to add new ones. Like in 7.1.8, fuzzy
search was added. In 7.2, attachment count is now part of the search results - https://pdn.pega.com/release-
note/case-manager-option-searching-only-attachments
-Rajiv
Elliott Davis
Apr 1, 2016 5:11 PM
Can I sort by columns in the initial search results from FindWork?
What would I need to do in order to enable such sorting?
In the screenshot below, my portal is using the OOTB ListView Pega-Search-Work-

Results.pySearchResultsWork.ALL for the presentation of the results

26
When I click a column to sort, the result is the same for each column: the error message above.
For most ListView rules, I would expect that error if I tried to sort by a property mapped to a column in the
BLOB (not exposed). I researched that. The class for this ListView, Pega-Search-Work-Results, seems to map
to PR_OTHER. If this were a ListView in my application, in a class from my application, I would create a table
other than PR_OTHER and expose columns as needed. However, this case seems different. I think you told
me a few weeks ago that for Elasticsearch, some ListView rules are used only for presentation, and I think this
may be one of those cases.
As with some of my other questions, I have tried to solve this with bothering you folks, but I don't seem to see
how to proceed on this one.
Elliott Davis
Apr 1, 2016 4:55 PM
What is the ordering logic behind the results returned by Elasticsearch?
• in the initial results
• in the window popped up when I click See All Results
Initial Results
I am using the section @baseclass.FindWork. For parameter FindWorkDisplay, I am passing either
Responsive or V6Search, which seem to produce the same result. Here is a screenshot of the initial results
from a search:

27
Observations
• these results are grouped by Case/Supporting Type
• I do not see any other consistent order such as creation date, update date, etc.
My business users asked me what is the logic behind which results are returned here. I tried to follow the logic
in the rules involved in the search. However, I hit a dead end in the activity Embed.ReportContent.pzSearch. It
appears that Pega performs the search via either
• pzSearch step 4: A SOAP service call to pzHttpLuceneSearchConnect, which has oddly few details
(most of the fields of the rule form are empty)
• pzSearch step 6, a call to pzSearchLucene, which, in turn, runs a single line of java:
pega.search(myStepPage);
Both of those are opaque to me, and the rules involved tend to have blank Description and Usage fields, so I
wasn't able to figure this out.
On the presentation side, I looked at some of the ListView rules involved and tried setting Sorting options in
some of them. Doing so didn't appear to have an effect on the ordering of the search results.
So, questions about the initial results:

• If we could describe the search behavior with a SELECT statement, what would be the ORDER BY
logic?
• Can we specify the ORDER BY logic?
• What would be the GROUP BY logic?
• Can we specify the GROUP BY logic?
The obvious answer to those questions is that once the results appear, I can click columns and make
selections in dropdowns to specify search behavior. My question is how I might affect the logic before the
results are returned to the UI.
Show all Results

When I tracked down the behavior after clicking Show All Results, I end up in nearly identical rules located in
different classes:
Service call

28
• Initial results: activity Embed-ReportContent.pzSearch makes Connect-HTTP call to

pzHttpLuceneSearchConnect
• Show All Results: activity Pega-Saerch-Work-Param.pzSearchWork, which seems to be identical to
Embed-ReportContent.pzSearch, makes Connect-HTTP call topySearchHTTPConnectWork
Java call
• Initial results: Activity Embed-ReportContent contains java pega.search(myStepPage);
• Show All Results: Activity Data-Find-Search runs the identical Java
Questions about Show All Results window:

Disregarding the back-end details, in terms of the behavior I see as an end user, the only differences between
initial results and the Show All Results window are obvious differences:
number of results returned
presentation
Aside from those obvious differences, are there any other differences in WHERE, ORDER BY or GROUP BY
logic?
Can I specify any of those?
Elliott Davis
Apr 1, 2016 4:06 PM
Hi Rajiv. Thanks for doing this Ask the Expert! I have several questions. To keep the discussion organized, I'll
try to ask one at a time.
I am running Pega 7.1.8.
First question: where can I find documentation on the differences between various styles of searching?
The standard section FindWork accepts a parameter "FindWorkDisplay". It offers four values:
• Advanced
• Basic
• Classic
• V6Search
In addition, it seems to accept a fifth option:

• Responsive
Where can I find documentation on how these affect the behavior of Pega searching?
History tab of FindWork contains the following brief information
Basic : Supports only to find by id.

Classic : Replicated the find gadget as seen in work user portal.

29
Advanced : Split button implementation.
Help topic "

How to create a composite portal"
By searching for "V6Search" on the PDN and in online help. I found mention of "V6Search"
• V6Search — Supports Elasticsearch full-text search as well as work "Entered by me," "Resolved by
me," and other choices. The user can refine search results in various ways, and can opt to search for
attached text documents that contain the search string.
Is more detailed information available?

I am particularly interested in Responsive and V6Search, which seem to result in the same behavior and UI.
Rajiv Nistala in response to kiran chilakala on page 30

Apr 1, 2016 2:48 AM
I have replied here - AES Configuration
kiran chilakala
Apr 1, 2016 2:14 AM
Hi Team,
Prior our application was developed in PRPC 5.5 version and we configured AES with AES 3.4 version on
6.1SP2 to monitor the application nodes.
Now currently our application is updated in PRPC 7.1.6 version and we would like to configure the AES with
current latest versions.
So we would like to please advise us on this regard.

30

Ask The Expert Full Text Search

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Ask The Expert Full Text Search

Hochgeladen von

Copyright:

Verfügbare Formate

Pega Product Support: Ask the Expert - Full

Posted by Marissa Rogers Apr 1, 2016

Message from Rajiv

Ask the Expert Rules

Generated by Pega Mesh on 2016-07-01-05:00

• Follow the Product Support Community's Community Rules of Engagement

Do You Have a Question for Rajiv? Leave a Comment below!

Brendan Horan in response to Rajiv Nistala on page 2

Ravi Kumar Pisupati in response to Rajiv Nistala on page 2

Rajiv Nistala in response to Ravi Kumar Pisupati on page 3

Generated by Pega Mesh on 2016-07-01-05:00

Rajiv Nistala in response to Markus Marbach on page 4

By expressions you mean declare expressions?

Ravi Kumar Pisupati

Generated by Pega Mesh on 2016-07-01-05:00

Thanks for all your insights of everything search related.

thanks in advance for your feedback.

Generated by Pega Mesh on 2016-07-01-05:00

Rajiv Nistala in response to Scott Holz on page 5

Rajiv Nistala in response to Halim Yahia on page 5

Halim Yahia in response to Rajiv Nistala on page 5

Rajiv Nistala in response to Halim Yahia on page 6

Generated by Pega Mesh on 2016-07-01-05:00

We have a lot of objects to index: arround 360000.

We tried to index several times, without success.

Have you any idea about this issue ?

Rajiv Nistala in response to Leonid Yakovlev on page 7

Generated by Pega Mesh on 2016-07-01-05:00

I suggest you raise an SR for this issue.

Leonid Yakovlev in response to Rajiv Nistala on page 7

WARN - Failed to index instance with key [ClassName P-4528 0 000000]

Rajiv Nistala in response to Leonid Yakovlev on page 7

ClassName key1 key2

Leonid Yakovlev in response to Rajiv Nistala on page 7

Rajiv Nistala in response to Leonid Yakovlev on page 8

Generated by Pega Mesh on 2016-07-01-05:00

Elastic Search log shows the following records:

Rajiv Nistala in response to Leonid Yakovlev on page 8

Generated by Pega Mesh on 2016-07-01-05:00

Apr 17, 2016 3:32 AM

Rajiv Nistala in response to Aakarsha K on page 9

Targets:Initialization,Database Server Installation,Data Load,Configuration,PRPC Launching

Generated by Pega Mesh on 2016-07-01-05:00

[replace] Replaced 1 occurrences in 1 files.

Database Server Installation:

[exec] Data page checksums are disabled.

[exec] creating directory F:/PRPCPersonalEdition/pgsql/data ... ok

[echo] Waiting for Postgres to start...

Generated by Pega Mesh on 2016-07-01-05:00

Total time: 45 seconds

Rajiv Nistala in response to David Kircheis on page 11

Rajiv Nistala in response to Alexey Muskancev on page 11

This question doesn't look related to full text search.

Thanks in advance for your thoughts on this!

Generated by Pega Mesh on 2016-07-01-05:00

Apr 12, 2016 7:26 AM

Rajiv Nistala in response to Ionut Hamin on page 12

Ionut Hamin in response to Rajiv Nistala on page 12

Rajiv Nistala in response to Ionut Hamin on page 13

Generated by Pega Mesh on 2016-07-01-05:00

Rajiv Nistala in response to Chiranjeet Mishra on page 13

Rajiv Nistala in response to Leonid Yakovlev on page 13