Question
Cooper Consulting
US
Last activity: 7 Apr 2017 7:10 EDT
OOTB work search not displaying newly created work objects
Hi,
We migrated from pega 6.1 sp2 to pega 7.1.8 recently.
We are in the user testing phase right now and we have an issue in work object search functionality.
We are not able to find some work objects from the pega OOTB work search.
particularity the newly created ones.
I found that the SystemWorkIndexer in the pega-rules agent in pega 7.1.8 is not enabled and its an final rule.
Not sure if this has anything to do with the issue.
Can any one of you had seen this issue before or has any idea why this is occuring.
Regards,
Pavan Kumar.
Accepted Solution
Cooper Consulting
US
1. Take backup of pr_sys_queue_ftsindexer table. – Don’t miss it, important.
2. Purge pr_sys_queue_ftsindexer table
3. Stop the JVM
4. Purge all the contents in the search index directory
5. Start the JVM
6. Go to SMA and make sure FTSIncrementalAgent is running
7. Go to search landing page and manually begin indexing RULE-, DATA-, WORK-
8. Make sure FTSIncrementalAgent is running every 60 seconds.
9. Test your user case to search newly create WO.
Following the above steps helped me to fix this issue.
thanks for all the responses which helped me to resolve this issue.
Updated: 17 Mar 2016 12:11 EDT
Pegasystems Inc.
FR
Hello,
You have to manage this agent at node level. Go to Records > SysAdmin > Agent Schedule to display agent schedule rule instances for nodes. Open the Pega-RULES instance related to your node and check the "Enabled?" checkbox for the SystemWorkIndexer agent.
Best regards,
Rodolphe
Cooper Consulting
US
Hi Rodolphe,
Thanks for you response.
That seems enabled in the agent scheduler.
But for some reason the ootb work search is not displaying the newly created work objects.
Basically I am not able to search new work objects from ootb work search feature.
Pegasystems
US
Go to System - > settings - > search
Make sure searching is turned on for work objects and/or try clicking the "reindex" button.
Also, check the setting for where the index is stored. Make sure it is in a folder that is not being shared with some other system that would interfere, and make sure it is in a folder that is no independently cleaned out when it shouldn't be. /Eric
Pegasystems
IN
Pega 7.1.8 uses Elastic Search (introduced in 7.1.7) and 6.1 sp2 uses Lucene. There is a difference in the agents and queues that we use in both cases.
Please try this - https://community.pega.com/support/support-articles/elastic-search-issues-work-and-rules
Cooper Consulting
US
Hi,
One small correction from my end is that,
We are not able to find the newly created work objects from the OOTB work search feature.
we are able to search the existing (already existing) work objects from the OOTB work search feature.
Pegasystems
IN
Did you check the settings mentioned in the PDN article on whether they are true or not after the upgrade?
Also, is FTSIncrementalIndexer agent running on all nodes?
Cooper Consulting
US
Yes FTSIncrementalIndexer is up and running.
checked all the settings mentioned in pdn and they are ok.
still the new work objects are not found in ootb work search feature.
Updated: 18 Mar 2016 16:00 EDT
Cooper Consulting
US
One thing i found is when i manually run the FTSIncrementalIndexer activity it is running indefinably and i found below exception in the log.
2016-03-18 14:24:26,644 [ Proprietary information hidden:8080-10] [ ] [ ] [ ] (ngineinterface.service.HttpAPI) ERROR terppgst|XX.XX.XX.XX - XX.XX.XX.XX: com.pega.pegarules.pub.context.RequestorLockException
com.pega.pegarules.pub.context.RequestorLockException: Unable to synchronize on requestor H08813734F79F9A5B929B8260D004CBC4 within 120 seconds: (thisThread = http-/XX.XX.XX.XX:8080-10) (originally locked by = http-XX.XX.XX.XX:8080-8) (finally locked by = http-/XX.XX.XX.XX:8080-8)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.getLockOnRequestor(PRSessionProviderImpl.java:1441)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:994)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:841)
at com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest(EngineAPI.java:331)
at com.pega.pegarules.session.internal.engineinterface.service.HttpAPI.invoke(HttpAPI.java:850)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:315)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)
One thing i found is when i manually run the FTSIncrementalIndexer activity it is running indefinably and i found below exception in the log.
2016-03-18 14:24:26,644 [ Proprietary information hidden:8080-10] [ ] [ ] [ ] (ngineinterface.service.HttpAPI) ERROR terppgst|XX.XX.XX.XX - XX.XX.XX.XX: com.pega.pegarules.pub.context.RequestorLockException
com.pega.pegarules.pub.context.RequestorLockException: Unable to synchronize on requestor H08813734F79F9A5B929B8260D004CBC4 within 120 seconds: (thisThread = http-/XX.XX.XX.XX:8080-10) (originally locked by = http-XX.XX.XX.XX:8080-8) (finally locked by = http-/XX.XX.XX.XX:8080-8)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.getLockOnRequestor(PRSessionProviderImpl.java:1441)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:994)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:841)
at com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest(EngineAPI.java:331)
at com.pega.pegarules.session.internal.engineinterface.service.HttpAPI.invoke(HttpAPI.java:850)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:315)
at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)
at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:225)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:367)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:408)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:223)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:245)
at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)
at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.tx.EjbBMTInterceptor.handleInvocation(EjbBMTInterceptor.java:104)
at org.jboss.as.ejb3.tx.BMTInterceptor.processInvocation(BMTInterceptor.java:56)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:86)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.EjbExceptionTransformingInterceptorFactories$2.processInvocation(EjbExceptionTransformingInterceptorFactories.java:103)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185)
at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
at com.pega.pegarules.internal.etier.interfaces.EngineLocal$$$view5.invokeEngine(Unknown Source)
at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngineInner(JNDIEnvironment.java:278)
at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngine(JNDIEnvironment.java:223)
at com.pega.pegarules.web.impl.WebStandardImpl.makeEtierRequest(WebStandardImpl.java:574)
at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:374)
at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:367)
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:408)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:223)
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:272)
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:121)
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doGet(WebStandardBoot.java:92)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:512)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
at java.lang.Thread.run(Thread.java:745)
If you run the FTSIndexer and it hangs, Pega7 should generate a thread dump at some point. Get the thread dump, find your thread and see what is happening.
As shipped, there is a mismatch between the indexes on the ftsindexer table and the actual implementation code, causing the queries to find the 'first' pending index request to run slowly and the indexer to fall behind. The slowness grows over time as it is a factor of the index queue table depth.
Elastic indexing requires that one or more online nodes be designated as indexers and all nodes must communicate with them. From designer studio, got to System / Settings / Search
a- look at the nodes and directories. Are they correct? Are all indexer nodes online and accessible?
b- Press the 'check' buttons. What do you get?
c- Do you have sufficient free disk space on the index-owning directories on all indexing nodes?
d- Go to class system-status-nodes / table pr_sys_statusnodes. Are the entries correct? Is the elastic endpoint of the indexer nodes an IO address accessible to other nodes?
e- how many rows are in the 'ftsindexer' queue table?
a-c are setup and environment issues; d could be network or operations, and e by itself is purely a DB performance issue.
If this is just db performance matter, simplest thing to do is
a- get the ftsindexer hotfix fir 718 from gcs support
b- apply hotfix and down systems
c- truncate table pr_sys_ftsindexer_queue
d- restart systems
If you run the FTSIndexer and it hangs, Pega7 should generate a thread dump at some point. Get the thread dump, find your thread and see what is happening.
As shipped, there is a mismatch between the indexes on the ftsindexer table and the actual implementation code, causing the queries to find the 'first' pending index request to run slowly and the indexer to fall behind. The slowness grows over time as it is a factor of the index queue table depth.
Elastic indexing requires that one or more online nodes be designated as indexers and all nodes must communicate with them. From designer studio, got to System / Settings / Search
a- look at the nodes and directories. Are they correct? Are all indexer nodes online and accessible?
b- Press the 'check' buttons. What do you get?
c- Do you have sufficient free disk space on the index-owning directories on all indexing nodes?
d- Go to class system-status-nodes / table pr_sys_statusnodes. Are the entries correct? Is the elastic endpoint of the indexer nodes an IO address accessible to other nodes?
e- how many rows are in the 'ftsindexer' queue table?
a-c are setup and environment issues; d could be network or operations, and e by itself is purely a DB performance issue.
If this is just db performance matter, simplest thing to do is
a- get the ftsindexer hotfix fir 718 from gcs support
b- apply hotfix and down systems
c- truncate table pr_sys_ftsindexer_queue
d- restart systems
e- rebuild the index
Accepted Solution
Cooper Consulting
US
1. Take backup of pr_sys_queue_ftsindexer table. – Don’t miss it, important.
2. Purge pr_sys_queue_ftsindexer table
3. Stop the JVM
4. Purge all the contents in the search index directory
5. Start the JVM
6. Go to SMA and make sure FTSIncrementalAgent is running
7. Go to search landing page and manually begin indexing RULE-, DATA-, WORK-
8. Make sure FTSIncrementalAgent is running every 60 seconds.
9. Test your user case to search newly create WO.
Following the above steps helped me to fix this issue.
thanks for all the responses which helped me to resolve this issue.
Pegasystems
IN
It's good to know that your issue is resolved.
Just to understand, why did you have to truncate the table? Were there too many records?
Select count(pxWorkKey), pxWorkClass from pr_sys_queue_ftsindexer group by pxWorkClass
The above query would give you the count of instances per class in the queue table. This would let you know if there are some classes which are generating too many entries in the table, clogging the queue. You might want to check if they are really meant to do so many changes. If not, please revisit your application logic.
Make sure you don't run into this again by looking into the backup table you took in step 1.