Skip to main content

This content is closed to future replies and is no longer being maintained or updated.
Links may no longer function. If you have a similar request, please write a new post.

Question

 Nigel Sheridan-Smith (NigelS42)
Link Group

Link Group
AU
NigelS42 Member since 2017 3 posts
Link Group
Posted: Apr 27, 2018
Last activity: Apr 29, 2018
Posted: 27 Apr 2018 0:06 EDT
Last activity: 29 Apr 2018 4:02 EDT
Closed
Solved

Saving data and indexing is taking up 50-100% of CPU on Pega nodes

I have a staging environment that has Pega 7.3.1 nodes running at anywhere from 50-100% CPU after starting Tomcat 8.5.24 with managed MS SQL Server in Azure.

Using JProfiler it has been traced back to stack traces such as this (see indexing-foobar.png), or another internal Pega indexing process:

Show More

I have a staging environment that has Pega 7.3.1 nodes running at anywhere from 50-100% CPU after starting Tomcat 8.5.24 with managed MS SQL Server in Azure.

Using JProfiler it has been traced back to stack traces such as this (see indexing-foobar.png), or another internal Pega indexing process:

89.1% - 12,462 ms com.pega.pegarules.integration.engine.internal.services.listener.ServiceListenerBaseImpl.run
89.1% - 12,462 ms com.pega.pegarules.integration.engine.internal.services.email.EmailListener.run_
89.1% - 12,462 ms com.pega.pegarules.integration.engine.internal.services.email.EmailListener.oneIteration
89.1% - 12,462 ms com.pega.pegarules.integration.engine.internal.services.email.EmailListener.emailProcess
89.1% - 12,462 ms com.pega.pegarules.integration.engine.internal.services.email.EmailListener.handleRequest
89.1% - 12,462 ms com.pega.pegarules.integration.engine.internal.services.email.EmailListener.handleRequestContents
89.1% - 12,462 ms com.pega.pegarules.integration.engine.internal.services.email.EmailListener.handleStandardRequest
89.1% - 12,462 ms com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest
89.1% - 12,462 ms com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked
89.1% - 12,462 ms com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked
89.1% - 12,462 ms com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock
89.1% - 12,462 ms java.lang.reflect.Method.invoke
89.1% - 12,462 ms com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequestInner
89.1% - 12,462 ms com.pega.pegarules.integration.engine.internal.services.ServiceAPI.runActivities
89.1% - 12,462 ms com.pega.pegarules.integration.engine.internal.RuleExecutionUtils.runServiceActivity
89.1% - 12,462 ms com.pega.pegarules.session.internal.mgmt.PRThreadImpl.runActivitiesAlt
89.1% - 12,462 ms com.pega.pegarules.session.internal.mgmt.base.ThreadRunner.runActivitiesAlt
89.1% - 12,462 ms com.pega.pegarules.session.internal.mgmt.Executable.doActivity
89.1% - 12,462 ms com.pegarules.generated.activity.ra_action_applicationsetup_6373072fcfbdf7c63dcbcbd1cc37776e.perform
89.1% - 12,462 ms com.pegarules.generated.activity.ra_action_applicationsetup_6373072fcfbdf7c63dcbcbd1cc37776e.step9_circum0
89.1% - 12,462 ms com.pega.pegarules.session.internal.mgmt.Executable.invokeActivity
89.1% - 12,462 ms com.pega.pegarules.session.internal.mgmt.Executable.doActivity
89.1% - 12,462 ms com.pegarules.generated.activity.ra_action_applicationprofilesetup_0cbae53d9a9a647fe2c096b4fe6f34e0.perform
89.1% - 12,462 ms com.pegarules.generated.activity.ra_action_applicationprofilesetup_0cbae53d9a9a647fe2c096b4fe6f34e0.step3_circum0
89.1% - 12,462 ms com.pega.pegarules.session.internal.mgmt.Executable.invokeActivity
89.1% - 12,462 ms com.pega.pegarules.session.internal.mgmt.Executable.doActivity
89.1% - 12,462 ms com.pegarules.generated.activity.ra_action_applicationprofilesetup_getnextworksetup_fe5444c660d9106a2a29b3143af298b4.perform
89.1% - 12,462 ms com.pegarules.generated.activity.ra_action_applicationprofilesetup_getnextworksetup_fe5444c660d9106a2a29b3143af298b4.step8_circum0
89.1% - 12,462 ms com.pega.pegarules.data.internal.access.DatabaseImpl.save
89.1% - 12,462 ms com.pega.pegarules.data.internal.access.DatabaseImpl.save
89.1% - 12,462 ms com.pega.pegarules.data.internal.access.Saver.save
89.1% - 12,462 ms com.pega.pegarules.data.internal.access.Saver.save
89.1% - 12,462 ms com.pega.pegarules.data.internal.access.Saver.doIndexing
89.1% - 12,462 ms com.pega.pegarules.data.internal.access.Indexer.updateIndexes
89.1% - 12,462 ms com.pega.pegarules.data.internal.access.Indexer.calulateIndexesIncrementallyForInstance
89.1% - 12,462 ms com.pega.pegarules.data.internal.access.Indexer$Updates.write
89.1% - 12,462 ms com.pega.pegarules.data.internal.access.Indexer.writeIndexesForDef
86.4% - 12,084 ms com.pega.pegarules.data.internal.access.Indexer.writeIndex
86.1% - 12,039 ms com.pega.pegarules.data.internal.access.DeferredOperationsImpl.add

Previous version of Pega 7.1 seem to be using Lucene (based on the Student Admin Guide) but we are using Elastic Search.

One of the Pega nodes doesn't seem to respond to HTTP requests on TCP port 9300, as the connection is closed immediately.

Is this likely to be due to this comment in the student guide?

"SystemIndexer is the agent that performs the background indexing for all Rule
records. SystemWorkIndexer is the agent that performs the background indexing for both Data and Work
records. This agent must run for either of these to be indexed. A common error encountered in indexing
comes when the administrator accidentally turns off the SystemWorkIndexer while leaving Data Indexing
enabled."

The Elastic Search instances were not previously able to communicate due to blocked TCP ports so I'm wondering if it relates to ES cluster nodes competing to write to a distributed index that is corrupted.

Is there a way to view the ES index status? I have used the ES Kopf plugin in a previous role.

https://github.com/lmenezes/elasticsearch-kopf

Cheers,

Nigel


Show Less
To see attachments, please log in.
System Administration

Accepted Solution

Posted: 6 years ago
Posted: 29 Apr 2018 4:00 EDT
Nigel Sheridan-Smith (NigelS42)
Link Group

Link Group
AU

Thanks Nick... 

For Elastic Search it's TCP port 7200 that is HTTP (REST API), not TCP port 7300... So I guess it shouldn't respond with HTTP requests (d'oh!).

That probably also means I can't install the Kopf plugin. However the SMA (/prsysmgmt) WAR has some useful information, as does the indexer status within the Pega UI itself (Designer Studio > System > Settings > Search).

Pega 7.3.1 uses FTSIncrementalIndexer (not SystemWorkIndexer anymore), and searching PDN for that keyword turns up a bunch of useful notes, such as this one:

https://community.pega.com/support/support-articles/ftsincrementalindexer-agent-queue-size-large

This describes the table for queued indexing updates, which can be queried for its status, or cleared.

I restarted the whole Pega / ES cluster from scratch by clearing out the %TEMP%\PegaSearchIndex directory first. This seemed to resolve the issue.

Cheers,

Nigel

Show More

Thanks Nick... 

For Elastic Search it's TCP port 7200 that is HTTP (REST API), not TCP port 7300... So I guess it shouldn't respond with HTTP requests (d'oh!).

That probably also means I can't install the Kopf plugin. However the SMA (/prsysmgmt) WAR has some useful information, as does the indexer status within the Pega UI itself (Designer Studio > System > Settings > Search).

Pega 7.3.1 uses FTSIncrementalIndexer (not SystemWorkIndexer anymore), and searching PDN for that keyword turns up a bunch of useful notes, such as this one:

https://community.pega.com/support/support-articles/ftsincrementalindexer-agent-queue-size-large

This describes the table for queued indexing updates, which can be queried for its status, or cleared.

I restarted the whole Pega / ES cluster from scratch by clearing out the %TEMP%\PegaSearchIndex directory first. This seemed to resolve the issue.

Cheers,

Nigel

 


Show Less
To see attachments, please log in.
Posted: 6 years ago
Posted: 27 Apr 2018 10:14 EDT
Nick Loving (NickLoving_GCS)
PEGA
Senior Principal Engineer, Technical Support
Pegasystems Inc.
US

Click on the Designer Studio > System > Settings > Search. This will load the Search Settings landing page and you can click "Check"  to see the status of each index. That's all we offer with the Pega Platform, but it's possible that ES Kopf could work if you need something more.

I don't think the SystemWorkIndexer agent is needed in 7.3.1, so it shouldn't be causing an issue if it's not running.

I wonder if the index is being completely rebuilt each time you restart... Next time you do so, check the search landing page and see if it looks like it's building indexes. You would see the Index file size increasing.

To see attachments, please log in.

Accepted Solution

Posted: 6 years ago
Posted: 29 Apr 2018 4:00 EDT
Nigel Sheridan-Smith (NigelS42)
Link Group

Link Group
AU

Thanks Nick... 

For Elastic Search it's TCP port 7200 that is HTTP (REST API), not TCP port 7300... So I guess it shouldn't respond with HTTP requests (d'oh!).

That probably also means I can't install the Kopf plugin. However the SMA (/prsysmgmt) WAR has some useful information, as does the indexer status within the Pega UI itself (Designer Studio > System > Settings > Search).

Pega 7.3.1 uses FTSIncrementalIndexer (not SystemWorkIndexer anymore), and searching PDN for that keyword turns up a bunch of useful notes, such as this one:

https://community.pega.com/support/support-articles/ftsincrementalindexer-agent-queue-size-large

This describes the table for queued indexing updates, which can be queried for its status, or cleared.

I restarted the whole Pega / ES cluster from scratch by clearing out the %TEMP%\PegaSearchIndex directory first. This seemed to resolve the issue.

Cheers,

Nigel

Show More

Thanks Nick... 

For Elastic Search it's TCP port 7200 that is HTTP (REST API), not TCP port 7300... So I guess it shouldn't respond with HTTP requests (d'oh!).

That probably also means I can't install the Kopf plugin. However the SMA (/prsysmgmt) WAR has some useful information, as does the indexer status within the Pega UI itself (Designer Studio > System > Settings > Search).

Pega 7.3.1 uses FTSIncrementalIndexer (not SystemWorkIndexer anymore), and searching PDN for that keyword turns up a bunch of useful notes, such as this one:

https://community.pega.com/support/support-articles/ftsincrementalindexer-agent-queue-size-large

This describes the table for queued indexing updates, which can be queried for its status, or cleared.

I restarted the whole Pega / ES cluster from scratch by clearing out the %TEMP%\PegaSearchIndex directory first. This seemed to resolve the issue.

Cheers,

Nigel

 


Show Less
To see attachments, please log in.

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice