Skip to main content

This content is closed to future replies and is no longer being maintained or updated.
Links may no longer function. If you have a similar request, please write a new post.

Question

 Gladson Soans (GladsonS3738)
Commonwealth Bank of Australia
Devops Engineer
Commonwealth Bank of Australia
AU
GladsonS3738 Member since 2017 9 posts
Commonwealth Bank of Australia
Posted: Jul 31, 2017
Last activity: Aug 28, 2017
Posted: 31 Jul 2017 22:15 EDT
Last activity: 28 Aug 2017 10:45 EDT
Closed
Solved

Pega client unable to connect to Pega-Ignite server

Hi,

I have spinned up 3 ignite docker containers in server mode connecting to DATA SCHEMA. When I start my pega application (separate docker container) with reference to ignite server in prconfig.xml, but neither the application starts up nor do I see any errors in the logs. When I remove the ignite reference from prconfig.xml in pega client, the application starts up successfully. Below are the config details

Ignite prconfig.xml

<?xml version="1.0" encoding="UTF-8"?>

<pegarules>
<!-- This is a minimum format prconfig.xml file. Only the settings which are required to access settings in the database are included.
All other settings which were formerly located only in this file are now Data-Admin-System-Settings. -->
<env name="initialization/settingsource" value="merged" />
<env name="database/databases/PegaRULES/dataSource" value="java:comp/env/jdbc/PegaRULES"/>
<env name="database/databases/PegaDATA/dataSource" value="java:comp/env/jdbc/PegaRULES"/>
<env name="security/urlaccesslog" value="NORMAL" />
<env name="security/urlaccessmode" value="WARN" />
<env name="identification/cluster/protocol" value="ignite" />
<env name="identification/cluster/public/address" value="public ip address xxxxxxx"/>

</pegarules>

Pega client prconfig.xml

Show More

Hi,

I have spinned up 3 ignite docker containers in server mode connecting to DATA SCHEMA. When I start my pega application (separate docker container) with reference to ignite server in prconfig.xml, but neither the application starts up nor do I see any errors in the logs. When I remove the ignite reference from prconfig.xml in pega client, the application starts up successfully. Below are the config details

Ignite prconfig.xml

<?xml version="1.0" encoding="UTF-8"?>

<pegarules>
<!-- This is a minimum format prconfig.xml file. Only the settings which are required to access settings in the database are included.
All other settings which were formerly located only in this file are now Data-Admin-System-Settings. -->
<env name="initialization/settingsource" value="merged" />
<env name="database/databases/PegaRULES/dataSource" value="java:comp/env/jdbc/PegaRULES"/>
<env name="database/databases/PegaDATA/dataSource" value="java:comp/env/jdbc/PegaRULES"/>
<env name="security/urlaccesslog" value="NORMAL" />
<env name="security/urlaccessmode" value="WARN" />
<env name="identification/cluster/protocol" value="ignite" />
<env name="identification/cluster/public/address" value="public ip address xxxxxxx"/>

</pegarules>

Pega client prconfig.xml

<?xml version="1.0" encoding="UTF-8"?>
<pegarules>
<!-- This is a minimum format prconfig.xml file. Only the settings which are required to access settings in the database are included.
All other settings which were formerly located only in this file are now Data-Admin-System-Settings. -->
<env name="initialization/settingsource" value="merged" />
<env name="database/databases/PegaRULES/dataSource" value="java:comp/env/jdbc/PegaRULES"/>
<env name="database/databases/PegaDATA/dataSource" value="java:comp/env/jdbc/PegaRULES"/>
<env name="agent/threadpoolsize" value="5" />
<env name="security/urlaccesslog" value="NORMAL" />
<env name="security/urlaccessmode" value="WARN" />
<!-- IGNITE -->
<env name="cluster/clientserver/clientmode" value="true" />
<env name="identification/cluster/protocol" value="ignite" />
<!-- IGNITE -->
</pegarules>

Log file:

Below is the log , there is no activity after loading Pega bootstrap configuration

01-Aug-2017 12:51:11.517 1050 [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRBootstrapDataSource. Bootstrap datatables schema: CEP_OUTBOUND_BR_DATA_OWNER
01-Aug-2017 12:51:12.397 1050 [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRBootstrapDataSource. Archive version [0 ]
01-Aug-2017 12:51:12.397 1050 [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRBootstrap. Engine schema defined in allclasses: null
01-Aug-2017 12:51:12.397 1050 [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRBootstrap. Engine classes schema: CEP_OUTBOUND_BR_RULES_OWNER
01-Aug-2017 12:51:12.397 1050 [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRBootstrap. Assembled classes schema: CEP_OUTBOUND_BR_RULES_OWNER
01-Aug-2017 12:51:12.398 1050 [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRBootstrap. prbootstrap.properties merged with prbootstrap entries in Data-Admin-System-Settings
01-Aug-2017 12:51:12.460 1050 [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRBootstrap. Using latest detected engine version:(EngineCodeSetVersion= 07-10-31). To specify a different version, use the setting com.pega.pegarules.bootstrap.codeset.version.Pega-EngineCode in prbootstrap.properties.
01-Aug-2017 12:51:12.462 1050 [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRMiniLoader. Will load phase 2 bootstrap from Pega-EngineCode:07-10-31
01-Aug-2017 12:51:12.788 INFO [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.phase2.PRBootstrapImpl. Using PRAppLoader implementation class: com.pega.pegarules.bootstrap.loader.PRAppLoader
01-Aug-2017 12:54:36.677 1050 [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.phase2.PRBootstrapImpl. PegaRULES signed jar files will be extracted to: /opt/apache-tomcat-8.5.11/work/Catalina/localhost/prweb/extractedFiles
01-Aug-2017 12:54:47.048 1050 [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.phase2.PRBootstrapImpl. PegaRULES classes will be loaded from the database
01-Aug-2017 12:54:47.909 1050 [localhost-startStop-1] com.pega.pegarules.internal.bootstrap.PRBootstrap.
============ PegaRULES Bootstrap Configuration ============
Date: Tue Aug 01 12:54:47 AEST 2017
Java Version: Oracle Corporation 1.8.0_60
Configuration: file:/opt/apache-tomcat-8.5.11/webapps/prweb/WEB-INF/classes/prbootstrap.properties
Pega-EngineCode: 07-10-31
ASM Version: 762781845
Database: Oracle Oracle Database 12c Enterprise Edition Release Proprietary information hidden.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
Driver: Oracle JDBC driver Proprietary information hidden.0
Active CodeSets: pega-enginecode:07-10-31 at (latest patch)
customer:06-01-01 at (latest patch)
=============== End Bootstrap Configuration ===============


Show Less
To see attachments, please log in.
DevOps
Installation and Deployment
Support Case Created

Accepted Solution

Posted: 7 years ago
Posted: 24 Aug 2017 19:25 EDT
Gladson Soans (GladsonS3738)
Commonwealth Bank of Australia
Devops Engineer
Commonwealth Bank of Australia
AU

Apologies for the delay Nick. Yes its the timezone issue where Ignite registers the time in UTC whereas Pega client compares in AEST. The engineering team have identified the bug and provided the hotfix. All good now

To see attachments, please log in.
Posted: 7 years ago
Posted: 1 Aug 2017 2:38 EDT
Gladson Soans (GladsonS3738)
Commonwealth Bank of Australia
Devops Engineer
Commonwealth Bank of Australia
AU

Hi,

I can see below logs when starting up Pega:

>>>    __________  ________________
>>>   /  _/ ___/ |/ /  _/_  __/ __/
>>>  _/ // (7 7    // /  / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 1.7.0#20160802-sha1:383273e3
>>> 2016 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org

Show More

Hi,

I can see below logs when starting up Pega:

>>>    __________  ________________
>>>   /  _/ ___/ |/ /  _/_  __/ __/
>>>  _/ // (7 7    // /  / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 1.7.0#20160802-sha1:383273e3
>>> 2016 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org

2017-08-01 16:28:23,964 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - Config URL: n/a
2017-08-01 16:28:23,964 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - Daemon mode: off
2017-08-01 16:28:23,964 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - OS: Linux 3.10.0-514.26.2.el7.x86_64 amd64
2017-08-01 16:28:23,964 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - OS user: root
2017-08-01 16:28:23,965 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - Language runtime: Java Platform API Specification ver. 1.8
2017-08-01 16:28:23,965 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - VM information: Java(TM) SE Runtime Environment 1.8.0_60-b27 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.60-b23
2017-08-01 16:28:23,965 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - VM total memory: 7.9GB
2017-08-01 16:28:23,965 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - Remote Management [restart: off, REST: on, JMX (remote: off)]
2017-08-01 16:28:23,965 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - IGNITE_HOME=null
2017-08-01 16:28:23,965 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - Configured caches ['ignite-marshaller-sys-cache', 'ignite-sys-cache', 'ignite-atomics-sys-cache']
2017-08-01 16:28:25,396 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - Local node user attribute [hostname=61bba557fa8d]
2017-08-01 16:28:25,397 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - Local node user attribute [node-type=PLATFORM_PRPC]
2017-08-01 16:28:25,397 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - Local node user attribute [organization=pega.com]
2017-08-01 16:28:35,237 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - Non-loopback local IPs: Proprietary information hidden, Proprietary information hidden
2017-08-01 16:28:35,237 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - Enabled local MACs: 02420A000703, 0242AC120004
2017-08-01 16:28:39,217 [        61bba557fa8d] [                    ] [                    ] (s.plugin.IgnitePluginProcessor) INFO    - Configured plugins:
2017-08-01 16:28:39,217 [        61bba557fa8d] [                    ] [                    ] (s.plugin.IgnitePluginProcessor) INFO    -   ^-- None
2017-08-01 16:28:39,217 [        61bba557fa8d] [                    ] [                    ] (s.plugin.IgnitePluginProcessor) INFO    -
2017-08-01 16:28:49,829 [        61bba557fa8d] [                    ] [                    ] (cation.tcp.TcpCommunicationSpi) WARN    - Failed to load 'igniteshmem' library from classpath. Will try to load it from IGNITE_HOME.
2017-08-01 16:28:49,876 [        61bba557fa8d] [                    ] [                    ] (cation.tcp.TcpCommunicationSpi) WARN    - Failed to start shared memory communication server.
2017-08-01 16:28:53,131 [        61bba557fa8d] [                    ] [                    ] (cation.tcp.TcpCommunicationSpi) INFO    - Successfully bound to TCP port [port=47100, locHost= Proprietary information hidden/ Proprietary information hidden]
2017-08-01 16:29:02,993 [        61bba557fa8d] [                    ] [                    ] (ckpoint.noop.NoopCheckpointSpi) WARN    - Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
2017-08-01 16:29:08,479 [        61bba557fa8d] [                    ] [                    ] (collision.GridCollisionManager) WARN    - Collision resolution is disabled (all jobs will be activated upon arrival).
2017-08-01 16:29:09,760 [        61bba557fa8d] [                    ] [                    ] (wapspace.noop.NoopSwapSpaceSpi) WARN    - Swap space is disabled. To enable use FileSwapSpaceSpi.
2017-08-01 16:29:09,896 [        61bba557fa8d] [                    ] [                    ] (ite.internal.IgniteKernal%PRPC) INFO    - Security status [authentication=off, tls/ssl=off]
2017-08-01 16:30:11,773 [        61bba557fa8d] [                    ] [                    ] (tocols.tcp.GridTcpRestProtocol) INFO    - Command protocol successfully started [name=TCP binary, host= Proprietary information hidden/ Proprietary information hidden, port=11211]
2017-08-01 16:30:31,192 [-msg-worker-#4%PRPC%] [                    ] [                    ] ( discovery.tcp.TcpDiscoverySpi) WARN    - IP finder returned empty addresses list. Please check IP finder configuration. Will retry every 2 secs.
2017-08-01 16:35:31,603 [-msg-worker-#4%PRPC%] [                    ] [                    ] ( discovery.tcp.TcpDiscoverySpi) WARN    - IP finder returned empty addresses list. Please check IP finder configuration. Will retry every 2 secs.

 


Show Less
To see attachments, please log in.
Posted: 7 years ago
Posted: 1 Aug 2017 4:21 EDT
Gladson Soans (GladsonS3738)
Commonwealth Bank of Australia
Devops Engineer
Commonwealth Bank of Australia
AU

Note that Pega client is able to get the ignite server details from prpc_ignite... table but its unable to get the ignite ip address as you can see in the logs highlighted in bold above. It looks like we need to pass in the ignite ip address explicitly to Pega client

To see attachments, please log in.
Posted: 7 years ago
Posted: 1 Aug 2017 9:57 EDT
Nick Loving (NickLoving_GCS)
PEGA
Senior Principal Engineer, Technical Support
Pegasystems Inc.
US

Hi Gladson,

Have you started one or all of the Ignite servers before trying to start the Pega clients? (Please specify)
You're specifying that the Ignite servers connect to the data schema... Can you provide the clusterService.properties file from one of the servers?
Can you provide the PegaCLUSTER log from one of the Ignite servers that is started?

To see attachments, please log in.
Posted: 7 years ago
Posted: 2 Aug 2017 4:34 EDT
Gladson Soans (GladsonS3738)
Commonwealth Bank of Australia
Devops Engineer
Commonwealth Bank of Australia
AU

Hi Nick,

I have found out the root cause here, but unable to find out the fix .​

Apparently Pega Client queries the table PRPC_IGNITE_SERVER_ADDR in DATA_SCHEMA for the last heartbeat of Ignite server and if the heartbeat is greater than 10 minutes, it would not connect to the Ignite server and hence go in a wait/hung state waiting for the heartbeat. What I have seen from Ignite server is that it only updates the hearbeat once Ignite server starts up but there isn't any regular update on pzlastheartbeat column in PRPC_IGNITE_SERVER_ADDR . I manually tried to update the pzlastheartbeat time to current time and noticed that Pega Client was able to connect the ignite server immediately. 

The question now is that why doesnt Ignite regularly update pzlastheartbeat column in PRPC_IGNITE_SERVER_ADDR ? Is it required for Pega-Client and Ignite server to start up simultaneously ?

Logs indicating Pega Client ignoring the ignite server due to heartbeat time order than 10mins

*/

(ging.ClientJdbcCandidateLogger) INFO    - Found potential members in database with last heartbeat time older than 10 minutes (these will not be included)

Show More

Hi Nick,

I have found out the root cause here, but unable to find out the fix .​

Apparently Pega Client queries the table PRPC_IGNITE_SERVER_ADDR in DATA_SCHEMA for the last heartbeat of Ignite server and if the heartbeat is greater than 10 minutes, it would not connect to the Ignite server and hence go in a wait/hung state waiting for the heartbeat. What I have seen from Ignite server is that it only updates the hearbeat once Ignite server starts up but there isn't any regular update on pzlastheartbeat column in PRPC_IGNITE_SERVER_ADDR . I manually tried to update the pzlastheartbeat time to current time and noticed that Pega Client was able to connect the ignite server immediately. 

The question now is that why doesnt Ignite regularly update pzlastheartbeat column in PRPC_IGNITE_SERVER_ADDR ? Is it required for Pega-Client and Ignite server to start up simultaneously ?

Logs indicating Pega Client ignoring the ignite server due to heartbeat time order than 10mins

*/

(ging.ClientJdbcCandidateLogger) INFO    - Found potential members in database with last heartbeat time older than 10 minutes (these will not be included)

2017-08-01 18:10:53,880 [-msg-worker-#4%PRPC%] [          ] [                    ] [                    ] (ging.ClientJdbcCandidateLogger) INFO    - ClusterNode [inetSocketAddress=/ Proprietary information hidden:5701, lastHeartbeat=2017-08-01T08:07:30.000Z UTC]

2017-08-01 18:10:53,880 [-msg-worker-#4%PRPC%] [          ] [                    ] [                    ] (ging.ClientJdbcCandidateLogger) INFO    - ClusterNode [inetSocketAddress=/ Proprietary information hidden:5701, lastHeartbeat=2017-08-01T08:01:30.000Z UTC]

2017-08-01 18:10:53,880 [-msg-worker-#4%PRPC%] [          ] [                    ] [                    ] (ging.ClientJdbcCandidateLogger) INFO    - ClusterNode [inetSocketAddress=/ Proprietary information hidden:5701, lastHeartbeat=2017-08-01T08:03:30.000Z UTC]


Show Less
To see attachments, please log in.
Posted: 7 years ago
Posted: 3 Aug 2017 9:42 EDT
Nick Loving (NickLoving_GCS)
PEGA
Senior Principal Engineer, Technical Support
Pegasystems Inc.
US

Hi Gladson,

So, even if left running for 30 minutes or an hour (just for testing / troubleshooting purposes), the pzlastheartbeat column still has the value from when the ignite server first started? Is that correct?

If not, I would suspect there may be a timezone issue... But perhaps there is something else going on.

To see attachments, please log in.

Accepted Solution

Posted: 7 years ago
Posted: 24 Aug 2017 19:25 EDT
Gladson Soans (GladsonS3738)
Commonwealth Bank of Australia
Devops Engineer
Commonwealth Bank of Australia
AU

Apologies for the delay Nick. Yes its the timezone issue where Ignite registers the time in UTC whereas Pega client compares in AEST. The engineering team have identified the bug and provided the hotfix. All good now

To see attachments, please log in.

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice