Question
Common Wealth Bank of Australia
AU
Last activity: 1 Aug 2017 2:29 EDT
CPU Usage 100%
We are facing CPU Usage as 100% in the Production environment.
Logs does not show any specific issues, but one of the thread dump revealed hazlecast took longer than usual.
How do we analyse the root cause? any thoughts?
Accepted Solution
Common Wealth Bank of Australia
AU
Hi Lochan,
Sorry to get back very late on this.
We found out that one of our app code was causing this issue. Hence, that particular code has been fixed and the issue is resolved.
We haven't created any SR for this issue.
Regards,
Sri
Pegasystems Inc.
US
A thread dump shows a single moment in time - and is not the best mechanism to identify CPU consumption issues.
I suggest opening a Support Request - providing a full set of artifacts for issue diagnosis:
- PegaRULES logfile
- PegaRULES-ALERT logfile
- App Server StdOut log
- Sample thread dumps (3 dumps taken 60 seconds apart)
- Sample heap dumps (3 dumps taken 5 minutes apart)
Pegasystems Inc.
US
One top of what Nick was asking above, if system runs on Linux, run 'top -H' command to see if java threads are consuming high CPU. If so, find out the PID of the offending thread(s) and convert the PID into hex and find the same thread id (in hex) in the thread dump to see what the thread is doing.
For example, below thread with nid=0x63b would have PID 1595 under top -H.
One top of what Nick was asking above, if system runs on Linux, run 'top -H' command to see if java threads are consuming high CPU. If so, find out the PID of the offending thread(s) and convert the PID into hex and find the same thread id (in hex) in the thread dump to see what the thread is doing.
For example, below thread with nid=0x63b would have PID 1595 under top -H.
"http-bio-8080-exec-10" daemon prio=10 tid=0x00007f79a8abe800 nid=0x63b runnable [0x00007f7a1190b000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at oracle.net.ns.Packet.receive(Packet.java:282)
at oracle.net.ns.DataPacket.receive(DataPacket.java:103)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:230)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:122)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:78)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1179)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1155)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:279)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1008)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1177)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1267)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3449)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3493)
- locked <0x000000070f568430> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at com.pega.pegarules.internal.bootstrap.phase2.jdbc.AbstractJdbcJarReader.readEntry(AbstractJdbcJarReader.java:334)
at com.pega.pegarules.internal.bootstrap.phase2.jdbc.AbstractJdbcJarReader.readBestEntry(AbstractJdbcJarReader.java:294)
.........
Pegasystems Inc.
IN
Hi Sri,
Please let us know the SR number once you create it!
Regards,
Lochan | Community Moderator | Pegasystems Inc.
Accepted Solution
Common Wealth Bank of Australia
AU
Hi Lochan,
Sorry to get back very late on this.
We found out that one of our app code was causing this issue. Hence, that particular code has been fixed and the issue is resolved.
We haven't created any SR for this issue.
Regards,
Sri
Pegasystems Inc.
IN
Hi Sri,
No worries! I'll mark your reply here as the Correct Answer. Hope to see more of you on PSC :)
Regards,