Unable to fill pool: exception
We are not able to connect to the pega (7.3) application as we are getting the below highlighted error.
Recently we moved the database from stack 7 to stack 8 and with the new user we are able to connect to oracle client.
However Jboss connection is getting failed with below error for one of our environment.
Previously we have done the same change for other environment and its working fine.
Please provide suggestions.
We are not able to connect to the pega (7.3) application as we are getting the below highlighted error.
Recently we moved the database from stack 7 to stack 8 and with the new user we are able to connect to oracle client.
However Jboss connection is getting failed with below error for one of our environment.
Previously we have done the same change for other environment and its working fine.
Please provide suggestions.
2022-06-07 00:04:22,635 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (JCA PoolFiller) IJ000610: Unable to fill pool: java:/jdbc/AdminPegaRULES: javax.resource.ResourceException: IJ031084: Unable to create connection at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:345) at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:352) at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:287) at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.createConnectionEventListener(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1322) at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.fillTo(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1142) at org.jboss.jca.core.connectionmanager.pool.mcp.PoolFiller.run(PoolFiller.java:97) at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:774) at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:688) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:39) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:691) at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:321) ... 6 more
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
2022-06-07 00:08:37,099 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (JCA PoolFiller) IJ000610: Unable to fill pool: java:/jdbc/PegaRULES: javax.resource.ResourceException: IJ031084: Unable to create connection at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:345) at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:352) at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:287) at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.createConnectionEventListener(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1322) at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.fillTo(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1142) at org.jboss.jca.core.connectionmanager.pool.mcp.PoolFiller.run(PoolFiller.java:97) at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
***Edited by Moderator Marije to add Support Case Details ***
Accepted Solution
Pegasystems Inc.
GB
The support ticket was resolved when it was found that you were getting a connection from DB is getting timed out.
Logs were showing that while connection to the DB the request got timedout.
Our support staff suggested to check with your network team on why it is taking long time to connect to the below highlighted address.
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:523) at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:521) at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:660) at oracle.net.ns.NSProtocol.connect(NSProtocol.java:286) at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1438) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:518) ... 10 more Caused by: java.io.IOException: Connection timed out, socket connect lapse 127232 ms. no8dt210_vip.os.amadeus.net/xx.xx.xx.xxx xxxxx 0 1 true at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:209) at oracle.net.nt.ConnOption.connect(ConnOption.java:161) at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:470) ... 15 more
The support ticket was resolved when it was found that you were getting a connection from DB is getting timed out.
Logs were showing that while connection to the DB the request got timedout.
Our support staff suggested to check with your network team on why it is taking long time to connect to the below highlighted address.
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:523) at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:521) at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:660) at oracle.net.ns.NSProtocol.connect(NSProtocol.java:286) at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1438) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:518) ... 10 more Caused by: java.io.IOException: Connection timed out, socket connect lapse 127232 ms. no8dt210_vip.os.amadeus.net/xx.xx.xx.xxx xxxxx 0 1 true at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:209) at oracle.net.nt.ConnOption.connect(ConnOption.java:161) at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:470) ... 15 more
Caused by: java.net.ConnectException: Connection timed out at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:482) at sun.nio.ch.Net.connect(Net.java:474) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) at java.nio.channels.SocketChannel.open(SocketChannel.java:189) at oracle.net.nt.TimeoutSocketChannel.<init>(TimeoutSocketChannel.java:81) at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:169) ... 17 more
I will close this thread based on the understanding that you can check further steps with your network team and DB team
Pegasystems Inc.
GB
@LivyaJ02 can you give us more details of the exact method used for your change involved with "..Recently we moved the database from stack 7 to stack 8 "?
What is the current Oracle version deployed?
The error shows that Pega successfully calls Data Source Manager but database connection is not returned from connection pool. This appears to be an app server issue rather than a Pega error. Please debug further what reason is of exception
OnePool] (JCA PoolFiller) IJ000610: Unable to fill pool: java:/jdbc/AdminPegaRULES: javax.resource.ResourceException: IJ031084: Unable to create connection
The resource from Jboss, might be helpful -
https://stackoverflow.com/questions/2579999/jboss-jdbc-warning-unable-to-fill-pool
https://access.redhat.com/solutions/767103
https://developer.jboss.org/thread/158347
https://community.oracle.com/tech/developers/discussion/1331914/unable-to-fill-pool
Please note that connection test validates direct connection. It does not get connection from a pool. Your issue appears to in how DB is handling stale connections
I have found an open support ticket - can you confirm the INC number that you have already logged with the support department?
Accepted Solution
Pegasystems Inc.
GB
The support ticket was resolved when it was found that you were getting a connection from DB is getting timed out.
Logs were showing that while connection to the DB the request got timedout.
Our support staff suggested to check with your network team on why it is taking long time to connect to the below highlighted address.
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:523) at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:521) at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:660) at oracle.net.ns.NSProtocol.connect(NSProtocol.java:286) at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1438) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:518) ... 10 more Caused by: java.io.IOException: Connection timed out, socket connect lapse 127232 ms. no8dt210_vip.os.amadeus.net/xx.xx.xx.xxx xxxxx 0 1 true at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:209) at oracle.net.nt.ConnOption.connect(ConnOption.java:161) at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:470) ... 15 more
The support ticket was resolved when it was found that you were getting a connection from DB is getting timed out.
Logs were showing that while connection to the DB the request got timedout.
Our support staff suggested to check with your network team on why it is taking long time to connect to the below highlighted address.
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:523) at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:521) at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:660) at oracle.net.ns.NSProtocol.connect(NSProtocol.java:286) at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1438) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:518) ... 10 more Caused by: java.io.IOException: Connection timed out, socket connect lapse 127232 ms. no8dt210_vip.os.amadeus.net/xx.xx.xx.xxx xxxxx 0 1 true at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:209) at oracle.net.nt.ConnOption.connect(ConnOption.java:161) at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:470) ... 15 more
Caused by: java.net.ConnectException: Connection timed out at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:482) at sun.nio.ch.Net.connect(Net.java:474) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) at java.nio.channels.SocketChannel.open(SocketChannel.java:189) at oracle.net.nt.TimeoutSocketChannel.<init>(TimeoutSocketChannel.java:81) at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:169) ... 17 more
I will close this thread based on the understanding that you can check further steps with your network team and DB team