Troubleshooting Weblogic Connection Pool Leak

Weblogic debugging through the IDE (Intellij IDEA) 1. Go to the weblogic console: http://localhost:7001/console 2. Select the server you want to debug 3. Click on the "Remote Start" tab for the selected server 4. Edit the "Arguments" and add: -Djava.compiler=NONE -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=57022,server=y,suspend=n Note: Make sure there are no spaces within each java argument 5. You need to restart the weblogic server for these changes to become active. So do that now. 6. In IDEA, select Run->Debug 7. Click the + sign at the top to add a new configuration. Select "remote", enter "localhost" (or the server name) and the port number selected in step 4 (in this example 57022), select debugger mode="Attach" and select Trasnport="socket". You might also not need to run make every time you start debugging, so unselect that. 8. Click DEBUG. This will make IDEA attach itself to your Weblogic server. Set breakpoints in your code and browse to the proper URLs to see execution stop at your breakpoints. Explore... So you think you are leaking connections... You see this error in the log: weblogic.jdbc.extensions.PoolLimitSQLException: weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool MyPool to allocate to applications, please increase the size of the pool and retry.. Step 1: enable "Profile Connection Leak" in the weblogic console under domain->Services->JDBC->Data Sources->Configuration->Diagnostics Step 2: keep using the app and keep a close eye on the "leaked connection Count" in the weblogic console under domain->Services->JDBC->Data Sources->Monitoring->Statistics. If you don't see that column, add it through the "customize this table" link above the table. You should see the number of connection leaks going up as you hit the problematic code Step 3: capture a diagnostic image when you know for sure that you have a couple of leaked connections (do this once you "leaked connection count" from step 2 is >=1, the higher the count, the higher your confidence will be that you nailed the problematic code). You capture an image through the weblogic console under domain->Diagnostics->Diagnostic Images, select your server (the one with the leaked connections in step 2) then press on "capture image". Step 4: analyze the diagnostic image by going through the steps below. Basically, you need to unzip the diagnostic image and look in the JDBC.img file. This will provide stack traces of where all leaked connections were created (the smoking gun). 4.1 ssh to your machine (if not a local weblogic server) then go to the diagnostics image dir: cd /bea/user_projects/domains/mydomain/servers/myserver1/logs/diagnostic_images 4.2 unzip the diagnostics image file just created: unzip diagnostic_image_myserver1_2009_11_24_16_36_28.zip 4.3 check out the diagnostic information created: ls APPLICATION.img CONNECTOR.img Deployment.img InstrumentationImageSource.img JDBC.img JMS.img JNDI_IMAGE_SOURCE.img JTA.img JVM.img Logging.img PERSISTENT_STORE.img PathService.img SAF.img WORK_MANAGER.img WatchSource.img configuration.img diagnostic_image_myserver1_2009_11_24_16_36_28.zip image.summary 4.4 dig right into the JDBC diagnostic image (see the smoking gun evidence on the connection leak below) 4.5 check the code and find the connection leak more JDBC.img Dumping Resource Pool:MyPool Resource Pool:mypoolname:dumpPool Current Capacity = 5 Resource Pool:mypoolname:dumpPool dumping available resources, #entries = 0 Resource Pool:mypoolname:dumpPool dumping reserved resources, #entries = 5 Resource Pool:mypoolname:dumpPool reserved[0] autoCommit true,enabled=true,isXA=false,isJTS=false,vendorID=100,connUsed=true,doInit=false,'null',destroyed=false,poolname=mypoolname,a ppname=null,moduleName=null,connectTime=44,dirtyIsolationLevel=false,initialIsolationLevel=2,infected=false,lastSuccessfulConnectionUse=1227215220003,secondsToTrustAnIdlePoolConn ection=10,currentUser=java.lang.Exception at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:291) at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:314) at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:292) at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:425) at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:316) at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93) at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:106) at weblogic.jdbc.pool.Driver.connect(Driver.java:149) at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:642) at weblogic.jdbc.jts.Driver.connect(Driver.java:124) at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:339) at mypackage.common.access.DataSourceFactory.getConnection(DataSourceFactory.java:355) at mypackage.common.access.DataSourceFactory.(DataSourceFactory:123)*<--- AHA!! Connection leaked here! at mypackage.common.access.DatabaseManagerPool.getManager(DatabaseManagerPool.java:321)