I've spent quite some time with one of the DBAs to try and determine the cause and came up with blanks. After a while I decided to check the deployment targets on the WebLogic server and found some oddities. Turns out someone half set up a cluster and didn't retarget everything to run on the cluster. The data source was targeted to the cluster, whereas the web service target was still just the managed server.
This is the error that I was seeing in the managed server logs:
WebLogic Managed Server Log
<14/05/2015 11:31:18 AM EST> <Warning> <JTA> <BEA-110478> <One-phase transaction [BEA1-000E3FC480F5EEF9DB72] is in an unknown state for logging resource [JDBC LLR, pool=MyDataSource, table=XXX.WL_LLR_XXX_SERVER]. The transaction operations either all rolled back, or all committed, or are blocked.
Transaction detail: [Name=[EJB com.example.ServiceStatelessCMTBean.submit()],Xid=BEA1-000E3FC480F5EEF9DB72(576254634),Status=Committing,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=4,seconds left=120,activeThread=Thread[pool-11-thread-7,5,Pooled Threads],NonXAServerResourceInfo[WL_JDBC_LLR_POOL.MyDataSource]=(ServerResourceInfo[WL_JDBC_LLR_POOL.MyDataSource]=(state=new,assigned=XXX_server),nonXAResource=JDBC LLR, pool=MyDataSource, table=XXX.WL_LLR_XXX_SERVER),SCInfo[XXX_domain+XXX_server]=(state=pre-prepared),properties=({weblogic.transaction.nonXAResource=WL_JDBC_LLR_POOL.MyDataSource, weblogic.transaction.name=[EJB com.example.ServiceStatelessCMTBean.submit()], weblogic.jdbc.llr=MyDataSource, LLR_TX_COMMIT=, weblogic.jdbc.remote.MyDataSource=t3://localhost:8001}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=XXX_server+localhost:8001+XXX_domain+t3+, XAResources={WLStore_XXX_domain_WseeJaxwsFileStore_auto_2, WLStore_XXX_domain__WLS_XXX_server, WSATGatewayRM_XXX_server_XXX_domain, eis/oracle/in-memory},NonXAResources={})],CoordinatorURL=XXX_server+localhost:8001+XXX_domain+t3+)].
Root cause: [weblogic.transaction.nonxa.NonXAException: IO Error: Connection reset].
weblogic.transaction.nonxa.NonXAException: IO Error: Connection reset
at weblogic.jdbc.wrapper.JTSLoggableResourceImpl.commit(JTSLoggableResourceImpl.java:371)
at weblogic.transaction.internal.NonXAServerResourceInfo.commit(NonXAServerResourceInfo.java:331)
at weblogic.transaction.internal.ServerSCInfo.startCommit(ServerSCInfo.java:569)
at weblogic.transaction.internal.ServerTransactionImpl.localCommit(ServerTransactionImpl.java:2063)
at weblogic.transaction.internal.ServerTransactionImpl.globalRetryCommit(ServerTransactionImpl.java:2857)
Truncated. see log file for complete stacktrace
Caused By: java.sql.SQLRecoverableException: IO Error: Connection reset
at oracle.jdbc.driver.T4CConnection.doCommit(T4CConnection.java:672)
at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3901)
at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3907)
at weblogic.jdbc.wrapper.JTSConnection.internalCommit(JTSConnection.java:501)
at weblogic.jdbc.wrapper.JTSLoggableResourceImpl.commit(JTSLoggableResourceImpl.java:356)
Truncated. see log file for complete stacktrace
Caused By: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at oracle.net.ns.Packet.send(Packet.java:403)
at oracle.net.ns.NSProtocol.sendMarker(NSProtocol.java:760)
at oracle.net.ns.NSProtocol.sendReset(NSProtocol.java:592)
Truncated. see log file for complete stacktrace
The fix is simple, have both the data source and the web service targets the same! Once I've made that change the errors went away.
-i
