Error:-
[2018-02-22T16:03:42.264+05:30]
[oim_server1] [NOTIFICATION] [] [org.eclipse.persistence.session.oim] [tid:
OIMQuartzScheduler_Worker-9] [userId: oiminternal] [ecid:
0000M6wj4gxFk3QLUa4Eyc1QZcOi000004, 1:22446] [APP: oim#11.1.2.0.0]
Communication failure detected when attempting to perform read query outside of
a transaction. Attempting to retry query. Error was: Exception
[EclipseLink-4002] (Eclipse Persistence Services - 2.3.1.v20111018-r10243): org.eclipse.persistence.exceptions.DatabaseException
[[Internal Exception: java.sql.SQLException: Internal error: Cannot obtain
XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources
currently available in pool oimOperationsDB to allocate to applications, please
increase the size of the pool and retry..at
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:604)
        at
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:353)
        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:340)
        at
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)
        at
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
        at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:125)
        at
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:77)
        at
weblogic.jdbc.jta.DataSource.getXAConnectionFromPool(DataSource.java:1791)
        at
weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1582)
        at
weblogic.jdbc.jta.DataSource.getConnectionInternal(DataSource.java:499)
        at
weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:483)
        at
weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:527)
        at
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:513)
        at
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:506)
        at
org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:123)
        at
org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
        at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:330)
        at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:293)
        at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:565)
        at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1508)
        at
org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:305)
        at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:579)
        at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:535)
        at
org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1717)
        at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:566)
        at
org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:207)
at
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207)
        at
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193)
        at
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:264)
        at
org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:646)
        at
org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2611)
        at
org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2570)
        at
org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:420)
        at
org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1081)
        at
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:844)
        at
org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1040)
        at
org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:392)
        at
org.eclipse.persistence.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:2831)
        at
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1516)
        at
org.eclipse.persistence.internal.sessions.AbstractSession.retryQuery(AbstractSession.java:1586)
        at
org.eclipse.persistence.sessions.server.ClientSession.retryQuery(ClientSession.java:593)
        at
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1552)
        at org.eclipse.persistence.internal.sessions.AbstractSession.retryQuery(AbstractSession.java:1586)
        at
org.eclipse.persistence.sessions.server.ClientSession.retryQuery(ClientSession.java:593)
        at
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1552)
        at
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1498)
        at
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1449)
        at
org.eclipse.persistence.internal.sessions.AbstractSession.readAllObjects(AbstractSession.java:3193)
        at
oracle.iam.scheduler.impl.util.SchedulerUtil.getLastHistoryOfJob(SchedulerUtil.java:1245)
        at
oracle.iam.scheduler.impl.quartz.QuartzJob$TaskExecutionAction.run(QuartzJob.java:188)
        at
java.security.AccessController.doPrivileged(Native Method)
        at
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:266)
        at
oracle.security.jps.internal.jaas.AccActionExecutor.execute(AccActionExecutor.java:65)
        at
oracle.security.jps.internal.jaas.AbstractSubjectSecurity$ActionExecutorWrapper.execute(AbstractSubjectSecurity.java:228)
        at oracle.security.jps.internal.jaas.CascadeActionExecutor$SubjectPrivilegedAction.run(CascadeActionExecutor.java:68)
        at
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at
weblogic.security.Security.runAs(Security.java:41)
        at
oracle.security.jps.wls.jaas.WlsActionExecutor.execute(WlsActionExecutor.java:47)
        at
oracle.security.jps.internal.jaas.CascadeActionExecutor.execute(CascadeActionExecutor.java:50)
        at
oracle.security.jps.internal.jaas.AbstractSubjectSecurity$ActionExecutorWrapper.execute(AbstractSubjectSecurity.java:228)
        at
Thor.API.Security.LoginHandler.AssertionLoginSession.runAs(AssertionLoginSession.java:93)
        at
oracle.iam.scheduler.impl.quartz.QuartzJob.execute(QuartzJob.java:85)
        at
org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
Cause:-
DB is down and listener is not up and running.
Solution:-
1. Open new terminal and check with lsnrctl status
2. Check DB connectivity through sqldeveloper
3. Contact DB team and ensure that DB and listener are up and running.
Thanks,
Aditya.