Intermittent NullPointerException in ConcurrentFixedCache.get [v2.6.7]

[mrakorcl]

I have observed a NullPointerException occurring intermittently when running queries. In the stack trace, you'll notice we use a session profiler, but that has no impact on internal EclipseLink state - we only use it to record events / metrics in memory.

Local Exception Stack: 
Exception [EclipseLink-6168] (Eclipse Persistence Services - 2.6.7.v20190604-418f1a1c56): org.eclipse.persistence.exceptions.QueryException
Exception Description: Query failed to prepare, unexpected error occurred: [java.lang.NullPointerException].
Internal Exception: java.lang.NullPointerException
Query: ReadAllQuery(referenceClass=JPAStep )
	at org.eclipse.persistence.exceptions.QueryException.prepareFailed(QueryException.java:1596)
	at org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:692)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.checkPrepare(ObjectLevelReadQuery.java:911)
	at org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:622)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:879)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1134)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:479)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1222)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2899)
	at org.eclipse.persistence.sessions.SessionProfilerAdapter.profileExecutionOfQuery(SessionProfilerAdapter.java:53)
	at oracle.bi.actio.core.persistence.performance.ActioEclipselinkPerformanceMonitor.profileExecutionOfQuery(ActioEclipselinkPerformanceMonitor.java:163)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1861)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1845)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1810)
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
	at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:478)
	at oracle.bi.actio.core.persistence.jpa.JPAExecTxns.replayEvents(JPAExecTxns.java:132)
	at oracle.bi.actio.core.persistence.jpa.JPAExecTxns.replayEvents(JPAExecTxns.java:43)
	at sun.reflect.GeneratedMethodAccessor316.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at oracle.bi.actio.core.transaction.TransactionInvocationHandler$1.doWork(TransactionInvocationHandler.java:66)
	at oracle.bi.actio.core.transaction.Transaction.doRetryTransaction(Transaction.java:213)
	at oracle.bi.actio.core.transaction.Transaction.doExceptionTransaction(Transaction.java:161)
	at oracle.bi.actio.core.transaction.TransactionInvocationHandler.invoke(TransactionInvocationHandler.java:56)
	at com.sun.proxy.$Proxy94.replayEvents(Unknown Source)
	at oracle.bi.actio.core.persistence.jpa.JPAExecPersistence.replayEvents(JPAExecPersistence.java:67)
	at oracle.bi.actio.core.journal.ExecutionTracker.<init>(ExecutionTracker.java:73)
	at oracle.bi.actio.core.executor.ExecutionContextImpl.<init>(ExecutionContextImpl.java:93)
	at oracle.bi.actio.core.executor.JobExecutor.runJobInPersistedJobsStore(JobExecutor.java:142)
	at oracle.bi.actio.core.server.quartz.ActiveServer.executeActioJobFromQuartz(ActiveServer.java:241)
	at oracle.bi.actio.core.server.quartz.ActioQuartzJob.execute(ActioQuartzJob.java:41)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
	at oracle.bi.actio.core.server.quartz.threadpool.TrackedRunnable.run(TrackedRunnable.java:26)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.NullPointerException
	at org.eclipse.persistence.internal.helper.ConcurrentFixedCache.get(ConcurrentFixedCache.java:66)
	at org.eclipse.persistence.descriptors.DescriptorQueryManager.getCachedExpressionQuery(DescriptorQueryManager.java:1806)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.prepareFromCachedQuery(ObjectLevelReadQuery.java:1971)
	at org.eclipse.persistence.queries.ReadAllQuery.prepare(ReadAllQuery.java:808)
	at org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:673)
	... 33 more

This cannot be reproduced at will - most likely occurs due to a race condition. EntityManager instance is never shared across threads, but we do have multiple threads using their own EntityManager and running queries concurrently. I see this in my integration test suite, occurring roughly once for every 1,000 test executions.

• EclipseLink version: 2.6.7
• Java/JDK version: 1.8
  persistence.xml settings:

   <persistence-unit name="JDBCUnit" transaction-type="RESOURCE_LOCAL">
    <provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>
    <mapping-file>META-INF/actio_orm.xml</mapping-file>
     <properties>
        <property name="eclipselink.logging.logger" value="JavaLogger"/>
        <property name="eclipselink.weaving" value="static"/>
        <property name="eclipselink.profiler" value="oracle.bi.actio.core.persistence.performance.ActioEclipselinkSIPerformanceMonitor"/>
     </properties>
</persistence-unit>

• Database provider/version: Derby v10.12.1.1
• JDBC driver: Apache Derby Embedded JDBC Driver v10.12.1.1

...
        final EntityManager em = emFactory.createEntityManager();
        final CriteriaBuilder cb = em.getCriteriaBuilder();
        final CriteriaQuery<JPAStep> cq = cb.createQuery(JPAStep.class);       
        final Root<JPAStep> jpaStepRoot = cq.from(JPAStep.class);
        cq.where(cb.equal(jpaStepRoot.get(JPAStep_.getJobInstanceId()),
                          SimpleConverter.instanceKeyToId(jobInstanceKey.getInstanceKey())));
        final TypedQuery<JPAStep> query = em.createQuery(cq);
        final List<JPAStep> results = query.getResultList();
...

Expected behavior
Query to succeed

Additional context
I only started to observe this issue once we started running these tests on new, ultra-fast VMs, further indicating that this is likely caused by a race condition.

[rfelcman]

Hello,
without context hard to say. If it's possible please provide:

• Thread dump to know what happens in the other threads as race condition is mentioned.
• Content of oracle.bi.actio.core.persistence.performance.ActioEclipselinkSIPerformanceMonitor
• Content of actio_orm.xml
• JPAStep.java
• DB table definition behind JPAStep.java
• How is emFactory initialized
• System load when NPE happens (number of threads and number of rows processed)
• Context how is mentioned code invoked

Does it happens in 2.6.9 or later versions?

I'd like ask You if there is chance to deliver some isolated test case as there is org.quartz thread pool used and context of oracle.bi.actio.core.persistence.jpa.JPAExecTxns.replayEvents(JPAExecTxns.java:132).
For cases like this "race condition" any customer test case is great, because it's usually about multiple conditions which leads into issue.

[mrakorcl]

I run the same test suite on v2.7.15 & java 17 and haven't encountered this issue.

Regardless of what's happening in other threads, the exception occurs while processing internal EclipseLink objects, which should not be corruptible by making calls to public API. The issue is clearly around thread safety and / or lack of defensive checks. I'm not at liberty to share the code beyond the public API calls being made that lead to this, and as indicated in my original post, this can't be reproduced at will