[ISSUE] Dependency Conflicts (log4j) & ClassCastException while using JDBC driver

[amarchigurupati]

Description
Using JDBC driver to interact with tables on Databricks catalog.
Connection is successful. But there are Exceptions in logs while accessing the result set. However, I can read results.

Perhaps this is the result of incompatibility between my log4j version and a version that's shaded in the databricks-jdbc JAR.

ERROR StatusLogger Unable to create Lookup for bundle
java.lang.ClassCastException: class org.apache.logging.log4j.core.lookup.ResourceBundleLookup
at java.base/java.lang.Class.asSubclass(Class.java:4102)

ERROR StatusLogger Unable to create Lookup for date
java.lang.ClassCastException: class org.apache.logging.log4j.core.lookup.DateLookup
at java.base/java.lang.Class.asSubclass(Class.java:4102)

mvn dependency:tree -Dverbose -Dincludes=org.apache.logging.log4j
Result of above command:

[INFO] com.xyz:jar:0.0.1-SNAPSHOT
[INFO] +- com.xyz-auth:jar:0.2.2:compile
[INFO] |  \- com.xyz.util:jar:0.2.2:compile
[INFO] |     \- org.apache.logging.log4j:log4j-core:jar:2.24.1:compile (version managed from 2.24.2)
[INFO] |        \- org.apache.logging.log4j:log4j-api:jar:2.24.1:compile (version managed from 2.24.1)
[INFO] \- org.springframework.boot:spring-boot-starter-actuator:jar:3.4.0:compile
[INFO]    \- org.springframework.boot:spring-boot-starter:jar:3.4.0:compile (version managed from 3.4.0; scope not updated to compile)
[INFO]       \- org.springframework.boot:spring-boot-starter-logging:jar:3.4.0:compile (version managed from 3.4.0)
[INFO]          \- org.apache.logging.log4j:log4j-to-slf4j:jar:2.24.1:compile (version managed from 2.24.1)
[INFO]             \- (org.apache.logging.log4j:log4j-api:jar:2.24.1:compile - version managed from 2.24.1; omitted for duplicate)

Reproduction
Nothing specific. We are using spring boot and databricks jdbc driver. Running simple queries to interact with catalog tables and iterating over result set.

Expected behavior
No Exception from databricks seen in the logs.

Is it a regression?
I don't no. we are using databricks-jdbc 2.6.40-patch-1

Debug Logs

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
ERROR StatusLogger Unable to create Lookup for bundle
java.lang.ClassCastException: class org.apache.logging.log4j.core.lookup.ResourceBundleLookup
at java.base/java.lang.Class.asSubclass(Class.java:4102)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:84)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:105)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:135)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.config.NullConfiguration.<init>(NullConfiguration.java:32)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.LoggerContext.<clinit>(LoggerContext.java:74)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.selector.ClassLoaderContextSelector.createContext(ClassLoaderContextSelector.java:254)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getDefault(ClassLoaderContextSelector.java:266)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:146)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:123)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:230)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.LogManager.getContext(LogManager.java:176)
at com.databricks.client.jdbc42.internal.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:60)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:47)
at com.databricks.client.jdbc42.internal.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:33)
at com.databricks.client.jdbc42.internal.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at com.databricks.client.jdbc42.internal.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
at com.databricks.client.jdbc42.internal.apache.arrow.memory.BaseAllocator.<clinit>(BaseAllocator.java:46)
at com.databricks.client.spark.arrow.ArrowBuffer.<init>(Unknown Source)
at com.databricks.client.spark.jdbc.SparkJDBC42Driver.createArrowBuffer(Unknown Source)
at com.databricks.client.hivecommon.api.HS2Client.makeRowsetBuffer(Unknown Source)
at com.databricks.client.hivecommon.dataengine.BackgroundFetcher.<init>(Unknown Source)
at com.databricks.client.hivecommon.dataengine.HiveJDBCResultSet.moveToNextRow(Unknown Source)
at com.databricks.client.jdbc.common.BaseForwardResultSet.next(Unknown Source)


ERROR StatusLogger Unable to create Lookup for ctx
java.lang.ClassCastException: class org.apache.logging.log4j.core.lookup.ContextMapLookup
at java.base/java.lang.Class.asSubclass(Class.java:4102)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:84)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:105)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:135)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.config.NullConfiguration.<init>(NullConfiguration.java:32)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.LoggerContext.<clinit>(LoggerContext.java:74)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.selector.ClassLoaderContextSelector.createContext(ClassLoaderContextSelector.java:254)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getDefault(ClassLoaderContextSelector.java:266)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:146)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:123)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:230)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.LogManager.getContext(LogManager.java:176)
at com.databricks.client.jdbc42.internal.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:60)
at com.databricks.client.jdbc42.internal.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:47)
at com.databricks.client.jdbc42.internal.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:33)
at com.databricks.client.jdbc42.internal.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at com.databricks.client.jdbc42.internal.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
at com.databricks.client.jdbc42.internal.apache.arrow.memory.BaseAllocator.<clinit>(BaseAllocator.java:46)
at com.databricks.client.spark.arrow.ArrowBuffer.<init>(Unknown Source)
at com.databricks.client.spark.jdbc.SparkJDBC42Driver.createArrowBuffer(Unknown Source)
at com.databricks.client.hivecommon.api.HS2Client.makeRowsetBuffer(Unknown Source)
at com.databricks.client.hivecommon.dataengine.BackgroundFetcher.<init>(Unknown Source)
at com.databricks.client.hivecommon.dataengine.HiveJDBCResultSet.moveToNextRow(Unknown Source)
at com.databricks.client.jdbc.common.BaseForwardResultSet.next(Unknown Source)


....
java.lang.ClassCastException: class org.apache.logging.log4j.core.lookup.DateLookup

java.lang.ClassCastException: class org.apache.logging.log4j.core.lookup.EnvironmentLookup

...

Other Information

• OS: [e.g. macOS]
• Version: [e.g. 0.1.0]

Additional context
As obvious, excluding log4j dependencies from databricks-jdbc didn't work. Since these dependencies maybe have repackaged to a different location/package.

<!-- JDBC drivers -->
        <dependency>
            <groupId>com.databricks</groupId>
            <artifactId>databricks-jdbc</artifactId>
            <version>2.6.40-patch-1</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-core</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-to-slf4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

[amarchigurupati]

Workaround
Adding EnableArrow=0 to the JDBC URL is not showing these Exceptions or conflicts.

But I'm interested to know how to get-rid of these exceptions with enabling apache arrow optimizations?

[doompling]

Wow I would have never figured that out. Thanks so much @amarchigurupati for the workaround, this issue was driving my crazy.