S3 minio deadlock catalog hadoop

[gesaleh]

S3 hadoop integration
configuration
minio runing in a docker
bucket created work
playground docker deployment with same network as minio docker

try to create a catalog_s3 using fileset (bucket name work)

# create a S3 catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "catalog",
  "type": "FILESET",
  "comment": "comment",
  "provider": "hadoop",
  "properties": {
    "location": "s3a://work/root",
    "s3-access-key-id": "access_key",
    "s3-secret-access-key": "secret_key",
    "s3-endpoint": "http://minio:9000",
    "filesystem-providers": "s3"
  }
}' http://localhost:8090/api/metalakes/metalake_demo/catalogs 

this works but when i try to create scheme it just hangs

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "s3_schema",
  "comment": "comment",
  "properties": {
    "location": "s3a://work/schemaS3"
  }
}' http://localhost:8090/api/metalakes/metalake_demo/catalogs/catalog_s3/schemas 

I get dead lock probably I'm doing wrong i'm suspecting the location value

2025-01-07 15:00:56 2025-01-07 14:00:56.133 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-57,5,main], ident=metalake_demo.catalog_s3} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=metalake_demo.catalog_s3}=1736258146570, ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=metalake_demo.catalog_s3}=1736258261725, ThreadIdentifier{thread=Thread[Gravitino-webserver-53,5,main], ident=metalake_demo.catalog_s3}=1736258145174, ThreadIdentifier{thread=Thread[Gravitino-webserver-71,5,main], ident=metalake_demo.catalog_s3}=1736258388375, ThreadIdentifier{thread=Thread[Gravitino-webserver-52,5,main], ident=metalake_demo.catalog_s3}=1736258343241, ThreadIdentifier{thread=Thread[Gravitino-webserver-59,5,main], ident=metalake_demo.catalog_s3}=1736258261725, ThreadIdentifier{thread=Thread[Gravitino-webserver-60,5,main], ident=metalake_demo.catalog_s3}=1736258385578, ThreadIdentifier{thread=Thread[Gravitino-webserver-68,5,main], ident=metalake_demo.catalog_s3}=1736258385578, ThreadIdentifier{thread=Thread[Gravitino-webserver-57,5,main], ident=metalake_demo.catalog_s3}=1736258336696, ThreadIdentifier{thread=Thread[Gravitino-webserver-70,5,main], ident=metalake_demo.catalog_s3}=1736258343242} 
2025-01-07 15:00:56 2025-01-07 14:00:56.133 WARN [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$checkDeadLock$1(LockManager.java:138)] - Dead lock detected for thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-70,5,main], ident=metalake_demo.catalog_s3} on node TreeLockNode{ident=/,hashCode=78}, threads that holding the node: {ThreadIdentifier{thread=Thread[Gravitino-webserver-65,5,main], ident=metalake_demo.catalog_s3}=1736258146570, ThreadIdentifier{thread=Thread[Gravitino-webserver-51,5,main], ident=metalake_demo.catalog_s3}=1736258261725, ThreadIdentifier{thread=Thread[Gravitino-webserver-53,5,main], ident=metalake_demo.catalog_s3}=1736258145174, ThreadIdentifier{thread=Thread[Gravitino-webserver-71,5,main], ident=metalake_demo.catalog_s3}=1736258388375, ThreadIdentifier{thread=Thread[Gravitino-webserver-52,5,main], ident=metalake_demo.catalog_s3}=1736258343241, ThreadIdentifier{thread=Thread[Gravitino-webserver-59,5,main], ident=metalake_demo.catalog_s3}=1736258261725, ThreadIdentifier{thread=Thread[Gravitino-webserver-60,5,main], ident=metalake_demo.catalog_s3}=1736258385578, ThreadIdentifier{thread=Thread[Gravitino-webserver-68,5,main], ident=metalake_demo.catalog_s3}=1736258385578, ThreadIdentifier{thread=Thread[Gravitino-webserver-57,5,main], ident=metalake_demo.catalog_s3}=1736258336696, ThreadIdentifier{thread=Thread[Gravitino-webserver-70,5,main], ident=metalake_demo.catalog_s3}=1736258343242} 

2025-01-07 13:55:45.209 INFO [Gravitino-webserver-53] [org.apache.gravitino.gcp.shaded.com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.<clinit>(GoogleHadoopFileSystemBase.java:611)] - GHFS version: 1.9.4-hadoop3
2025-01-07 13:55:45.308 INFO [Gravitino-webserver-53] [org.apache.commons.beanutils.FluentPropertyBeanIntrospector.introspect(FluentPropertyBeanIntrospector.java:147)] - Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property.
2025-01-07 13:55:45.313 WARN [Gravitino-webserver-53] [org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:134)] - Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties

i added the jar file aws-bundle-0.7.0-incubating.jar to the root/gravitino/catalogs/hadoop/libs i didn't change any value in the conf files

is there a need ? why it is going to deadlock ?

# ls gravitino*
gravitino-api-0.7.0-incubating.jar             gravitino-core-0.7.0-incubating.jar  gravitino-server-0.7.0-incubating.jar
gravitino-catalog-common-0.7.0-incubating.jar  gravitino-docs-0.7.0-incubating.jar  gravitino-server-common-0.7.0-incubating.jar
gravitino-common-0.7.0-incubating.jar          gravitino-meta-0.7.0-incubating.jar
# pwd
/root/gravitino/libs 

root@7c2e4f9a4c6e:~/gravitino/conf# more gravitino.conf 
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

# THE CONFIGURATION FOR Gravitino SERVER
gravitino.server.shutdown.timeout = 3000

# THE CONFIGURATION FOR Gravitino WEB SERVER
# The host name of the built-in web server
gravitino.server.webserver.host = 0.0.0.0
# The http port number of the built-in web server
gravitino.server.webserver.httpPort = 8090
# The min thread size of the built-in web server
gravitino.server.webserver.minThreads = 24
# The max thread size of the built-in web server
gravitino.server.webserver.maxThreads = 200
# The stop timeout of the built-in web server
gravitino.server.webserver.stopTimeout = 30000
# The timeout of idle connections
gravitino.server.webserver.idleTimeout = 30000
# The executor thread pool work queue size of the built-in web server
gravitino.server.webserver.threadPoolWorkQueueSize = 100
# The request header size of the built-in web server
gravitino.server.webserver.requestHeaderSize = 131072
# The response header size of the built-in web server
gravitino.server.webserver.responseHeaderSize = 131072

# THE CONFIGURATION FOR Gravitino ENTITY STORE
# The entity store to use
gravitino.entity.store = relational
# The backend for the entity store, we only supports JDBC
gravitino.entity.store.relational = JDBCBackend

# The JDBC URL for the entity store
gravitino.entity.store.relational.jdbcUrl = jdbc:h2
# The JDBC driver class name
gravitino.entity.store.relational.jdbcDriver = org.h2.Driver
# The JDBC user name
gravitino.entity.store.relational.jdbcUser = gravitino
# The JDBC password
gravitino.entity.store.relational.jdbcPassword = gravitino

# THE CONFIGURATION FOR Gravitino CATALOG
# The interval in milliseconds to evict the catalog cache
gravitino.catalog.cache.evictionIntervalMs = 3600000

# THE CONFIGURATION FOR AUXILIARY SERVICE
# Auxiliary service names, separate by ','
gravitino.auxService.names = iceberg-rest
# Iceberg REST service classpath
gravitino.auxService.iceberg-rest.classpath = iceberg-rest-server/libs, iceberg-rest-server/conf
# Iceberg REST service host
gravitino.auxService.iceberg-rest.host = 0.0.0.0
# Iceberg REST service http port
gravitino.auxService.iceberg-rest.httpPort = 9001
gravitino.auxService.iceberg-rest.catalog-backend = jdbc
gravitino.auxService.iceberg-rest.uri = jdbc:mysql://:3306/db
gravitino.auxService.iceberg-rest.warehouse = hdfs://:9000/user/iceberg/warehouse/
gravitino.auxService.iceberg-rest.jdbc.user = mysql
gravitino.auxService.iceberg-rest.jdbc.password = xxxxxxxxxxxx
gravitino.auxService.iceberg-rest.jdbc-driver = com.mysql.cj.jdbc.Driver
root@7c2e4f9a4c6e:~/gravitino/conf# 

root@7c2e4f9a4c6e:~/gravitino/conf# curl -I http://minio:9000//minio/health/live
HTTP/1.1 400 Bad Request
Accept-Ranges: bytes
Content-Length: 261
Content-Type: application/xml
Server: MinIO
Vary: Origin
Date: Tue, 07 Jan 2025 17:26:33 GMT

root@7c2e4f9a4c6e:~/gravitino/conf# curl -I minio:9000//minio/health/live
HTTP/1.1 400 Bad Request
Accept-Ranges: bytes
Content-Length: 261
Content-Type: application/xml
Server: MinIO
Vary: Origin
Date: Tue, 07 Jan 2025 17:26:40 GMT

[yuqi1129]

@gesaleh
We haven't tested compatibility with Minio using the S3 connector. I believe the problem you encountered was that the Hadoop File System could not connect to Minio, and the timeout was very long, causing the process to appear to hang.

If you are interested in solving the problem, please see HadoopCatalogOperations#getFileSystem or create a issue for it, we will try to solve it as soon as possible.

[yuqi1129]

Please double-check that you have provided the correct AKSK and endpoint for Minio. @gesaleh

[gesaleh]

Please double-check that you have provided the correct AKSK and endpoint for Minio. @gesaleh

I added the logs the minio is reachable from the gravitons container

There is a core-site.xml do we need to update this file ?

The goal is to be able to use the gods to list files

[gesaleh]

open issue #6156

[yuqi1129]

@gesaleh , This problem has been resolved by #5406 and I will close this dicussion temporarily, please reopen it if there are still some problems