Merge pull request #2 from timeplus-io/proton

0.0.1 a working version metabase driver for Timeplus Proton. More enhancements to do
timeplus-io · Nov 18, 2023 · a5cdef5 · a5cdef5
2 parents 6377168 + c38ed89
commit a5cdef5
Show file tree

Hide file tree

Showing 12 changed files with 340 additions and 361 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -21,13 +21,13 @@ assignees: ''
 
 ### Configuration
 #### Environment
-* metabase-clickhouse-driver version:
-* metabase-clickhouse-driver configuration:
+* metabase-proton-driver version:
+* metabase-proton-driver configuration:
 * Metabase version:
 * OS:
 
-#### ClickHouse server
-* ClickHouse Server version:
-* ClickHouse Server non-default settings, if any:
-* `CREATE TABLE` statements for tables involved:
-* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
+#### Proton server
+* Proton Server version:
+* Proton Server non-default settings, if any:
+* `CREATE STREAM` statements:
+* Sample data
diff --git a/.gitignore b/.gitignore
@@ -12,3 +12,5 @@
 .calva
 .cpcache
 .joyride
+.DS_Store
+repo
diff --git a/README.md b/README.md
@@ -1,146 +1,69 @@
-<p align="center" style="font-size:300%">
-<img src="https://www.metabase.com/images/logo.svg" width="200px" align="center">
-<img src=".static/clickhouse.svg" width="180px" align="center">
-<h1 align="center">ClickHouse driver for Metabase</h1>
-</p>
-<br/>
-<p align="center">
-<a href="https://github.com/enqueue/metabase-clickhouse-driver/actions/workflows/check.yml">
-<img src="https://github.com/enqueue/metabase-clickhouse-driver/actions/workflows/check.yml/badge.svg?branch=master">
-</a>
-<a href="https://github.com/enqueue/metabase-clickhouse-driver/releases">
-<img src="https://img.shields.io/github/release/enqueue/metabase-clickhouse-driver.svg?label=latest%20release">
-</a>
-<a href="https://raw.githubusercontent.com/enqueue/metabase-clickhouse-driver/master/LICENSE">
-<img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg">
-</a>
-</p>
-
-## About
-
-[ClickHouse](https://clickhouse.com) ([GitHub](https://github.com/ClickHouse/ClickHouse)) database driver for the [Metabase](https://metabase.com) ([GitHub](https://github.com/metabase/metabase)) business intelligence front-end.
-
-## Installation
-
-### Run using Metabase Jar
-
-1. Download a fairly recent Metabase binary release (jar file) from the [Metabase distribution page](https://metabase.com/start/jar.html).
-2. Download the ClickHouse driver jar from this repository's [Releases](https://github.com/enqueue/metabase-clickhouse-driver/releases) page
-3. Create a directory and copy the `metabase.jar` to it.
-4. In that directory create a sub-directory called `plugins`.
-5. Copy the ClickHouse driver jar to the `plugins` directory.
-6. Make sure you are the in the directory where your `metabase.jar` lives.
-7. Run `MB_PLUGINS_DIR=./plugins; java -jar metabase.jar`.
-
-For example [(using Metabase v0.47.2 and ClickHouse driver 1.2.2)](#choosing-the-right-version):
-
-```bash
-export METABASE_VERSION=v0.47.2
-export METABASE_CLICKHOUSE_DRIVER_VERSION=1.2.2
-
-mkdir -p mb/plugins && cd mb
-curl -o metabase.jar https://downloads.metabase.com/$METABASE_VERSION/metabase.jar
-curl -L -o plugins/ch.jar https://github.com/ClickHouse/metabase-clickhouse-driver/releases/download/$METABASE_CLICKHOUSE_DRIVER_VERSION/clickhouse.metabase-driver.jar
-MB_PLUGINS_DIR=./plugins; java -jar metabase.jar
-```
-
-### Run as a Docker container
+# Proton driver for Metabase
 
-Alternatively, if you don't want to run Metabase Jar, you can use a Docker image:
+This repo is a forked from https://github.com/ClickHouse/metabase-clickhouse-driver with necessary revisions to better fit streaming processing and Proton.
 
-```bash
-export METABASE_DOCKER_VERSION=v0.47.2
-export METABASE_CLICKHOUSE_DRIVER_VERSION=1.2.2
+## Add database
 
-mkdir -p mb/plugins && cd mb
-curl -L -o plugins/ch.jar https://github.com/ClickHouse/metabase-clickhouse-driver/releases/download/$METABASE_CLICKHOUSE_DRIVER_VERSION/clickhouse.metabase-driver.jar
-docker run -d -p 3000:3000 \
-  --mount type=bind,source=$PWD/plugins/ch.jar,destination=/plugins/clickhouse.jar \
-  metabase/metabase:$METABASE_DOCKER_VERSION
-```
-
-## Choosing the Right Version
-
-| Metabase Release | Driver Version |
-| ---------------- | -------------- |
-| 0.33.x           | 0.6            |
-| 0.34.x           | 0.7.0          |
-| 0.35.x           | 0.7.1          |
-| 0.37.3           | 0.7.3          |
-| 0.38.1+          | 0.7.5          |
-| 0.41.2           | 0.8.0          |
-| 0.41.3.1         | 0.8.1          |
-| 0.42.x           | 0.8.1          |
-| 0.44.x           | 0.9.1          |
-| 0.45.x           | 1.1.0          |
-| 0.46.x           | 1.1.7          |
-| 0.47.x           | 1.2.3          |
-
-## Creating a Metabase Docker image with ClickHouse driver
-
-You can use a convenience script `build_docker_image.sh`, which takes three arguments: Metabase version, ClickHouse driver version, and the desired final Docker image tag.
-
-```bash
-./build_docker_image.sh v0.47.2 1.2.2 my-metabase-with-clickhouse:v0.0.1
-```
+1. Once you've started up Metabase, open http://localhost:3000 , go to "Admin settings" (top-right), then "Databases" tab and add a database and select "Timieplus Proton".
+2. You'll need to provide the Host/Port. Default localhost and 3218 just work.
 
-where `v0.47.2` is Metabase version, `1.2.2` is ClickHouse driver version, and `my-metabase-with-clickhouse:v0.0.1` being the tag.
+## Run Query
+Please note, by default Proton's query behavior is streaming SQL, looking for new data in the future and never ends. This can be considered as hang for Metabase. So please use `select .. from .. LIMIT 100` to stop the query at 100 events. Or use a historical query, such as `select .. from table(car_live_data)..`
 
-Then you should be able to run it:
+### Build from source
+The build process is largely based on https://github.com/databendcloud/metabase-databend-driver. (IMHO, Leiningen provides much better compiling error message than the built-in `clojure -X:build:drivers:build/driver`)
 
-```bash
-docker run -d -p 3000:3000 --name my-metabase my-metabase-with-clickhouse:v0.0.1
-```
+### Prerequisites
 
-or use it with Docker compose, for example:
-
-```yaml
-version: '3.8'
-services:
-  clickhouse:
-    image: 'clickhouse/clickhouse-server:23.8-alpine'
-    container_name: 'metabase-clickhouse-server'
-    ports:
-      - '8123:8123'
-      - '9000:9000'
-    ulimits:
-      nofile:
-        soft: 262144
-        hard: 262144
-  metabase:
-    image: 'my-metabase-with-clickhouse:v0.0.1'
-    container_name: 'metabase-with-clickhouse'
-    ports:
-      - '3000:3000'
-```
+- [Leiningen](https://leiningen.org/)
 
-## Using certificates
+## Steps
 
-In the "Advanced options", add the following to the "Additional JDBC connection string options" input:
+1. Clone and build metabase dependency jar.
 
-```
-sslrootcert=/path/to/ca.crt
-```
+   ```shell
+   git clone https://github.com/metabase/metabase
+   cd metabase
+   clojure -X:deps prep
+   cd modules/drivers
+   clojure -X:deps prep
+   cd ../..
+   ./bin/build.sh
+   ```
 
-where `/path/to/ca.crt` is the absolute path to the server CA on the Metabase host or Docker container (depends on your deployment).
+2. Clone metabase-proton-driver repo
 
-Make sure that you tick "Use a secure connection (SSL)" as well.
+   ```shell
+   cd modules/drivers
+   git clone https://github.com/timeplus-io/metabase-proton-driver
+   ```
 
-## Operations
+3. Prepare metabase dependencies
 
-The driver should work fine for many use cases. Please consider the following items when running a Metabase instance with this driver:
+   ```shell
+   cp ../../target/uberjar/metabase.jar metabase-proton-driver/
+   cd metabase-proton-driver
+   mkdir repo
+   lein pom
+   mvn deploy:deploy-file -Durl=file:repo -DgroupId=metabase-core -DartifactId=metabase-core -Dversion=1.40 -Dpackaging=jar -Dfile=metabase.jar
+   ```
 
-* Create a dedicated user for Metabase, whose profile has `readonly` set to 2.
-* Consider running the Metabase instance in the same time zone as your ClickHouse database; the more time zones involved the more issues.
-* Compare the results of the queries with the results returned by `clickhouse-client`.
-* Metabase is a good tool for organizing questions, dashboards etc. and to give non-technical users a good way to explore the data and share their results. The driver cannot support all the cool special features of ClickHouse, e.g. array functions. You are free to use native queries, of course.
+4. Build the jar
 
+   ```shell
+   LEIN_SNAPSHOTS_IN_RELEASE=true DEBUG=1 lein uberjar
+   ```
 
-## Known limitations
+5. Let's assume we download `metabase.jar` from the [Metabase jar](https://www.metabase.com/docs/latest/operations-guide/running-the-metabase-jar-file.html) to `~/metabase/` and we built the project above. Copy the built jar(proton.metabase-driver.jar) to the Metabase plugins directly and run Metabase from there!
 
-* As the underlying JDBC driver version does not support columns with `AggregateFunction` type, these columns are excluded from the table metadata and data browser result sets to prevent sync or data browsing errors.
-* If the past month/week/quarter/year filter over a DateTime64 column is not working as intended, this is likely due to a [type conversion issue](https://github.com/ClickHouse/ClickHouse/pull/50280). See [this report](https://github.com/ClickHouse/metabase-clickhouse-driver/issues/164) for more details. This issue was resolved as of ClickHouse 23.5.
+   ```shell
+   cd ~/metabase/
+   java -jar metabase.jar
+   ```
 
-## Contributing
+You should see a message on startup similar to:
 
-Check out our [contributing guide](./CONTRIBUTING.md).
+```
+2023-11-18 09:55:37,102 DEBUG plugins.lazy-loaded-driver :: Registering lazy loading driver :proton...
+2023-11-18 09:55:37,102 INFO driver.impl :: Registered driver :proton (parents: [:sql-jdbc]) 🚚
+```
diff --git a/deps.edn b/deps.edn
@@ -2,8 +2,11 @@
  ["src" "resources"]
 
  :deps
- {com.clickhouse/clickhouse-jdbc$http
+ {com.timeplus/proton-jdbc {:mvn/version "0.4.0"}
+  com.widdindustries/cljc.java-time {:mvn/version "0.1.21"}
+  com.clickhouse/clickhouse-jdbc$http
   {:mvn/version "0.4.6"
    :exclusions [com.clickhouse/clickhouse-cli-client$shaded
                 com.clickhouse/clickhouse-grpc-client$shaded]}
-  com.widdindustries/cljc.java-time {:mvn/version "0.1.21"}}}
+ }
+}  
diff --git a/pom.xml b/pom.xml
@@ -0,0 +1,97 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <groupId>metabase</groupId>
+  <artifactId>proton-driver</artifactId>
+  <packaging>jar</packaging>
+  <version>0.0.1</version>
+  <name>proton-driver</name>
+  <description>Timeplus Proton driver for Metabase</description>
+  <url>http://github.com/timeplus-io/metabase-proton-driver</url>
+  <licenses>
+    <license>
+      <name>Apache License 2.0</name>
+    </license>
+  </licenses>
+  <scm>
+    <url>https://github.com/timeplus-io/metabase-proton-driver</url>
+    <connection>scm:git:git://github.com/timeplus-io/metabase-proton-driver.git</connection>
+    <developerConnection>scm:git:ssh://git@github.com/timeplus-io/metabase-proton-driver.git</developerConnection>
+    <tag>9b989f07a7fad7e41074806dd4872bd3bc7f9cc8</tag>
+  </scm>
+  <build>
+    <sourceDirectory>src</sourceDirectory>
+    <testSourceDirectory>test</testSourceDirectory>
+    <resources>
+      <resource>
+        <directory>resources</directory>
+      </resource>
+    </resources>
+    <testResources>
+      <testResource>
+        <directory>resources</directory>
+      </testResource>
+    </testResources>
+    <directory>target</directory>
+    <outputDirectory>target/classes</outputDirectory>
+    <plugins/>
+  </build>
+  <repositories>
+    <repository>
+      <id>central</id>
+      <url>https://repo1.maven.org/maven2/</url>
+      <snapshots>
+        <enabled>false</enabled>
+      </snapshots>
+      <releases>
+        <enabled>true</enabled>
+      </releases>
+    </repository>
+    <repository>
+      <id>clojars</id>
+      <url>https://repo.clojars.org/</url>
+      <snapshots>
+        <enabled>true</enabled>
+      </snapshots>
+      <releases>
+        <enabled>true</enabled>
+      </releases>
+    </repository>
+    <repository>
+      <id>project</id>
+      <url>file:repo</url>
+      <snapshots>
+        <enabled>true</enabled>
+      </snapshots>
+      <releases>
+        <enabled>true</enabled>
+      </releases>
+    </repository>
+  </repositories>
+  <dependencyManagement>
+    <dependencies/>
+  </dependencyManagement>
+  <dependencies>
+    <dependency>
+      <groupId>com.timeplus</groupId>
+      <artifactId>proton-jdbc</artifactId>
+      <version>0.4.0</version>
+    </dependency>
+    <dependency>
+      <groupId>clojure.java-time</groupId>
+      <artifactId>clojure.java-time</artifactId>
+      <version>0.3.2</version>
+    </dependency>
+    <dependency>
+      <groupId>metabase-core</groupId>
+      <artifactId>metabase-core</artifactId>
+      <version>1.40</version>
+      <scope>provided</scope>
+    </dependency>
+  </dependencies>
+</project>
+
+<!-- This file was autogenerated by Leiningen.
+  Please do not edit it directly; instead edit project.clj and regenerate it.
+  It should not be considered canonical data. For more information see
+  https://codeberg.org/leiningen/leiningen -->
diff --git a/project.clj b/project.clj
@@ -0,0 +1,28 @@
+(defproject metabase/proton-driver "0.0.1"
+  :description "Timeplus Proton driver for Metabase"
+  :license "Apache License 2.0"
+  :url "http://github.com/timeplus-io/metabase-proton-driver"
+  :min-lein-version "2.5.0"
+
+  :dependencies
+  [[com.timeplus/proton-jdbc "0.4.0"]
+   [clojure.java-time "0.3.2"]
+  ]
+
+  :repositories [
+                ["project" "file:repo"]]
+  :aliases
+  {"test"       ["with-profile" "test"]}
+
+
+  :profiles
+  {:provided
+   {:dependencies [[metabase-core "1.40"]]}
+
+   :uberjar
+   {:auto-clean    true
+    :aot           :all
+    :javac-options ["-target" "1.8", "-source" "1.8"]
+    :target-path   "target/%s"
+    :jvm-opts      ["-Dclojure.compiler.direct-linking=true"]
+    :uberjar-name  "proton.metabase-driver.jar"}})