Skip to content

Commit

Permalink
Merge pull request #27 from sranka/feat/modernize-libs
Browse files Browse the repository at this point in the history
feat: modernize 3rd party libraries
  • Loading branch information
sranka authored Feb 13, 2025
2 parents bf4aecb + 38932fc commit b673a96
Show file tree
Hide file tree
Showing 15 changed files with 89 additions and 42 deletions.
21 changes: 10 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,9 @@ The tool ignores missing tables and columns when importing the data.
## Quick Start
1. Java must be installed, 1.8 or newer
2. Install `jdbcimage` tool
* download the latest [release](https://github.com/sranka/jdbcimage/releases) as `jdbcimage${version}.tar.gz` or `jdbcimage${version}.zip`, or build it from sources
* mvn install, the same files appear in the `target` directory
* `tar xvf jdbcimage${version}.tar.gz` or `unzip jdbcimage${version}.zip` in a directory of your choice,
* the examples below assume that the directory is in your `PATH` environment variable
* download the latest [release](https://github.com/sranka/jdbcimage/releases) as `jdbcimage-${version}.tar.gz` or `jdbcimage-${version}.zip`, or build it from sources
* `tar xvf jdbcimage-${version}.tar.gz` or `unzip jdbcimage-${version}.zip` in a directory of your choice,
* the examples below assume that the extracted directory `jdbcimage-${version}` is in your `PATH` environment variable
* if you are using oracle database, copy its JDBC drivers to the lib directory
3. Know how to connect to your database
* *url* - JDBC connection URL, observe examples below
Expand Down Expand Up @@ -43,24 +42,24 @@ The tool ignores missing tables and columns when importing the data.
The key principles are:
1. More threads are used to speed up data export/import, OOTB all (client)
machine processors should be used. See `tool_concurrency` parameter in the scripts.
1. The lowest transaction isolation level is used to make database export/import faster.
1. Database metadata are used to export/import all user+schema tables to/from a zip file with entries
2. The lowest transaction isolation level is used to make database export/import faster.
3. Database metadata are used to export/import all user+schema tables to/from a zip file with entries
per exported/imported table.
1. Table and column names are always case-insensitive internally, error is reported when there are more tables
4. Table and column names are always case-insensitive internally, error is reported when there are more tables
with the same case-insensitive name.
1. Concurrent execution requires an extra setup/teardown instructions during data import.
5. Concurrent execution requires an extra setup/teardown instructions during data import.
These vary between database types, but they always include disabling/enabling foreign
key constraints, see database classes defined in the [main package](https://github.com/sranka/jdbcimage/tree/master/src/main/java/io/github/sranka/jdbcimage/main) for more details.
* All triggers are disabled on Oracle before data import and then enabled after data import.
* Oracle sequences, when used, are out of scope and usually have to be reset manually after data import.
* All foreign key constraints are dropped by the tool on Postgress before importing the data, but a table jdbcimage_create_constraints is created with rows that are used to recreate them after data import.
* Identity column sequences are reset to the lowest value after data import on Postgres.
1. Streams of data are used for both export and import to have the lowest memory footprint, typically 256M of heap
6. Streams of data are used for both export and import to have the lowest memory footprint, typically 256M of heap
memory is good enough. BLOB, CLOBs and lengthy columns still might require more heap memory depending on data
and JDBC driver in use, so you also might have to increase java heap memory and/or lower batch size used during
data import. There are parameters in the scripts to do so.
1. The result files are binary encoded using Kryo and zipped to be small on file system.
1. The scripts accept several properties as arguments supplied as -property=value pairs
7. The result files are binary encoded using Kryo and zipped to be small on file system.
8. The scripts accept several properties as arguments supplied as -property=value pairs
* -url=jdbc:mariadb://localhost:3306/qa - JDBC connection string
* -user=user
* -password=password
Expand Down
2 changes: 1 addition & 1 deletion jdbcimage
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,4 @@ done
INSTALLDIR=`dirname "$PRG"`

echo "Starting Java with: $JAVA_OPTS"
java $JAVA_OPTS -classpath "$INSTALLDIR/target/jdbcimage.jar:$INSTALLDIR/lib/*" io.github.sranka.jdbcimage.main.JdbcImageMain "$@"
java $JAVA_OPTS -classpath "$INSTALLDIR/jdbcimage.jar:$INSTALLDIR/lib/*" io.github.sranka.jdbcimage.main.JdbcImageMain "$@"
2 changes: 1 addition & 1 deletion jdbcimage.bat
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@setlocal
@echo off
set JAVA_OPTS=-Xmx1024m
java %JAVA_OPTS% -classpath "%~dp0target/jdbcimage.jar;%~dp0lib/*" io.github.sranka.jdbcimage.main.JdbcImageMain %*
java %JAVA_OPTS% -classpath "%~dp0jdbcimage.jar;%~dp0lib/*" io.github.sranka.jdbcimage.main.JdbcImageMain %*
Binary file removed lib/mariadb-java-client-3.5.2.jar
Binary file not shown.
Binary file removed lib/mssql-jdbc-12.8.1.jre8.jar
Binary file not shown.
Binary file removed lib/postgresql-42.7.5.jar
Binary file not shown.
52 changes: 47 additions & 5 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<groupId>com.github.sranka</groupId>
<groupId>io.github.sranka</groupId>
<artifactId>jdbcimage</artifactId>
<version>0.9.2-SNAPSHOT</version>
<packaging>jar</packaging>
Expand All @@ -14,6 +14,7 @@
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<version.jdk>1.8</version.jdk>
<app.main.class>io.github.sranka.jdbcimage.main.JdbcImageMain</app.main.class>
</properties>

<licenses>
Expand All @@ -34,7 +35,7 @@
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-dbcp2</artifactId>
<version>2.9.0</version>
<version>2.13.0</version>
</dependency>
<!-- logging -->
<dependency>
Expand All @@ -49,22 +50,41 @@
<version>4.13.2</version>
<scope>test</scope>
</dependency>
<!-- JDBC drivers -->
<dependency>
<groupId>org.mariadb.jdbc</groupId>
<artifactId>mariadb-java-client</artifactId>
<version>3.5.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.7.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>12.8.1.jre8</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<finalName>jdbcimage-${project.version}</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version> <!-- only because of maven warnings -->
<version>3.4.2</version>
<configuration>
<finalName>jdbcimage</finalName>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>license-maven-plugin</artifactId>
<version>1.16</version>
<version>2.5.0</version>
<executions>
<execution>
<id>add-third-party</id>
Expand All @@ -88,7 +108,20 @@
<artifactId>maven-shade-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<artifactSet>
<excludes>
<exclude>org.mariadb.jdbc:mariadb-java-client:*</exclude>
<exclude>org.postgresql:postgresql:*</exclude>
<exclude>com.microsoft.sqlserver:mssql-jdbc:*</exclude>
</excludes>
</artifactSet>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>${app.main.class}</Main-Class>
<X-Compile-Source-JDK>${version.jdk}</X-Compile-Source-JDK>
</manifestEntries>
</transformer>
<!-- merge licenses -->
<transformer
implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer">
Expand All @@ -97,6 +130,15 @@
implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer">
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/MANIFEST.MF</exclude>
<exclude>META-INF/versions/9/module-info.class</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
Expand Down
5 changes: 2 additions & 3 deletions src/main/assembly/assembly.xml
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
<id>assembly</id>
<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.2.0 https://maven.apache.org/xsd/assembly-2.2.0.xsd"> <id>assembly</id>
<formats>
<format>tar.gz</format>
<format>zip</format>
Expand Down
34 changes: 24 additions & 10 deletions src/main/assembly/component.xml
Original file line number Diff line number Diff line change
@@ -1,5 +1,27 @@
<component xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/component/1.1.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/component/1.1.2 http://maven.apache.org/xsd/component-1.1.2.xsd">
<?xml version="1.0" encoding="utf-8"?>
<component xmlns="http://maven.apache.org/ASSEMBLY-COMPONENT/2.2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/ASSEMBLY-COMPONENT/2.2.0 https://maven.apache.org/xsd/assembly-component-2.2.0.xsd">
<dependencySets>
<dependencySet>
<outputDirectory>lib</outputDirectory>
<includes>
<include>org.mariadb.jdbc:mariadb-java-client:*</include>
<include>org.postgresql:postgresql:*</include>
<include>com.microsoft.sqlserver:mssql-jdbc:*</include>
</includes>
<scope>test</scope>
<useStrictFiltering>true</useStrictFiltering>
</dependencySet>
<dependencySet>
<useProjectArtifact>true</useProjectArtifact>
<outputDirectory/>
<includes>
<include>io.github.sranka:jdbcimage:jar</include>
</includes>
<outputFileNameMapping>jdbcimage.jar</outputFileNameMapping>
<useStrictFiltering>true</useStrictFiltering>
</dependencySet>
</dependencySets>
<fileSets>
<fileSet>
<directory/>
Expand All @@ -25,7 +47,6 @@
<directory>lib</directory>
<outputDirectory>lib</outputDirectory>
<includes>
<include>*.jar</include>
<include>THIRD-PARTY.txt</include>
</includes>
<fileMode>644</fileMode>
Expand All @@ -39,12 +60,5 @@
</includes>
<fileMode>644</fileMode>
</fileSet>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>target</outputDirectory>
<includes>
<include>jdbcimage.jar</include>
</includes>
</fileSet>
</fileSets>
</component>
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ public int read() throws IOException {
return retVal;
}

@SuppressWarnings("NullableProblems")
@Override
public int read(byte[] b, int off, int len) throws IOException {
if (finished) return -1;
Expand Down
2 changes: 2 additions & 0 deletions src/main/java/io/github/sranka/jdbcimage/ChunkedReader.java
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ public long length() {
* @param count count of position to move
* @return false if an end of stream was reached
*/
@SuppressWarnings("DuplicatedCode")
private boolean forward(int count) {
while (count >= (currentChunk.length - pos)) {
count -= currentChunk.length - pos;
Expand All @@ -72,6 +73,7 @@ public int read() throws IOException {
return retVal;
}

@SuppressWarnings({"DuplicatedCode", "NullableProblems"})
@Override
public int read(char[] b, int off, int len) throws IOException {
if (finished) return -1;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,6 @@ public interface ResultProducer {
/**
* Called to inform about finished processing.
*/
@SuppressWarnings("EmptyMethod")
void close();
}
5 changes: 0 additions & 5 deletions src/main/java/io/github/sranka/jdbcimage/main/MariaDB.java
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,4 @@ public void modifyIndexes(boolean enable) {
public boolean canCreateBlobs() {
return false;
}

@Override
public void importStarted() {
super.importStarted();
}
}
5 changes: 0 additions & 5 deletions src/main/java/io/github/sranka/jdbcimage/main/Oracle.java
Original file line number Diff line number Diff line change
Expand Up @@ -198,11 +198,6 @@ public void modifyIndexes(boolean enable) throws SQLException {
.collect(Collectors.toList()));
}

@Override
public void importStarted() {
super.importStarted();
}

public static class Types {

public static final int BINARY_DOUBLE = 101;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
/**
* DB facade for PostgreSQL database.
*/
@SuppressWarnings("SpellCheckingInspection")
public class PostgreSQL extends DBFacade {
public static final String STATE_TABLE_NAME = "jdbcimage_create_constraints";
public static final String STATE_TABLE_DDL = "CREATE TABLE " + STATE_TABLE_NAME + "( tableName varchar(64),constraintName varchar(64),sql varchar(512))";
Expand Down

0 comments on commit b673a96

Please sign in to comment.