diff --git a/README.md b/README.md index a28884a..3f17065 100644 --- a/README.md +++ b/README.md @@ -2,164 +2,255 @@ [![GoDoc Widget]][GoDoc] +`go‑dbkit` is a Go library designed to simplify and streamline working with SQL databases. +It provides a solid foundation for connection management, instrumentation, error retry mechanisms, and transaction handling. +Additionally, `go‑dbkit` offers a suite of specialized sub‑packages that address common database challenges—such as [distributed locking](./distrlock), [schema migrations](./migrate), and query builder utilities - making it a one‑stop solution for applications that needs to interact with multiple SQL databases. + +## Features +- **Transaction Management**: Execute functions within transactions that automatically commit on success or roll back on error. The transaction runner abstracts the boilerplate, ensuring cleaner and more reliable code. +- **Retryable Queries**: Built‑in support for detecting and automatically retrying transient errors (e.g., deadlocks, lock timeouts) across various databases. +- **Distributed Locking**: Implement SQL‑based distributed locks to coordinate exclusive access to shared resources across multiple processes. +- **Database Migrations**: Seamlessly manage schema changes with support for both embedded SQL migrations (using Go’s embed package) and programmatic migration definitions. +- **Query Builder Utilities**: Enhance your query‐building experience with utilities for popular libraries: + * [dbrutil](./dbrutil): Simplifies working with the [dbr query builder](https://github.com/gocraft/dbr), adding instrumentation (Prometheus metrics, slow query logging) and transaction support. + * [goquutil](./goquutil): Provides helper routines for the [goqu query builder](https://github.com/doug-martin/goqu) (currently, no dedicated README exists—refer to the source for usage). + +## Packages Overview +- Root `go‑dbkit` package provides configuration management, DSN generation, and the foundational retryable query functionality used across the library. +- [dbrutil](./dbrutil) offers utilities for the dbr query builder, including: + * Instrumented connection opening with Prometheus metrics. + * Automatic slow query logging based on configurable thresholds. + * A transaction runner that simplifies commit/rollback logic. + Read more in [dbrutil/README.md](./dbrutil/README.md). +- [distrlock](./distrlock) implements a lightweight, SQL‑based distributed locking mechanism that ensures exclusive execution of critical sections across concurrent services. + Read more in [distrlock/README.md](./distrlock/README.md). +- [migrate](./migrate): + Manage your database schema changes effortlessly with support for both embedded SQL files and programmatic migrations. + Read more in [migrate/README.md](./migration/README.md). +- [goquutil](./goquutil) provides helper functions for working with the goqu query builder, streamlining common operations. (This package does not have its own README yet, so please refer to the source code for more details.) +- RDBMS‑Specific dedicated sub‑packages are provided for various relational databases: + * [mysql](./mysql) includes DSN generation, retryable error handling, and other MySQL‑specific utilities. + * [sqlite](./sqlite) contains helpers to integrate SQLite seamlessly into your projects. + * [postgres](./postgres) & [pgx](./pgx) offers tools and error handling improvements for PostgreSQL using both the lib/pq and pgx drivers. + * [mssql](./mssql) provides MSSQL‑specific error handling, including registration of retryable functions for deadlocks and related transient errors. + Each of these packages registers its own retryable function in the init() block, ensuring that transient errors (like deadlocks or cached plan invalidations) are automatically retried.o + ## Installation ``` go get -u github.com/acronis/go-dbkit ``` -## Structure - -### `/` -Package `dbkit` provides helpers for working with different SQL databases (MySQL, PostgreSQL, SQLite and MSSQL). +## Usage -### `/distrlock` -Package distrlock contains DML (distributed lock manager) implementation (now DMLs based on MySQL and PostgreSQL are supported). -Now only manager that uses SQL database (PostgreSQL and MySQL are currently supported) is available. -Other implementations (for example, based on Redis) will probably be implemented in the future. +### Basic Example -### `/migrate` -Package migrate provides functionality for applying database migrations. - -### `/mssql` -Package mssql provides helpers for working with MSSQL. -Should be imported explicitly. -To register mssql as retryable func use side effect import like so: +Below is a simple example that demonstrates how to register a retryable function for a MySQL database connection and execute a query within a transaction with a custom retry policy on transient errors. ```go -import _ "github.com/acronis/go-dbkit/mssql" -``` +package main -### `/mysql` -Package mysql provides helpers for working with MySQL. -Should be imported explicitly. -To register mysql as retryable func use side effect import like so: +import ( + "context" + "database/sql" + "log" + "os" + "time" -```go -import _ "github.com/acronis/go-dbkit/mysql" -``` + "github.com/acronis/go-appkit/retry" -### `/pgx` -Package pgx provides helpers for working with Postgres via `jackc/pgx` driver. -Should be imported explicitly. -To register postgres as retryable func use side effect import like so: + "github.com/acronis/go-dbkit" -```go -import _ "github.com/acronis/go-dbkit/pgx" -``` + // Import the `mysql` package for registering the retryable function for MySQL transient errors (like deadlocks). + _ "github.com/acronis/go-dbkit/mysql" +) -### `/postgres` -Package postgres provides helpers for working with Postgres via `lib/pq` driver. -Should be imported explicitly. -To register postgres as retryable func use side effect import like so: +func main() { + // Configure the database using the dbkit.Config struct. + // In this example, we're using MySQL. Adjust Dialect and config fields for your target DB. + cfg := &dbkit.Config{ + Dialect: dbkit.DialectMySQL, + MySQL: dbkit.MySQLConfig{ + Host: os.Getenv("MYSQL_HOST"), + Port: 3306, + User: os.Getenv("MYSQL_USER"), + Password: os.Getenv("MYSQL_PASSWORD"), + Database: os.Getenv("MYSQL_DATABASE"), + }, + MaxOpenConns: 16, + MaxIdleConns: 8, + } -```go -import _ "github.com/acronis/go-dbkit/postgres" + // Open the database connection. + // The 2nd parameter is a boolean that indicates whether to ping the database. + db, err := dbkit.Open(cfg, true) + if err != nil { + log.Fatalf("failed to open database: %v", err) + } + defer db.Close() + + // Execute a transaction with a custom retry policy (exponential backoff with 3 retries, starting from 10ms). + retryPolicy := retry.NewConstantBackoffPolicy(10*time.Millisecond, 3) + if err = dbkit.DoInTx(context.Background(), db, func(tx *sql.Tx) error { + // Execute your transactional operations here. + // Example: _, err := tx.Exec("UPDATE users SET last_login = ? WHERE id = ?", time.Now(), 1) + return nil + }, dbkit.WithRetryPolicy(retryPolicy)); err != nil { + log.Fatal(err) + } +} ``` -### `/sqlite` -Package sqlite provides helpers for working with SQLite. -Should be imported explicitly. -To register sqlite as retryable func use side effect import like so: +### `dbrutil` Usage Example + +The following basic example demonstrates how to use `dbrutil` to open a database connection with instrumentation, +and execute queries with an automatic slow query logging and Prometheus metrics collection within transaction. ```go -import _ "github.com/acronis/go-dbkit/sqlite" -``` +package main + +import ( + "context" + "database/sql" + "errors" + "fmt" + stdlog "log" + "net/http" + "os" + "time" + + "github.com/acronis/go-appkit/log" + "github.com/gocraft/dbr/v2" + + "github.com/acronis/go-dbkit" + "github.com/acronis/go-dbkit/dbrutil" +) -### `/dbrutil` -Package dbrutil provides utilities and helpers for [dbr](https://github.com/gocraft/dbr) query builder. +func main() { + logger, loggerClose := log.NewLogger(&log.Config{Output: log.OutputStderr, Level: log.LevelInfo}) + defer loggerClose() + + // Create a Prometheus metrics collector. + promMetrics := dbkit.NewPrometheusMetrics() + promMetrics.MustRegister() + defer promMetrics.Unregister() + + // Open the database connection with instrumentation. + // Instrumentation includes collecting metrics about SQL queries and logging slow queries. + eventReceiver := dbrutil.NewCompositeReceiver([]dbr.EventReceiver{ + dbrutil.NewQueryMetricsEventReceiver(promMetrics, queryAnnotationPrefix), + dbrutil.NewSlowQueryLogEventReceiver(logger, 100*time.Millisecond, queryAnnotationPrefix), + }) + conn, err := openDB(eventReceiver) + if err != nil { + stdlog.Fatal(err) + } + defer conn.Close() -### `/goquutil` -Package goquutil provides auxiliary routines for working with [goqu](https://github.com/doug-martin/goqu) query builder. + txRunner := dbrutil.NewTxRunner(conn, &sql.TxOptions{Isolation: sql.LevelReadCommitted}, nil) + + // Execute function in a transaction. + // The transaction will be automatically committed if the function returns nil, otherwise it will be rolled back. + if dbErr := txRunner.DoInTx(context.Background(), func(tx dbr.SessionRunner) error { + var result int + return tx.Select("SLEEP(1)"). + Comment(annotateQuery("long_operation")). // Annotate the query for Prometheus metrics and slow query log. + LoadOne(&result) + }); dbErr != nil { + stdlog.Fatal(dbErr) + } -## Examples + // The following log message will be printed: + // {"level":"warn","time":"2025-02-14T16:29:55.429257+02:00","msg":"slow SQL query","pid":14030,"annotation":"query:long_operation","duration_ms":1007} -### Open database connection using the `dbrutil` package + // Prometheus metrics will be collected: + // db_query_duration_seconds_bucket{query="query:long_operation",le="2.5"} 1 + // db_query_duration_seconds_sum{query="query:long_operation"} 1.004573875 + // db_query_duration_seconds_count{query="query:long_operation"} 1 +} -```go -func main() { - // Create a new database configuration - cfg := &db.Config{ - Driver: db.DialectMySQL, - Host: "localhost", - Port: 3306, - Username: "your-username", - Password: "your-password", - Database: "your-database", - } +const queryAnnotationPrefix = "query:" - // Open a connection to the database - conn, err := dbrutil.Open(cfg, true, nil) - if err != nil { - fmt.Println("Failed to open database connection:", err) - return +func annotateQuery(queryName string) string { + return queryAnnotationPrefix + queryName +} + +func openDB(eventReceiver dbr.EventReceiver) (*dbr.Connection, error) { + cfg := &dbkit.Config{ + Dialect: dbkit.DialectMySQL, + MySQL: dbkit.MySQLConfig{ + Host: os.Getenv("MYSQL_HOST"), + Port: 3306, + User: os.Getenv("MYSQL_USER"), + Password: os.Getenv("MYSQL_PASSWORD"), + Database: os.Getenv("MYSQL_DATABASE"), + }, } - defer conn.Close() - // Create a new transaction runner - runner := dbrutil.NewTxRunner(conn, &sql.TxOptions{}, nil) - - // Execute code inside a transaction - err = runner.DoInTx(context.Background(), func(runner dbr.SessionRunner) error { - // Perform database operations using the runner - _, err := runner.InsertInto("users"). - Columns("name", "email"). - Values("Bob", "bob@example.com"). - Exec() - if err != nil { - return err - } - - // Return nil to commit the transaction - return nil - }) + // Open database with instrumentation based on the provided event receiver (see github.com/gocraft/dbr doc for details). + // Opening includes configuring the max open/idle connections and their lifetime and pinging the database. + conn, err := dbrutil.Open(cfg, true, eventReceiver) if err != nil { - fmt.Println("Failed to execute transaction:", err) - return + return nil, fmt.Errorf("open database: %w", err) } + return conn, nil } ``` -### Usage of `distrlock` package +More examples and detailed usage instructions can be found in the `dbrutil` package [README](./dbrutil/README.md). + +### `distrlock` Usage Example + +The following basic example demonstrates how to use `distrlock` to ensure exclusive execution of a critical section of code. ```go -// Create a new DBManager with the MySQL dialect -dbManager, err := distrlock.NewDBManager(db.DialectMySQL) -if err != nil { - log.Fatal(err) -} +package main -// Open a connection to the MySQL database -dbConn, err := sql.Open("mysql", "user:password@tcp(localhost:3306)/database") -if err != nil { - log.Fatal(err) -} -defer dbConn.Close() +import ( + "context" + "database/sql" + "log" + "os" + "time" -// Create a new lock -lock, err := dbManager.NewLock(context.Background(), dbConn, "my_lock") -if err != nil { - log.Fatal(err) -} + "github.com/acronis/go-dbkit" + "github.com/acronis/go-dbkit/distrlock" +) -// Acquire the lock -err = lock.Acquire(context.Background(), dbConn, 5*time.Second) -if err != nil { - log.Fatal(err) -} +func main() { + // Setup database connection + db, err := sql.Open("mysql", os.Getenv("MYSQL_DSN")) + if err != nil { + log.Fatal(err) + } + defer db.Close() -// Do some work while holding the lock -fmt.Println("Lock acquired, doing some work...") + ctx := context.Background() -// Release the lock -err = lock.Release(context.Background(), dbConn) -if err != nil { - log.Fatal(err) -} + // Create "distributed_locks" table for locks. + createTableSQL, err := distrlock.CreateTableSQL(dbkit.DialectMySQL) + if err != nil { + log.Fatal(err) + } + _, err = db.ExecContext(ctx, createTableSQL) + if err != nil { + log.Fatal(err) + } -fmt.Println("Lock released") + // Do some work exclusively. + const lockKey = "test-lock-key-1" // Unique key that will be used to ensure exclusive execution among multiple instances + err = distrlock.DoExclusively(ctx, db, dbkit.DialectMySQL, lockKey, func(ctx context.Context) error { + time.Sleep(10 * time.Second) // Simulate work. + return nil + }) + if err != nil { + log.Fatal(err) + } +} ``` +More examples and detailed usage instructions can be found in the `distrlock` package [README](./distrlock/README.md). + ## License Copyright © 2024 Acronis International GmbH. diff --git a/db.go b/db.go index 0579520..91324d9 100644 --- a/db.go +++ b/db.go @@ -11,31 +11,72 @@ import ( "database/sql" "fmt" "time" + + "github.com/acronis/go-appkit/retry" ) +// Open opens a new database connection using the provided configuration. +// If ping is true, it will check the connection by sending a ping to the database. +func Open(cfg *Config, ping bool) (*sql.DB, error) { + driver, dsn := cfg.DriverNameAndDSN() + db, err := sql.Open(driver, dsn) + if err != nil { + return nil, err + } + return db, InitOpenedDB(db, cfg, ping) +} + // InitOpenedDB initializes early opened *sql.DB instance. func InitOpenedDB(db *sql.DB, cfg *Config, ping bool) error { db.SetMaxOpenConns(cfg.MaxOpenConns) db.SetMaxIdleConns(cfg.MaxIdleConns) db.SetConnMaxLifetime(time.Duration(cfg.ConnMaxLifetime)) - if ping { if err := db.Ping(); err != nil { return err } } - return nil } +type doInTxOptions struct { + txOpts *sql.TxOptions + retryPolicy retry.Policy +} + +// DoInTxOption is a functional option for DoInTx. +type DoInTxOption func(*doInTxOptions) + +// WithTxOptions sets transaction options for DoInTx. +func WithTxOptions(txOpts *sql.TxOptions) DoInTxOption { + return func(opts *doInTxOptions) { + opts.txOpts = txOpts + } +} + +// WithRetryPolicy sets retry policy for DoInTx. +func WithRetryPolicy(policy retry.Policy) DoInTxOption { + return func(opts *doInTxOptions) { + opts.retryPolicy = policy + } +} + // DoInTx begins a new transaction, calls passed function and do commit or rollback // depending on whether the function returns an error or not. -func DoInTx(ctx context.Context, dbConn *sql.DB, fn func(tx *sql.Tx) error) (err error) { - return DoInTxWithOpts(ctx, dbConn, nil, fn) +func DoInTx(ctx context.Context, dbConn *sql.DB, fn func(tx *sql.Tx) error, options ...DoInTxOption) (err error) { + var opts doInTxOptions + for _, opt := range options { + opt(&opts) + } + if opts.retryPolicy == nil { + return doInTx(ctx, dbConn, fn, opts.txOpts) + } + return retry.DoWithRetry(ctx, opts.retryPolicy, GetIsRetryable(dbConn.Driver()), nil, func(ctx context.Context) error { + return doInTx(ctx, dbConn, fn, opts.txOpts) + }) } -// DoInTxWithOpts is a bit more configurable version of DoInTx that allows passing tx options. -func DoInTxWithOpts(ctx context.Context, dbConn *sql.DB, txOpts *sql.TxOptions, fn func(tx *sql.Tx) error) (err error) { +func doInTx(ctx context.Context, dbConn *sql.DB, fn func(tx *sql.Tx) error, txOpts *sql.TxOptions) (err error) { var tx *sql.Tx if tx, err = dbConn.BeginTx(ctx, txOpts); err != nil { return fmt.Errorf("begin tx: %w", err) @@ -53,6 +94,5 @@ func DoInTxWithOpts(ctx context.Context, dbConn *sql.DB, txOpts *sql.TxOptions, err = fmt.Errorf("commit tx: %w", err) } }() - return fn(tx) } diff --git a/db_test.go b/db_test.go index 382aaf0..e477b6c 100644 --- a/db_test.go +++ b/db_test.go @@ -9,106 +9,262 @@ package dbkit import ( "context" "database/sql" + "errors" "fmt" - "io" "testing" + "time" "github.com/DATA-DOG/go-sqlmock" + "github.com/acronis/go-appkit/config" + "github.com/acronis/go-appkit/retry" + _ "github.com/mattn/go-sqlite3" "github.com/stretchr/testify/require" ) +func TestOpen(t *testing.T) { + tests := []struct { + name string + cfg *Config + ping bool + wantErr bool + }{ + { + name: "successful open with ping", + cfg: &Config{ + Dialect: DialectSQLite, + SQLite: SQLiteConfig{Path: ":memory:"}, + MaxOpenConns: 10, + MaxIdleConns: 5, + ConnMaxLifetime: config.TimeDuration(time.Minute * 10), + }, + ping: true, + wantErr: false, + }, + { + name: "error on open", + cfg: &Config{ + Dialect: Dialect("unknown"), + SQLite: SQLiteConfig{Path: ":memory:"}, + MaxOpenConns: 10, + MaxIdleConns: 5, + ConnMaxLifetime: config.TimeDuration(time.Minute * 10), + }, + ping: false, + wantErr: true, + }, + { + name: "error on ping", + cfg: &Config{ + Dialect: DialectSQLite, + SQLite: SQLiteConfig{Path: "internal"}, // directory is not a valid path + MaxOpenConns: 10, + MaxIdleConns: 5, + ConnMaxLifetime: config.TimeDuration(time.Minute * 10), + }, + ping: true, + wantErr: true, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + dbConn, err := Open(tt.cfg, tt.ping) + if tt.wantErr { + require.Error(t, err) + } else { + require.NoError(t, err) + require.NotNil(t, dbConn) + } + }) + } +} + func TestDoInTx(t *testing.T) { tests := []struct { - Name string - InitMock func(m sqlmock.Sqlmock) - Fn func(tx *sql.Tx) error - WantErr error - WantPanicErr error + name string + initMock func(m sqlmock.Sqlmock) + fn func(tx *sql.Tx) error + wantErr error + wantPanicErr error }{ { - Name: "success", - InitMock: func(m sqlmock.Sqlmock) { + name: "success", + initMock: func(m sqlmock.Sqlmock) { m.ExpectBegin() m.ExpectCommit() }, - Fn: func(tx *sql.Tx) error { + fn: func(tx *sql.Tx) error { return nil }, }, { - Name: "error on begin", - InitMock: func(m sqlmock.Sqlmock) { + name: "error on begin", + initMock: func(m sqlmock.Sqlmock) { m.ExpectBegin().WillReturnError(fmt.Errorf("begin error")) }, - Fn: func(tx *sql.Tx) error { + fn: func(tx *sql.Tx) error { return nil }, - WantErr: fmt.Errorf("begin tx: begin error"), + wantErr: fmt.Errorf("begin tx: begin error"), }, { - Name: "error on commit", - InitMock: func(m sqlmock.Sqlmock) { + name: "error on commit", + initMock: func(m sqlmock.Sqlmock) { m.ExpectBegin() m.ExpectCommit().WillReturnError(fmt.Errorf("commit error")) }, - Fn: func(tx *sql.Tx) error { + fn: func(tx *sql.Tx) error { return nil }, - WantErr: fmt.Errorf("commit tx: commit error"), + wantErr: fmt.Errorf("commit tx: commit error"), }, { - Name: "error in func", - InitMock: func(m sqlmock.Sqlmock) { + name: "error in func", + initMock: func(m sqlmock.Sqlmock) { m.ExpectBegin() m.ExpectRollback() }, - Fn: func(tx *sql.Tx) error { + fn: func(tx *sql.Tx) error { return fmt.Errorf("fn error") }, - WantErr: fmt.Errorf("fn error"), + wantErr: fmt.Errorf("fn error"), }, { - Name: "panic in func", - InitMock: func(m sqlmock.Sqlmock) { + name: "panic in func", + initMock: func(m sqlmock.Sqlmock) { m.ExpectBegin() m.ExpectRollback() }, - Fn: func(tx *sql.Tx) error { + fn: func(tx *sql.Tx) error { panic(fmt.Errorf("panic")) }, - WantPanicErr: fmt.Errorf("panic"), + wantPanicErr: fmt.Errorf("panic"), }, } - for i := range tests { - tt := tests[i] - t.Run(tt.Name, func(t *testing.T) { + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { db, mock, err := sqlmock.New() require.NoError(t, err) defer func() { - requireNoErrOnClose(t, db) require.NoError(t, mock.ExpectationsWereMet()) }() - tt.InitMock(mock) - mock.ExpectClose() + tt.initMock(mock) - if tt.WantPanicErr != nil { - require.PanicsWithError(t, tt.WantPanicErr.Error(), func() { - _ = DoInTx(context.Background(), db, tt.Fn) + if tt.wantPanicErr != nil { + require.PanicsWithError(t, tt.wantPanicErr.Error(), func() { + _ = DoInTx(context.Background(), db, tt.fn) }) return } - err = DoInTx(context.Background(), db, tt.Fn) - if tt.WantErr == nil { + err = DoInTx(context.Background(), db, tt.fn) + if tt.wantErr == nil { require.NoError(t, err) return } - require.EqualError(t, err, tt.WantErr.Error()) + require.EqualError(t, err, tt.wantErr.Error()) }) } } -func requireNoErrOnClose(t *testing.T, closer io.Closer) { - t.Helper() - require.NoError(t, closer.Close()) +func TestDoInTxWithRetryPolicy(t *testing.T) { + retryableError := errors.New("retryable error") + + retryPolicy := retry.NewConstantBackoffPolicy(time.Millisecond*50, 3) + + tests := []struct { + name string + initMock func(m sqlmock.Sqlmock) + fnProvider func() func(tx *sql.Tx) error + wantErr error + }{ + { + name: "success, no retry attempts", + initMock: func(m sqlmock.Sqlmock) { + m.ExpectBegin() + m.ExpectQuery("SELECT 1").WillReturnRows(sqlmock.NewRows([]string{"1"}).AddRow(1)) + m.ExpectCommit() + }, + fnProvider: func() func(tx *sql.Tx) error { + return func(tx *sql.Tx) error { + _, queryErr := tx.Query("SELECT 1") + return queryErr + } + }, + }, + { + name: "success after retry", + initMock: func(m sqlmock.Sqlmock) { + m.ExpectBegin() + m.ExpectRollback() + m.ExpectBegin() + m.ExpectQuery("SELECT 1").WillReturnRows(sqlmock.NewRows([]string{"1"}).AddRow(1)) + m.ExpectCommit() + }, + fnProvider: func() func(tx *sql.Tx) error { + var attempts int + return func(tx *sql.Tx) error { + attempts++ + if attempts < 2 { + return retryableError + } + _, queryErr := tx.Query("SELECT 1") + return queryErr + } + }, + }, + { + name: "fail, no retry on non-retryable error", + initMock: func(m sqlmock.Sqlmock) { + m.ExpectBegin() + m.ExpectRollback() + }, + fnProvider: func() func(tx *sql.Tx) error { + return func(tx *sql.Tx) error { + return fmt.Errorf("non-retryable error") + } + }, + wantErr: fmt.Errorf("non-retryable error"), + }, + { + name: "fail, max retry attempts exceeded", + initMock: func(m sqlmock.Sqlmock) { + // 4 attempts: 1 initial + 3 retries + m.ExpectBegin() + m.ExpectRollback() + m.ExpectBegin() + m.ExpectRollback() + m.ExpectBegin() + m.ExpectRollback() + m.ExpectBegin() + m.ExpectRollback() + }, + fnProvider: func() func(tx *sql.Tx) error { + return func(tx *sql.Tx) error { + return retryableError + } + }, + wantErr: retryableError, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + db, mock, err := sqlmock.New() + require.NoError(t, err) + + UnregisterAllIsRetryableFuncs(db.Driver()) + RegisterIsRetryableFunc(db.Driver(), func(err error) bool { + return errors.Is(err, retryableError) + }) + + tt.initMock(mock) + + err = DoInTx(context.Background(), db, tt.fnProvider(), WithRetryPolicy(retryPolicy)) + if tt.wantErr == nil { + require.NoError(t, err) + } else { + require.EqualError(t, err, tt.wantErr.Error()) + } + require.NoError(t, mock.ExpectationsWereMet()) + }) + } } diff --git a/dbrutil/README.md b/dbrutil/README.md index b82ef1e..4a8b41f 100644 --- a/dbrutil/README.md +++ b/dbrutil/README.md @@ -2,8 +2,6 @@ [![GoDoc Widget]][GoDoc] -## Overview - `dbrutil` is a Go package that provides utilities and helpers for working with the [dbr query builder](https://github.com/gocraft/dbr). It simplifies database operations by offering: - **Database Connection Management**: Open a database connection with instrumentation for collecting metrics and logging slow queries. @@ -14,7 +12,8 @@ It simplifies database operations by offering: ## Usage -The following basic example demonstrates how to use `dbrutil` to create a database connection and run a transaction: +The following basic example demonstrates how to use `dbrutil` to open a database connection with instrumentation, +and execute queries with an automatic slow query logging and Prometheus metrics collection within transaction. ```go package main diff --git a/distrlock/README.md b/distrlock/README.md index e6b9024..83db374 100644 --- a/distrlock/README.md +++ b/distrlock/README.md @@ -2,15 +2,13 @@ [![GoDoc Widget]][GoDoc] -## Overview - `distrlock` is a Go package that implements distributed locking using SQL databases. It allows multiple processes or services to coordinate access to shared resources by acquiring and releasing locks stored in a database. ### Features -- Distributed lock management using SQL databases (PostgreSQL, MySQL are supported now) -- Support for acquiring, releasing, and extending locks -- Configurable lock expiration times +- Distributed lock management using SQL databases (PostgreSQL, MySQL are supported now). +- Support for acquiring, releasing, and extending locks. +- Configurable lock expiration times. ## How It Works diff --git a/example_test.go b/example_test.go new file mode 100644 index 0000000..c8cef64 --- /dev/null +++ b/example_test.go @@ -0,0 +1,51 @@ +package dbkit_test + +import ( + "context" + "database/sql" + "log" + "os" + "time" + + "github.com/acronis/go-appkit/retry" + + "github.com/acronis/go-dbkit" + + // Import the `mysql` package for registering the retryable function for MySQL transient errors (like deadlocks). + _ "github.com/acronis/go-dbkit/mysql" +) + +func Example() { + // Configure the database using the dbkit.Config struct. + // In this example, we're using MySQL. Adjust Dialect and config fields for your target DB. + cfg := &dbkit.Config{ + Dialect: dbkit.DialectMySQL, + MySQL: dbkit.MySQLConfig{ + Host: os.Getenv("MYSQL_HOST"), + Port: 3306, + User: os.Getenv("MYSQL_USER"), + Password: os.Getenv("MYSQL_PASSWORD"), + Database: os.Getenv("MYSQL_DATABASE"), + }, + MaxOpenConns: 16, + MaxIdleConns: 8, + } + + // Open the database connection. + // The 2nd parameter is a boolean that indicates whether to ping the database. + db, err := dbkit.Open(cfg, true) + if err != nil { + log.Fatalf("failed to open database: %v", err) + } + defer db.Close() + + // Execute a transaction with a custom retry policy (exponential backoff with 3 retries, starting from 10ms). + retryPolicy := retry.NewConstantBackoffPolicy(10*time.Millisecond, 3) + if err = dbkit.DoInTx(context.Background(), db, func(tx *sql.Tx) error { + // Execute your transactional operations here. + // Example: _, err := tx.Exec("UPDATE users SET last_login = ? WHERE id = ?", time.Now(), 1) + return nil + }, dbkit.WithRetryPolicy(retryPolicy)); err != nil { + log.Fatal(err) + } +} diff --git a/internal/testing/deadlock.go b/internal/testing/deadlock.go index 0aaa837..37d013b 100644 --- a/internal/testing/deadlock.go +++ b/internal/testing/deadlock.go @@ -49,7 +49,7 @@ func DeadlockTest(t *testing.T, dialect dbkit.Dialect, checkDeadlockErr func(err go func(ctx context.Context) { defer done.Done() - tx1Err = dbkit.DoInTxWithOpts(ctx, dbConn, txOpts, func(tx *sql.Tx) error { + tx1Err = dbkit.DoInTx(ctx, dbConn, func(tx *sql.Tx) error { if _, err := tx.Exec(fmt.Sprintf("UPDATE %s SET name=$1 WHERE id=$2", table1Name), "test100", 1); err != nil { return err } @@ -59,13 +59,13 @@ func DeadlockTest(t *testing.T, dialect dbkit.Dialect, checkDeadlockErr func(err return err } return nil - }) + }, dbkit.WithTxOptions(txOpts)) }(ctx) done.Add(1) go func(ctx context.Context) { defer done.Done() - tx2Err = dbkit.DoInTxWithOpts(ctx, dbConn, txOpts, func(tx *sql.Tx) error { + tx2Err = dbkit.DoInTx(ctx, dbConn, func(tx *sql.Tx) error { if _, err := tx.Exec(fmt.Sprintf("UPDATE %s SET name=$1 WHERE id=$2", table2Name), "test100", 1); err != nil { return err } @@ -78,7 +78,7 @@ func DeadlockTest(t *testing.T, dialect dbkit.Dialect, checkDeadlockErr func(err } return nil - }) + }, dbkit.WithTxOptions(txOpts)) }(ctx) done.Wait() diff --git a/migrate/README.md b/migrate/README.md index 00b45c7..18f1746 100644 --- a/migrate/README.md +++ b/migrate/README.md @@ -2,11 +2,9 @@ [![GoDoc Widget]][GoDoc] -The `migrate` package provides functionality for applying database migrations in your Go applications. It leverages [github.com/rubenv/sql-migrate](https://github.com/rubenv/sql-migrate) under the hood, ensuring a reliable and consistent approach to managing database schema changes. +`migrate` package provides functionality for applying database migrations in your Go applications. It leverages [github.com/rubenv/sql-migrate](https://github.com/rubenv/sql-migrate) under the hood, ensuring a reliable and consistent approach to managing database schema changes. -## Overview - -The `migrate` package offers two primary approaches for defining your migrations: +It offers two primary approaches for defining your migrations: - **Embedded SQL Migrations**: Store your migrations as plain SQL files (with separate `.up.sql` and `.down.sql` files) and embed them into your Go binary using Go's built-in embed package. This approach is straightforward and keeps your SQL scripts separate from your application code. - **Programmatic SQL Migrations**: Define your migrations directly in Go code. This method is more suitable when you require additional customization or more control over your migrations. It lets you write migrations as Go functions, while still leveraging SQL commands. diff --git a/retryable.go b/retryable.go index 7063a50..e2d7390 100644 --- a/retryable.go +++ b/retryable.go @@ -41,3 +41,9 @@ func RegisterIsRetryableFunc(d driver.Driver, retryable retry.IsRetryable) { return retryable(e) } } + +// UnregisterAllIsRetryableFuncs removes previously registered IsRetryable function for the given driver. +func UnregisterAllIsRetryableFuncs(d driver.Driver) { + t := reflect.TypeOf(d) + delete(retryableErrors, t) +} diff --git a/retryable_test.go b/retryable_test.go index 97fe28a..8bb653b 100644 --- a/retryable_test.go +++ b/retryable_test.go @@ -9,24 +9,20 @@ package dbkit import ( "context" "fmt" - "reflect" "testing" + "time" "github.com/acronis/go-appkit/retry" - "github.com/cenkalti/backoff/v4" "github.com/stretchr/testify/assert" ) func TestMultipleIsRetryError(t *testing.T) { - var called string + retryPolicy := retry.NewExponentialBackoffPolicy(time.Millisecond*50, 10) - // cleanup handlers - oldHandlers := retryableErrors - retryableErrors = map[reflect.Type]retry.IsRetryable{} - defer func() { - retryableErrors = oldHandlers - }() + UnregisterAllIsRetryableFuncs(nil) + // test multiple IsRetryable functions + called := "" RegisterIsRetryableFunc(nil, func(e error) bool { called += "1" return false @@ -39,11 +35,16 @@ func TestMultipleIsRetryError(t *testing.T) { called += "3" return false }) - - p := retry.NewExponentialBackoffPolicy(backoff.DefaultInitialInterval, 10) - _ = retry.DoWithRetry(context.Background(), p, GetIsRetryable(nil), nil, func(ctx context.Context) error { + _ = retry.DoWithRetry(context.Background(), retryPolicy, GetIsRetryable(nil), nil, func(ctx context.Context) error { return fmt.Errorf("fake error") }) - assert.Equal(t, "123", called, "Wrong call order") + + // unregister all functions and test that no one is called + UnregisterAllIsRetryableFuncs(nil) + called = "" + _ = retry.DoWithRetry(context.Background(), retryPolicy, GetIsRetryable(nil), nil, func(ctx context.Context) error { + return fmt.Errorf("fake error") + }) + assert.Equal(t, "", called) }