Skip to content

Commit fcd9b61

Browse files
vasayxtxvasayxtx
committed
Update README.md and introduce variadic options for dbkit.DoInTx()
1 parent bdad532 commit fcd9b61

File tree

10 files changed

+531
-191
lines changed

10 files changed

+531
-191
lines changed

‎README.md‎

Lines changed: 206 additions & 115 deletions
Original file line numberDiff line numberDiff line change
@@ -2,164 +2,255 @@
22

33
[![GoDoc Widget]][GoDoc]
44

5+
`go‑dbkit` is a Go library designed to simplify and streamline working with SQL databases.
6+
It provides a solid foundation for connection management, instrumentation, error retry mechanisms, and transaction handling.
7+
Additionally, `go‑dbkit` offers a suite of specialized sub‑packages that address common database challenges—such as [distributed locking](./distrlock), [schema migrations](./migrate), and query builder utilities - making it a one‑stop solution for applications that needs to interact with multiple SQL databases.
8+
9+
## Features
10+
- **Transaction Management**: Execute functions within transactions that automatically commit on success or roll back on error. The transaction runner abstracts the boilerplate, ensuring cleaner and more reliable code.
11+
- **Retryable Queries**: Built‑in support for detecting and automatically retrying transient errors (e.g., deadlocks, lock timeouts) across various databases.
12+
- **Distributed Locking**: Implement SQL‑based distributed locks to coordinate exclusive access to shared resources across multiple processes.
13+
- **Database Migrations**: Seamlessly manage schema changes with support for both embedded SQL migrations (using Go’s embed package) and programmatic migration definitions.
14+
- **Query Builder Utilities**: Enhance your query‐building experience with utilities for popular libraries:
15+
* [dbrutil](./dbrutil): Simplifies working with the [dbr query builder](https://github.com/gocraft/dbr), adding instrumentation (Prometheus metrics, slow query logging) and transaction support.
16+
* [goquutil](./goquutil): Provides helper routines for the [goqu query builder](https://github.com/doug-martin/goqu) (currently, no dedicated README exists—refer to the source for usage).
17+
18+
## Packages Overview
19+
- Root `go‑dbkit` package provides configuration management, DSN generation, and the foundational retryable query functionality used across the library.
20+
- [dbrutil](./dbrutil) offers utilities for the dbr query builder, including:
21+
* Instrumented connection opening with Prometheus metrics.
22+
* Automatic slow query logging based on configurable thresholds.
23+
* A transaction runner that simplifies commit/rollback logic.
24+
Read more in [dbrutil/README.md](./dbrutil/README.md).
25+
- [distrlock](./distrlock) implements a lightweight, SQL‑based distributed locking mechanism that ensures exclusive execution of critical sections across concurrent services.
26+
Read more in [distrlock/README.md](./distrlock/README.md).
27+
- [migrate](./migrate):
28+
Manage your database schema changes effortlessly with support for both embedded SQL files and programmatic migrations.
29+
Read more in [migrate/README.md](./migration/README.md).
30+
- [goquutil](./goquutil) provides helper functions for working with the goqu query builder, streamlining common operations. (This package does not have its own README yet, so please refer to the source code for more details.)
31+
- RDBMS‑Specific dedicated sub‑packages are provided for various relational databases:
32+
* [mysql](./mysql) includes DSN generation, retryable error handling, and other MySQL‑specific utilities.
33+
* [sqlite](./sqlite) contains helpers to integrate SQLite seamlessly into your projects.
34+
* [postgres](./postgres) & [pgx](./pgx) offers tools and error handling improvements for PostgreSQL using both the lib/pq and pgx drivers.
35+
* [mssql](./mssql) provides MSSQL‑specific error handling, including registration of retryable functions for deadlocks and related transient errors.
36+
Each of these packages registers its own retryable function in the init() block, ensuring that transient errors (like deadlocks or cached plan invalidations) are automatically retried.o
37+
538
## Installation
639

740
```
841
go get -u github.com/acronis/go-dbkit
942
```
1043

11-
## Structure
12-
13-
### `/`
14-
Package `dbkit` provides helpers for working with different SQL databases (MySQL, PostgreSQL, SQLite and MSSQL).
44+
## Usage
1545

16-
### `/distrlock`
17-
Package distrlock contains DML (distributed lock manager) implementation (now DMLs based on MySQL and PostgreSQL are supported).
18-
Now only manager that uses SQL database (PostgreSQL and MySQL are currently supported) is available.
19-
Other implementations (for example, based on Redis) will probably be implemented in the future.
46+
### Basic Example
2047

21-
### `/migrate`
22-
Package migrate provides functionality for applying database migrations.
23-
24-
### `/mssql`
25-
Package mssql provides helpers for working with MSSQL.
26-
Should be imported explicitly.
27-
To register mssql as retryable func use side effect import like so:
48+
Below is a simple example that demonstrates how to register a retryable function for a MySQL database connection and execute a query within a transaction with a custom retry policy on transient errors.
2849

2950
```go
30-
import _ "github.com/acronis/go-dbkit/mssql"
31-
```
51+
package main
3252

33-
### `/mysql`
34-
Package mysql provides helpers for working with MySQL.
35-
Should be imported explicitly.
36-
To register mysql as retryable func use side effect import like so:
53+
import (
54+
"context"
55+
"database/sql"
56+
"log"
57+
"os"
58+
"time"
3759

38-
```go
39-
import _ "github.com/acronis/go-dbkit/mysql"
40-
```
60+
"github.com/acronis/go-appkit/retry"
4161

42-
### `/pgx`
43-
Package pgx provides helpers for working with Postgres via `jackc/pgx` driver.
44-
Should be imported explicitly.
45-
To register postgres as retryable func use side effect import like so:
62+
"github.com/acronis/go-dbkit"
4663

47-
```go
48-
import _ "github.com/acronis/go-dbkit/pgx"
49-
```
64+
// Import the `mysql` package for registering the retryable function for MySQL transient errors (like deadlocks).
65+
_ "github.com/acronis/go-dbkit/mysql"
66+
)
5067

51-
### `/postgres`
52-
Package postgres provides helpers for working with Postgres via `lib/pq` driver.
53-
Should be imported explicitly.
54-
To register postgres as retryable func use side effect import like so:
68+
func main() {
69+
// Configure the database using the dbkit.Config struct.
70+
// In this example, we're using MySQL. Adjust Dialect and config fields for your target DB.
71+
cfg := &dbkit.Config{
72+
Dialect: dbkit.DialectMySQL,
73+
MySQL: dbkit.MySQLConfig{
74+
Host: os.Getenv("MYSQL_HOST"),
75+
Port: 3306,
76+
User: os.Getenv("MYSQL_USER"),
77+
Password: os.Getenv("MYSQL_PASSWORD"),
78+
Database: os.Getenv("MYSQL_DATABASE"),
79+
},
80+
MaxOpenConns: 16,
81+
MaxIdleConns: 8,
82+
}
5583

56-
```go
57-
import _ "github.com/acronis/go-dbkit/postgres"
84+
// Open the database connection.
85+
// The 2nd parameter is a boolean that indicates whether to ping the database.
86+
db, err := dbkit.Open(cfg, true)
87+
if err != nil {
88+
log.Fatalf("failed to open database: %v", err)
89+
}
90+
defer db.Close()
91+
92+
// Execute a transaction with a custom retry policy (exponential backoff with 3 retries, starting from 10ms).
93+
retryPolicy := retry.NewConstantBackoffPolicy(10*time.Millisecond, 3)
94+
if err = dbkit.DoInTx(context.Background(), db, func(tx *sql.Tx) error {
95+
// Execute your transactional operations here.
96+
// Example: _, err := tx.Exec("UPDATE users SET last_login = ? WHERE id = ?", time.Now(), 1)
97+
return nil
98+
}, dbkit.WithRetryPolicy(retryPolicy)); err != nil {
99+
log.Fatal(err)
100+
}
101+
}
58102
```
59103

60-
### `/sqlite`
61-
Package sqlite provides helpers for working with SQLite.
62-
Should be imported explicitly.
63-
To register sqlite as retryable func use side effect import like so:
104+
### `dbrutil` Usage Example
105+
106+
The following basic example demonstrates how to use `dbrutil` to open a database connection with instrumentation,
107+
and execute queries with an automatic slow query logging and Prometheus metrics collection within transaction.
64108

65109
```go
66-
import _ "github.com/acronis/go-dbkit/sqlite"
67-
```
110+
package main
111+
112+
import (
113+
"context"
114+
"database/sql"
115+
"errors"
116+
"fmt"
117+
stdlog "log"
118+
"net/http"
119+
"os"
120+
"time"
121+
122+
"github.com/acronis/go-appkit/log"
123+
"github.com/gocraft/dbr/v2"
124+
125+
"github.com/acronis/go-dbkit"
126+
"github.com/acronis/go-dbkit/dbrutil"
127+
)
68128

69-
### `/dbrutil`
70-
Package dbrutil provides utilities and helpers for [dbr](https://github.com/gocraft/dbr) query builder.
129+
func main() {
130+
logger, loggerClose := log.NewLogger(&log.Config{Output: log.OutputStderr, Level: log.LevelInfo})
131+
defer loggerClose()
132+
133+
// Create a Prometheus metrics collector.
134+
promMetrics := dbkit.NewPrometheusMetrics()
135+
promMetrics.MustRegister()
136+
defer promMetrics.Unregister()
137+
138+
// Open the database connection with instrumentation.
139+
// Instrumentation includes collecting metrics about SQL queries and logging slow queries.
140+
eventReceiver := dbrutil.NewCompositeReceiver([]dbr.EventReceiver{
141+
dbrutil.NewQueryMetricsEventReceiver(promMetrics, queryAnnotationPrefix),
142+
dbrutil.NewSlowQueryLogEventReceiver(logger, 100*time.Millisecond, queryAnnotationPrefix),
143+
})
144+
conn, err := openDB(eventReceiver)
145+
if err != nil {
146+
stdlog.Fatal(err)
147+
}
148+
defer conn.Close()
71149

72-
### `/goquutil`
73-
Package goquutil provides auxiliary routines for working with [goqu](https://github.com/doug-martin/goqu) query builder.
150+
txRunner := dbrutil.NewTxRunner(conn, &sql.TxOptions{Isolation: sql.LevelReadCommitted}, nil)
151+
152+
// Execute function in a transaction.
153+
// The transaction will be automatically committed if the function returns nil, otherwise it will be rolled back.
154+
if dbErr := txRunner.DoInTx(context.Background(), func(tx dbr.SessionRunner) error {
155+
var result int
156+
return tx.Select("SLEEP(1)").
157+
Comment(annotateQuery("long_operation")). // Annotate the query for Prometheus metrics and slow query log.
158+
LoadOne(&result)
159+
}); dbErr != nil {
160+
stdlog.Fatal(dbErr)
161+
}
74162

75-
## Examples
163+
// The following log message will be printed:
164+
// {"level":"warn","time":"2025-02-14T16:29:55.429257+02:00","msg":"slow SQL query","pid":14030,"annotation":"query:long_operation","duration_ms":1007}
76165

77-
### Open database connection using the `dbrutil` package
166+
// Prometheus metrics will be collected:
167+
// db_query_duration_seconds_bucket{query="query:long_operation",le="2.5"} 1
168+
// db_query_duration_seconds_sum{query="query:long_operation"} 1.004573875
169+
// db_query_duration_seconds_count{query="query:long_operation"} 1
170+
}
78171

79-
```go
80-
func main() {
81-
// Create a new database configuration
82-
cfg := &db.Config{
83-
Driver: db.DialectMySQL,
84-
Host: "localhost",
85-
Port: 3306,
86-
Username: "your-username",
87-
Password: "your-password",
88-
Database: "your-database",
89-
}
172+
const queryAnnotationPrefix = "query:"
90173

91-
// Open a connection to the database
92-
conn, err := dbrutil.Open(cfg, true, nil)
93-
if err != nil {
94-
fmt.Println("Failed to open database connection:", err)
95-
return
174+
func annotateQuery(queryName string) string {
175+
return queryAnnotationPrefix + queryName
176+
}
177+
178+
func openDB(eventReceiver dbr.EventReceiver) (*dbr.Connection, error) {
179+
cfg := &dbkit.Config{
180+
Dialect: dbkit.DialectMySQL,
181+
MySQL: dbkit.MySQLConfig{
182+
Host: os.Getenv("MYSQL_HOST"),
183+
Port: 3306,
184+
User: os.Getenv("MYSQL_USER"),
185+
Password: os.Getenv("MYSQL_PASSWORD"),
186+
Database: os.Getenv("MYSQL_DATABASE"),
187+
},
96188
}
97-
defer conn.Close()
98189

99-
// Create a new transaction runner
100-
runner := dbrutil.NewTxRunner(conn, &sql.TxOptions{}, nil)
101-
102-
// Execute code inside a transaction
103-
err = runner.DoInTx(context.Background(), func(runner dbr.SessionRunner) error {
104-
// Perform database operations using the runner
105-
_, err := runner.InsertInto("users").
106-
Columns("name", "email").
107-
Values("Bob", "[email protected]").
108-
Exec()
109-
if err != nil {
110-
return err
111-
}
112-
113-
// Return nil to commit the transaction
114-
return nil
115-
})
190+
// Open database with instrumentation based on the provided event receiver (see github.com/gocraft/dbr doc for details).
191+
// Opening includes configuring the max open/idle connections and their lifetime and pinging the database.
192+
conn, err := dbrutil.Open(cfg, true, eventReceiver)
116193
if err != nil {
117-
fmt.Println("Failed to execute transaction:", err)
118-
return
194+
return nil, fmt.Errorf("open database: %w", err)
119195
}
196+
return conn, nil
120197
}
121198
```
122199

123-
### Usage of `distrlock` package
200+
More examples and detailed usage instructions can be found in the `dbrutil` package [README](./dbrutil/README.md).
201+
202+
### `distrlock` Usage Example
203+
204+
The following basic example demonstrates how to use `distrlock` to ensure exclusive execution of a critical section of code.
124205

125206
```go
126-
// Create a new DBManager with the MySQL dialect
127-
dbManager, err := distrlock.NewDBManager(db.DialectMySQL)
128-
if err != nil {
129-
log.Fatal(err)
130-
}
207+
package main
131208

132-
// Open a connection to the MySQL database
133-
dbConn, err := sql.Open("mysql", "user:password@tcp(localhost:3306)/database")
134-
if err != nil {
135-
log.Fatal(err)
136-
}
137-
defer dbConn.Close()
209+
import (
210+
"context"
211+
"database/sql"
212+
"log"
213+
"os"
214+
"time"
138215

139-
// Create a new lock
140-
lock, err := dbManager.NewLock(context.Background(), dbConn, "my_lock")
141-
if err != nil {
142-
log.Fatal(err)
143-
}
216+
"github.com/acronis/go-dbkit"
217+
"github.com/acronis/go-dbkit/distrlock"
218+
)
144219

145-
// Acquire the lock
146-
err = lock.Acquire(context.Background(), dbConn, 5*time.Second)
147-
if err != nil {
148-
log.Fatal(err)
149-
}
220+
func main() {
221+
// Setup database connection
222+
db, err := sql.Open("mysql", os.Getenv("MYSQL_DSN"))
223+
if err != nil {
224+
log.Fatal(err)
225+
}
226+
defer db.Close()
150227

151-
// Do some work while holding the lock
152-
fmt.Println("Lock acquired, doing some work...")
228+
ctx := context.Background()
153229

154-
// Release the lock
155-
err = lock.Release(context.Background(), dbConn)
156-
if err != nil {
157-
log.Fatal(err)
158-
}
230+
// Create "distributed_locks" table for locks.
231+
createTableSQL, err := distrlock.CreateTableSQL(dbkit.DialectMySQL)
232+
if err != nil {
233+
log.Fatal(err)
234+
}
235+
_, err = db.ExecContext(ctx, createTableSQL)
236+
if err != nil {
237+
log.Fatal(err)
238+
}
159239

160-
fmt.Println("Lock released")
240+
// Do some work exclusively.
241+
const lockKey = "test-lock-key-1" // Unique key that will be used to ensure exclusive execution among multiple instances
242+
err = distrlock.DoExclusively(ctx, db, dbkit.DialectMySQL, lockKey, func(ctx context.Context) error {
243+
time.Sleep(10 * time.Second) // Simulate work.
244+
return nil
245+
})
246+
if err != nil {
247+
log.Fatal(err)
248+
}
249+
}
161250
```
162251

252+
More examples and detailed usage instructions can be found in the `distrlock` package [README](./distrlock/README.md).
253+
163254
## License
164255

165256
Copyright © 2024 Acronis International GmbH.

0 commit comments

Comments
 (0)