Skip to content

Commit 7260962

Browse files
ncrucesncruces
committed
Update README.md
1 parent e503be6 commit 7260962

File tree

6 files changed

+81
-49
lines changed

6 files changed

+81
-49
lines changed

‎README.md

Lines changed: 4 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -72,57 +72,18 @@ Go, wazero and [`x/sys`](https://pkg.go.dev/golang.org/x/sys) are the _only_ run
7272
### Caveats
7373

7474
This module replaces the SQLite [OS Interface](https://sqlite.org/vfs.html)
75-
(aka VFS) with a [pure Go](vfs/) implementation.
76-
This has benefits, but also comes with some drawbacks.
75+
(aka VFS) with a [pure Go](vfs/) implementation,
76+
which has advantages and disadvantages.
7777

78-
#### Write-Ahead Logging
79-
80-
Because Wasm does not support shared memory,
81-
[WAL](https://sqlite.org/wal.html) support is [limited](https://sqlite.org/wal.html#noshm).
82-
83-
To work around this limitation, SQLite is [patched](sqlite3/locking_mode.patch)
84-
to always use `EXCLUSIVE` locking mode for WAL databases.
85-
86-
Because connection pooling is incompatible with `EXCLUSIVE` locking mode,
87-
to use the [`database/sql`](https://pkg.go.dev/database/sql) driver
88-
with WAL mode databases you should disable connection pooling by calling
89-
[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns).
90-
91-
#### File Locking
92-
93-
POSIX advisory locks, which SQLite uses on Unix, are
94-
[broken by design](https://sqlite.org/src/artifact/2e8b12?ln=1073-1161).
95-
96-
On Linux, macOS and illumos, this module uses
97-
[OFD locks](https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html)
98-
to synchronize access to database files.
99-
OFD locks are fully compatible with POSIX advisory locks.
100-
101-
On BSD Unixes, this module uses
102-
[BSD locks](https://man.freebsd.org/cgi/man.cgi?query=flock&sektion=2).
103-
On BSD Unixes, BSD locks are fully compatible with POSIX advisory locks.
104-
105-
On Windows, this module uses `LockFileEx` and `UnlockFileEx`,
106-
like SQLite.
107-
108-
On all other platforms, file locking is not supported, and you must use
109-
[`nolock=1`](https://sqlite.org/uri.html#urinolock)
110-
(or [`immutable=1`](https://sqlite.org/uri.html#uriimmutable))
111-
to open database files.
112-
You can use [`vfs.SupportsFileLocking`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#SupportsFileLocking)
113-
to check if your platform supports file locking.
114-
115-
To use the [`database/sql`](https://pkg.go.dev/database/sql) driver
116-
with `nolock=1` you must disable connection pooling by calling
117-
[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns).
78+
Read more about the Go VFS design [here](vfs/README.md).
11879

11980
### Testing
12081

12182
This project aims for [high test coverage](https://github.com/ncruces/go-sqlite3/wiki/Test-coverage-report).
12283
It also benefits greatly from [SQLite's](https://sqlite.org/testing.html) and
12384
[wazero's](https://tetrate.io/blog/introducing-wazero-from-tetrate/#:~:text=Rock%2Dsolid%20test%20approach) thorough testing.
12485

125-
The pure Go VFS is tested by running SQLite's
86+
The Go VFS is tested by running SQLite's
12687
[mptest](https://github.com/sqlite/sqlite/blob/master/mptest/mptest.c)
12788
on Linux, macOS, Windows and FreeBSD.
12889

‎internal/util/mmap.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//go:build (linux || darwin) && (amd64 || arm64) && !sqlite3_flock && !sqlite3_noshm && !sqlite3_nosys
1+
//go:build (darwin || linux || illumos) && (amd64 || arm64) && !sqlite3_flock && !sqlite3_noshm && !sqlite3_nosys
22

33
package util
44

‎internal/util/mmap_other.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
//go:build !(linux || darwin) || !(amd64 || arm64) || sqlite3_flock || sqlite3_noshm || sqlite3_nosys
1+
//go:build !(darwin || linux || illumos) || !(amd64 || arm64) || sqlite3_flock || sqlite3_noshm || sqlite3_nosys
22

33
package util
44

‎vfs/README.md

Lines changed: 71 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,75 @@
22

33
This package implements the SQLite [OS Interface](https://sqlite.org/vfs.html) (aka VFS).
44

5-
It replaces the default SQLite VFS with a pure Go implementation.
5+
It replaces the default SQLite VFS with a **pure Go** implementation.
66

7-
It also exposes interfaces that should allow you to implement your own custom VFSes.
7+
It also exposes [interfaces](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#VFS)
8+
that should allow you to implement your own custom VFSes.
9+
10+
Since it is a from scratch reimplementation,
11+
there are naturally some ways it deviates from the original.
12+
13+
The main differences are [file locking](#file-locking) and [WAL mode](write-ahead-logging) support.
14+
15+
### File Locking
16+
17+
POSIX advisory locks, which SQLite uses on Unix, are
18+
[broken by design](https://sqlite.org/src/artifact/2e8b12?ln=1073-1161).
19+
20+
On Linux, macOS and illumos, this module uses
21+
[OFD locks](https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html)
22+
to synchronize access to database files.
23+
OFD locks are fully compatible with POSIX advisory locks.
24+
25+
On BSD Unixes, this module uses
26+
[BSD locks](https://man.freebsd.org/cgi/man.cgi?query=flock&sektion=2).
27+
On BSD Unixes, BSD locks are fully compatible with POSIX advisory locks.
28+
However, concurrency is reduced with BSD locks
29+
(`BEGIN IMMEDIATE` behaves the same as `BEGIN EXCLUSIVE`).
30+
31+
On Windows, this module uses `LockFileEx` and `UnlockFileEx`,
32+
like SQLite.
33+
34+
On all other platforms, file locking is not supported, and you must use
35+
[`nolock=1`](https://sqlite.org/uri.html#urinolock)
36+
(or [`immutable=1`](https://sqlite.org/uri.html#uriimmutable))
37+
to open database files.
38+
39+
To use the [`database/sql`](https://pkg.go.dev/database/sql) driver
40+
with `nolock=1` you must disable connection pooling by calling
41+
[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns).
42+
43+
You can use [`vfs.SupportsFileLocking`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#SupportsFileLocking)
44+
to check if your platform supports file locking.
45+
46+
### Write-Ahead Logging
47+
48+
On 64-bit Linux, macOS and illumos, this module uses `mmap` to implement
49+
[shared-memory for the WAL-index](https://sqlite.org/wal.html#implementation_of_shared_memory_for_the_wal_index),
50+
like SQLite.
51+
52+
To allow `mmap` to work, each connection needs to reserve a lot of address space.\
53+
To limit the amount of address space each connection needs,
54+
use [`WithMemoryLimitPages`](../tests/parallel/parallel_test.go#L21).
55+
56+
On all other platforms, [WAL](https://sqlite.org/wal.html) support is
57+
[limited](https://sqlite.org/wal.html#noshm).
58+
59+
To work around that limitation, SQLite is [patched](sqlite3/locking_mode.patch)
60+
to automatically use `EXCLUSIVE` locking mode for WAL databases on such platforms.
61+
62+
Because connection pooling is incompatible with `EXCLUSIVE` locking mode,
63+
to use the [`database/sql`](https://pkg.go.dev/database/sql) driver
64+
with WAL mode databases you should disable connection pooling by calling
65+
[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns).
66+
67+
You can use [`vfs.SupportsSharedMemory`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#SupportsSharedMemory)
68+
to check if your platform supports shared memory.
69+
70+
### Build tags
71+
72+
The VFS can be customized with a few build tags:
73+
- `sqlite3_flock` forces the use of BSD locks; it can be used on macOS to test the BSD locking implementation.
74+
- `sqlite3_nosys` prevents importing [`x/sys`](https://pkg.go.dev/golang.org/x/sys);
75+
disables locking _and_ shared memory on all platforms.
76+
- `sqlite3_noshm` disables shared memory on all platforms.

‎vfs/os_ofd.go

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
package vfs
44

55
import (
6+
"math/rand"
67
"os"
78
"time"
89

@@ -43,7 +44,7 @@ func osLock(file *os.File, typ int16, start, len int64, timeout time.Duration, d
4344
if timeout < time.Since(before) {
4445
break
4546
}
46-
osSleep(time.Millisecond)
47+
osSleep(time.Duration(rand.Int63n(int64(time.Millisecond))))
4748
}
4849
}
4950
return osLockErrorCode(err, def)

‎vfs/os_windows.go

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
package vfs
44

55
import (
6+
"math/rand"
67
"os"
78
"time"
89

@@ -135,7 +136,7 @@ func osLock(file *os.File, flags, start, len uint32, timeout time.Duration, def
135136
if timeout < time.Since(before) {
136137
break
137138
}
138-
osSleep(time.Millisecond)
139+
osSleep(time.Duration(rand.Int63n(int64(time.Millisecond))))
139140
}
140141
}
141142
return osLockErrorCode(err, def)

0 commit comments

Comments
 (0)