Commit 9b79d59
authored
⚡️ Speed up method
Below is an optimized version of your program with respect to the provided line profiler results.
The major bottleneck is `self._connection.commit()` and, to a lesser extent, `cur.executemany(...)`.
We can **greatly accelerate SQLite bulk inserts** and commits by.
- Disabling SQLite's default autocommit mode and wrapping the bulk inserts in a single explicit transaction.
- Using `with self._connection`, which ensures a transaction/commit automatically or using `begin`/`commit` explicitly.
- Setting SQLite's `PRAGMA synchronous = OFF` and `PRAGMA journal_mode = MEMORY` if durability is not absolutely required, since this will make writes much faster (you may enable this once per connection only).
**Note:**
– These changes keep the function return value and signature identical.
– Connection and PRAGMA are set only once per connection.
– All existing comments are preserved, and new comments only explain modifications.
### Why this is faster.
- **Explicit transaction**: `self._connection.execute('BEGIN')` + one `commit()` is far faster than relying on SQLite's default behavior.
- **PRAGMA tweaks**: `synchronous = OFF` and `journal_mode = MEMORY` massively reduce disk sync/write overhead for benchmark data.
- **Batching**: Still using `executemany` for the most efficient bulk insert.
- **Single cursor, closed immediately**.
*If your use case absolutely requires durability against power loss, remove the two PRAGMA settings (or use `WAL` and `NORMAL` modes). This code retains the exact logic and return values, but will be considerably faster in typical benchmarking scenarios.*CodeFlashBenchmarkPlugin.write_benchmark_timings by 148% in PR #217 (proper-cleanup)1 parent 191317b commit 9b79d59
1 file changed
+18
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
51 | | - | |
| 51 | + | |
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
55 | | - | |
| 55 | + | |
| 56 | + | |
56 | 57 | | |
57 | 58 | | |
58 | 59 | | |
59 | 60 | | |
60 | 61 | | |
| 62 | + | |
61 | 63 | | |
62 | 64 | | |
63 | 65 | | |
| |||
290 | 292 | | |
291 | 293 | | |
292 | 294 | | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
293 | 309 | | |
294 | 310 | | |
0 commit comments