How does GOMEMLIMIT work with DuckDB? #89
-
|
If I set a GOMEMLIMIT of say 100GiB, does that mean that both the go and duckdb process both share that pool, since duckdb is started as a process from go, or is only for the pure Go memory allocations? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments
-
|
if I'm not wrong. CGO is not controled by go memory management. |
Beta Was this translation helpful? Give feedback.
-
|
GOMEMLIMIT only applies to memory allocated in Go and managed by the Go garbage collector. DuckDB has its own memory_limit setting which it will use as a soft limit to decide when to spill intermediate query results to disk. For query results, the memory is going to stay on the DuckDB / C++ side until it’s read by a So you have to tune these specifically for your workload. I would recommend monitoring each in production and then tuning from there. |
Beta Was this translation helpful? Give feedback.
GOMEMLIMIT only applies to memory allocated in Go and managed by the Go garbage collector.
DuckDB has its own memory_limit setting which it will use as a soft limit to decide when to spill intermediate query results to disk.
For query results, the memory is going to stay on the DuckDB / C++ side until it’s read by a
rows.Scan(&val)call in your app. At that point duckdb-go reads the duckdb data chunk memory directly and writes the converted value into thevalvariable you passed in. I’m not 100% sure if the data chunk on the DuckDB side is deallocated as results are read or only oncerows.Close()is called, but that’s a pretty fine grained detail that you don’t need to care about in most …