Skip to content

Commit c7f27f9

Browse files
author
Evert
committed
Review feedback
1 parent 2e2ad70 commit c7f27f9

File tree

1 file changed

+47
-54
lines changed

1 file changed

+47
-54
lines changed

docs/stable/dev/building/python.md

Lines changed: 47 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -18,15 +18,15 @@ For everything described on this page we make the following assumptions:
1818

1919
Make sure you have checked out the [DuckDB source](https://github.com/duckdb/duckdb/) and that you are in its root. E.g.:
2020

21-
```bash
21+
```batch
2222
$ git clone https://github.com/duckdb/duckdb.git
2323
...
2424
$ cd duckdb
2525
```
2626

2727
If you've _forked_ DuckDB you may run into trouble when building the Python package when you haven't pulled in the tags.
2828

29-
```bash
29+
```batch
3030
# Check your remotes
3131
git remote -v
3232
@@ -46,7 +46,7 @@ While we use Python's built-in `venv` module in our examples below, and technica
4646

4747
Create and activate a virtual env as follows:
4848

49-
```bash
49+
```batch
5050
# Create a virtual environment in the .venv folder (in the duckdb source root)
5151
$ python3 -m venv --prompt duckdb .venv
5252
@@ -56,14 +56,14 @@ $ source .venv/bin/activate
5656

5757
Make sure you have a modern enough version of pip available in your virtual env:
5858

59-
```bash
59+
```batch
6060
# Print pip's help
6161
$ python3 -m pip install --upgrade pip
6262
```
6363

6464
If that fails with `No module named pip` and you use `uv`, then run:
6565

66-
```bash
66+
```batch
6767
# Install pip
6868
$ uv pip install pip
6969
```
@@ -78,41 +78,19 @@ The following will build the package with the default set of extensions (json, p
7878

7979
#### Release build
8080

81-
```bash
81+
```batch
8282
GEN=ninja BUILD_PYTHON=1 make release
8383
```
8484

8585
#### Debug build
8686

87-
```bash
87+
```batch
8888
GEN=ninja BUILD_PYTHON=1 make debug
8989
```
9090

91-
#### Cloud Storage
92-
93-
You may need the package files to reside under the same prefix where the library is installed; e.g., when installing to cloud storage from a notebook.
94-
95-
First, get the repository based version number and extract the source distribution.
96-
97-
```bash
98-
python3 -m pip install build # required for PEP 517 compliant source dists
99-
cd tools/pythonpkg
100-
export SETUPTOOLS_SCM_PRETEND_VERSION=$(python3 -m setuptools_scm)
101-
pyproject-build . --sdist
102-
cd ../..
103-
```
104-
105-
Next, copy over the python package related files, and install the package.
106-
107-
```bash
108-
mkdir -p $DUCKDB_PREFIX/src/duckdb-pythonpkg
109-
tar --directory=$DUCKDB_PREFIX/src/duckdb-pythonpkg -xzpf tools/pythonpkg/dist/duckdb-${SETUPTOOLS_SCM_PRETEND_VERSION}.tar.gz
110-
pip install --prefix $DUCKDB_PREFIX -e $DUCKDB_PREFIX/src/duckdb-pythonpkg/duckdb-${SETUPTOOLS_SCM_PRETEND_VERSION}
111-
```
112-
11391
#### Verify
11492

115-
```bash
93+
```batch
11694
python3 -c "import duckdb; print(duckdb.sql('SELECT 42').fetchall())"
11795
```
11896

@@ -155,15 +133,24 @@ The following mechanisms add to the set of **_excluded_ extensions**:
155133

156134
### Show all installed extensions
157135

158-
```bash
136+
```batch
159137
python3 -c "import duckdb; print(duckdb.sql('SELECT extension_name, installed, description FROM duckdb_extensions();'))"
160138
```
161139

162140
## Development Environment
163141

164-
To set up the codebase for development you should run build duckdb as follows:
142+
This section walks you through the following steps:
165143

166-
```bash
144+
* Creating a CMake profile for development
145+
* Debugging the Python extension code with lldb
146+
147+
You can do this either on the CLI or from an IDE. The documentation below shows the configuration for CLion, but you should be able to get it to work with other IDEs like VSCode as well.
148+
149+
### Debugging From the CLI
150+
151+
Run this to configure the CMake profile needed to debug on the CLI:
152+
153+
```batch
167154
GEN=ninja BUILD_PYTHON=1 PYTHON_DEV=1 make debug
168155
```
169156

@@ -174,13 +161,11 @@ This will take care of the following:
174161

175162
Once the build completes, do a sanity check to make sure everything works:
176163

177-
```bash
164+
```batch
178165
python3 -c "import duckdb; print(duckdb.sql('SELECT 42').fetchall())"
179166
```
180167

181-
### Debugging
182-
183-
The basic recipe is to start `lldb` with your virtual env's Python interpreter and your script, then set a breakpoint and run your script.
168+
To debug, the basic recipe is to start `lldb` with your virtual env's Python interpreter and your script, then set a breakpoint and run your script.
184169

185170
For example, given a script `dataframe.df` with the following contents:
186171

@@ -191,7 +176,7 @@ print(duckdb.sql("select * from range(1000)").df())
191176

192177
The following should work:
193178

194-
```bash
179+
```batch
195180
lldb -- .venv/bin/python3 my_script.py
196181
...
197182
# Set a breakpoint
@@ -216,30 +201,38 @@ Target 0: (python3) stopped.
216201

217202
### Debugging in an IDE / CLion
218203

219-
After creating a debug build with `PYTHON_DEV` enabled, you should be able to get debugging going in an IDE that support `lldb`. Below are the instructions for CLion, but you should be able to get this going in e.g. VSCode as well.
204+
You should be able to get debugging going in an IDE that support `lldb`. Below are the instructions for CLion but you can copy the settings for your favorite IDE.
220205

221-
#### Configure the CMake Debug Profile
206+
#### Configure a CMake Debug Profile
222207

223-
This is a prerequisite for debugging, and will enable Intellisense and clang-tidy by generating a `compile-commands.json` file so your IDE knows how to inspect the source code. It also makes sure your Python virtual env can be found by your IDE's cmake.
208+
The following CMake profile enables Intellisense and clang-tidy by generating a `compile-commands.json` file so your IDE knows how to inspect the source code, and makes sure that the Python package will be built and installed in your Python virtual env.
224209

225-
Under `Settings | Build, Execution, Deployment | CMake` add the following CMake options:
226-
```console
227-
-DCMAKE_PREFIX_PATH=$CMakeProjectDir$/.venv;$CMAKE_PREFIX_PATH
228-
-DPython3_EXECUTABLE=$CMakeProjectDir$/.venv/bin/python3
229-
-DBUILD_PYTHON=1 -DPYTHON_DEV=1
230-
```
210+
Under `Settings | Build, Execution, Deployment | CMake`, add a profile and set the fields as follows:
211+
212+
* **Name**: Debug
213+
* **Build type**: Debug
214+
* **Generator**: Ninja
215+
* **CMake Options** (on a single line):
216+
```console
217+
-DCMAKE_PREFIX_PATH=$CMakeProjectDir$/.venv;$CMAKE_PREFIX_PATH
218+
-DPython3_EXECUTABLE=$CMakeProjectDir$/.venv/bin/python3
219+
-DBUILD_PYTHON=1
220+
-DPYTHON_DEV=1
221+
```
231222

232223
#### Create a run config for debugging
233224

234225
Under Run -> Edit Configurations... create a new CMake Application. Use the following values:
235-
* Name: Python Debug
236-
* Target: `python_src` (it doesn't actually matter what you select here)
237-
* Program arguments: `$FilePath$`
238-
* Working directory: `$ProjectFileDir$`
226+
* **Name**: Python Debug
227+
* **Target**: `All targets`
228+
* **Executable**: `[ABS_PATH_TO_YOUR_VENV]/bin/python3` (careful: this is a symlink and sometimes an IDE might try and follow it and fill in the path to the actual executable, but that will not work)
229+
* **Program arguments**: `$FilePath$`
230+
* **Working directory**: `$ProjectFileDir$`
231+
* **Before Launch**: `Build` (this should already be set)
239232

240233
That should be enough: Save and close.
241234

242-
Now you can set a breakpoint in a C++ file. You then open your Python script in your editor and use this config to start a debug session.
235+
Now you can set a breakpoint in a C++ file. You then open your Python script in your editor and use this config and run `Python Debug` in debug mode.
243236

244237
### Development and Stubs
245238

@@ -250,7 +243,7 @@ Now you can set a breakpoint in a C++ file. You then open your Python script in
250243
These stubs are important for autocomplete in many IDEs, as static-analysis based language servers can't introspect `duckdb`'s binary module.
251244

252245
To verify the stubs match the actual implementation:
253-
```bash
246+
```batch
254247
python3 -m pytest tests/stubs
255248
```
256249

@@ -274,7 +267,7 @@ When directly interacting with python functions that return a `PyObject*`, such
274267

275268
If you've forked DuckDB you may run into trouble when building the Python package when you haven't pulled in the tags.
276269

277-
```bash
270+
```batch
278271
# Check your remotes
279272
git remote -v
280273

0 commit comments

Comments
 (0)