Large performance gap on Windows between Python installs (Store vs. official installer) when exporting Parquet via DuckDB

[hmilbi]

What happens?

Summary
Exporting a ~90,000,000-row table to Parquet (~300 MB) shows a large performance discrepancy depending on which Windows Python build is used. The Microsoft Store Python is dramatically faster than the official Python.org installer. Similar slowdowns appear across other workloads and languages.

Observed behavior

Python (official installer, python-3.13.7-amd64.exe): ~130 s

Python (Microsoft Store): ~18 s

Other workloads are also faster with the Microsoft Store Python.

CLI, Rust, C++, and C# implementations match the slower Python performance.

Disabling Windows Defender real-time protection makes no difference.

Data size

Table: ~90,000,000 rows

Resulting Parquet file: ~300 MB

Question
What could explain the consistent performance advantage of the Microsoft Store Python over the official Python.org build on Windows for this workload? Any guidance on likely causes or what to check next is appreciated.

To Reproduce

Repro snippet

import duckdb
import pytz

dbfile = "C:\\temp\\test\\db.duckdb"
output_path = "C:\\temp\\test\\test.parquet";

with duckdb.connect(database=dbfile) as conn:
  c = conn.execute("select count(*) from large_table;").fetchall()

  print(f"  exporting {c[0][0]} rows")
  
  copy_start_time = conn.execute("select now();").fetchone()[0] # needs pytz

  conn.sql(f"""
    COPY (
      select
        *
      from large_table
      order by col1,col2,col3,col4
    )
    TO '{output_path}'
    WITH (FORMAT PARQUET);
  """)

  copy_end_time = conn.execute("select now();").fetchone()[0]
  copy_duration = copy_end_time - copy_start_time;

  print(f"  exported in {copy_duration}")

OS:

Windows 11 Pro 24H2

DuckDB Package Version:

1.4.0

Python Version:

3.13.7

Full Name:

H M

Affiliation:

None

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

No - I cannot share the data sets because they are confidential

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

[evertlammerts]

Just to make sure: are both Python distributions the same minor version, i.e. 3.13?

CLI, Rust, C++, and C# implementations match the slower Python performance.

Do you mean to say that:

• the DuckDB CLI matches the slower Python performance
• you also tested C++, C# and Rust language bindings?

There might be a number of different reasons why performance is so different between the two. E.g. the Windows Store version might be compiled in a way that optimizes for your platform in ways that the "regular" CPython distros can't. Figuring out what exactly causes these differences might be complex.

That said, you could try and profile the code. I don't have the means to reproduce this at the moment, but if you'd want to you could try to profile the runs of both versions with Windows ADK. I have no experience with profiling on Windows so you'll have to figure most of it out yourself. That said, my understanding is that after installing you can use wpr -start CPU to start recording a profile, then run your script, and then run wpr -stop profile.etl to save the recorded profile in profile.etl. You should then be able to use the Windows Performance Analyzer (also in the ADK toolkit) to look at the profile and see where the most time is spent.

[paultiq]

I was able to reproduce this... there's a significant performance difference.

* I'm assuming it's library related, since Microsoft Store uses different library resolution, but I didn't look any further than this:

3.13.7 via UV

> uv run --active test_107.py
Not Microsoft Store
sys.version='3.13.7 (main, Sep 2 2025, 14:16:00) [MSC v.1944 64 bit (AMD64)]'
duckdb.version='1.4.1'
Elapsed: 2.735427299980074

3.13.7 via Microsoft Store

> %LocalAppData%\Microsoft\WindowsApps\python3.13.exe test_107.py
Microsoft Store
sys.version='3.13.7 (tags/v3.13.7:bcee1c3, Aug 14 2025, 14:15:11) [MSC v.1944 64 bit (AMD64)]'
duckdb.version='1.4.1'
Elapsed: 1.9414653000130784

Test Case:

import sys
import pathlib
import duckdb
import time

print(
    "Microsoft Store"
    if any(p.lower() == "windowsapps" for p in pathlib.Path(sys.executable).parts)
    else "Not Microsoft Store"
)

print(f"{sys.version=}")
print(f"{duckdb.__version__=}")

with duckdb.connect() as con:
    con.execute(
        "create table foo as select random(), r, r || 'something' from range(90000000) t(r)"
    )

    start = time.perf_counter()
    con.execute("copy foo to 'myfile.parquet'")
    end = time.perf_counter()
    print(f"Elapsed: {end - start}")

[hmilbi]

Just to make sure: are both Python distributions the same minor version, i.e. 3.13?

CLI, Rust, C++, and C# implementations match the slower Python performance.

Do you mean to say that:

• the DuckDB CLI matches the slower Python performance
• you also tested C++, C# and Rust language bindings?

There might be a number of different reasons why performance is so different between the two. E.g. the Windows Store version might be compiled in a way that optimizes for your platform in ways that the "regular" CPython distros can't. Figuring out what exactly causes these differences might be complex.

That said, you could try and profile the code. I don't have the means to reproduce this at the moment, but if you'd want to you could try to profile the runs of both versions with Windows ADK. I have no experience with profiling on Windows so you'll have to figure most of it out yourself. That said, my understanding is that after installing you can use wpr -start CPU to start recording a profile, then run your script, and then run wpr -stop profile.etl to save the recorded profile in profile.etl. You should then be able to use the Windows Performance Analyzer (also in the ADK toolkit) to look at the profile and see where the most time is spent.

Yes botth python versions are the same
Yes CLI and the other bindings perform not as fast as the microsoft store pyhon setup

[staticlibs]

@hmilbi

Python executable installed from Microsoft Store and the one installed from python.org run under different security contexts.

This is Microsoft Store one:

C:\Users\vboxuser\AppData\Local\Microsoft\WindowsApps\python.exe

And this is python.org one:

C:\Users\vboxuser\AppData\Local\Programs\Python\Python313\python.exe

MS Store Python is packaged with some sandboxing features (I believe it uses MSIX). For example, it does not allow for third-party apps (e.g. ProcMon) to listen to its events, like file system access.

What is possibly happening in your environment is that some EDR tool (not Windows Defender) is actively intercepting file system access for all processes. It can intercept all ordinary processes (Python from python.org, DuckDB CLI etc) and makes FS calls for them much slower. But it cannot intercept MS Store sandboxed python.exe so this one runs with full speed.

[paultiq]

My test case/repro did not have a third party EDR tool, it was plain vanilla Windows 11.

I saw about a 40% hit writing to parquet.

I'll run a test case on a clean ec2 instance.

[staticlibs]

@paultiq

I cannot reproduce this on clean Win11, the following script runs the same for me on both pythons from above:

import duckdb, os, time
con = duckdb.connect()
con.sql("INSTALL tpch")
con.sql("LOAD tpch")
con.sql("CALL dbgen(sf = 1)")
con.sql("SELECT COUNT(*) FROM lineitem").show()

parquet_file = "C:/tmp/lineitem.parquet"
if os.path.exists(parquet_file):
    os.remove(parquet_file)

start = time.time()
con.sql(f"COPY lineitem TO '{parquet_file}' WITH (FORMAT PARQUET)")
print(time.time() - start)

con.sql(f"SELECT COUNT(*) FROM '{parquet_file}'").show()

[paultiq]

I spun up a clean Azure Windows VM:

• E2s_v3
• Windows 11 24H2
• Microsoft Visual C++ 2015-2022 Redistributable (x64) - 14.44.35211
• Windows Store Python 3.13.8 (released today)
• Python.org Python 3.13.8
• I made no changes to the VM, other than installing the redistributable MSVC, and two python's.

Result of a 10 iterations of the above test (see modified test below):

• Windows Store Python: Avg 5.01 over 10 tests
• Python.org Download: Avg 5.77 over 10 tests

Same test run locally on my desktop:

• Windows Store: 0.78 over 10 tests
• Python.org Download: 1.001

Both tests show about a 15% difference, at least in my controlled test. I don't know the dataset the original poster had, I'd be curious what's different about their case that saw a 10x difference.

Modified Test

import duckdb, os, time

con = duckdb.connect()
con.sql("INSTALL tpch")
con.sql("LOAD tpch")
con.sql("CALL dbgen(sf = 1)")
con.sql("SELECT COUNT(*) FROM lineitem").show()


for i in range(10):
    parquet_file = f"C:/tmp/lineitem_{i}.parquet"
    if os.path.exists(parquet_file):
        os.remove(parquet_file)
    start = time.time()
    con.sql(f"COPY lineitem TO '{parquet_file}' WITH (FORMAT PARQUET)")
    print(time.time() - start)

10 Iterations Store: Avg 5.01

4.986903667449951
4.677879810333252
4.919991970062256
4.995419502258301
5.123619318008423
5.031739711761475
5.148375988006592
5.116974115371704
5.233185291290283
4.873164892196655

10 Iterations download: Avg 5.77

5.450824499130249
5.53841757774353
5.379317760467529
5.657989501953125
6.463369369506836
5.9332544803619385
6.2255940437316895
5.7328596115112305
5.574861288070679
5.7717931270599365

[staticlibs]

@paultiq

Both tests show about a 15% difference

My VM is way too noisy to catch 15% (due to virtual disk flushes and other background activity). I see larger difference between consecutive runs. By it would have definitely caught 2x+ difference. Given that all actual work is done by the same _duckdb.cp313-win_amd64.pyd binary from PIP, I would think that security explanation can be plausible for both 15% and 700% from OP.

[paultiq]

What is possibly happening in your environment is that some EDR tool (not Windows Defender) is actively intercepting file system access for all processes.
...
I would think that security explanation can be plausible for both 15% and 700% from OP.

My response above was only in response to the EDR comment. Neither of the systems had a third-party EDR, and disabling defender also didn't make a difference.

[paultiq]

As a side note, after running the "copy" multiple times, there appears to be an accumulated delay.

Running with 100 iterations, the final "copy" iteration took 2x longer than the first few iterations. I also see it with duckdb cli, as submitted noted, so this may be better fit for duckdb/duckdb. But, I don't see the slowdown with Windows Store python.

This is what I ran:

import duckdb, os, time, sys, pathlib

print(
    "Microsoft Store"
    if any(p.lower() == "windowsapps" for p in pathlib.Path(sys.executable).parts)
    else "Not Microsoft Store"
)

print(f"{sys.version=}")
print(f"{duckdb.__version__=}")

con = duckdb.connect()
con.sql("INSTALL tpch")
con.sql("LOAD tpch")
con.sql("CALL dbgen(sf = 1)")
con.sql("SELECT COUNT(*) FROM lineitem").show()


times = []
for i in range(100):
    parquet_file = f"C:/tmp/lineitem_{i}.parquet"

    if os.path.exists(parquet_file):
        os.remove(parquet_file)

    start = time.perf_counter()
    con.sql(f"COPY lineitem TO '{parquet_file}' WITH (FORMAT PARQUET)")
    end = time.perf_counter()
    print(end - start)
    times.append(end - start)

print(f"Avg: {sum(times) / len(times)}")


print(f"{sys.version=}")
print(f"{duckdb.__version__=}")
times = []
for i in range(10):
    # Create fresh connection each time
    with duckdb.connect() as con:
        con.sql("INSTALL tpch")
        con.sql("LOAD tpch")
        con.sql("CALL dbgen(sf = 1)")

        parquet_file = f"C:/tmp/lineitem_{i}.parquet"

        t1 = time.perf_counter()
        exists = os.path.exists(parquet_file)
        t2 = time.perf_counter()

        if exists:
            os.remove(parquet_file)
        t3 = time.perf_counter()

        con.sql(f"COPY lineitem TO '{parquet_file}' WITH (FORMAT PARQUET)")
        t4 = time.perf_counter()

        con.close()

        exists_time = t2 - t1
        remove_time = t3 - t2
        copy_time = t4 - t3
        total_time = t4 - t1

        times.append(total_time)

        print(
            f"Iter {i}: exists={exists_time:.5f}s, remove={remove_time:.5f}s, copy={copy_time:.4f}s, total={total_time:.4f}s"
        )


print(f"\nAvg: {sum(times) / len(times):.4f}s")

[hmilbi]

The script above does reproduce the issue on my system.

I have for MS Store vs. Not MS Store:

Avg 100 iterations: 0.6374516449999829 (no significant increase) vs. 1.2860171360000368 (increase from 0.7234887000004164 up to 1.6314998999996533)
Avg 10 : 0.5771s (no significant increase) vs. 1.8803s (increase from 1.5621s up to 2.1251s)

Considering that you cannot reproduce the ms store performance difference with the script above, I have to assume that it is an issue of my current configuration. I'll keep digging. Hopefully I can find the turbo switch.