Quarto speed significantly slower than other python IDEs when working with huge dataframes

[hrushikul]

The issue that we see is that Quarto takes a significantly longer time in executing a Python code chunk when compared to running Python script on Rstudio, Jupyter notebook or PyCharm.
Below is the time taken to execute them on two different windows machines with 8GB RAM and a processing speed of 2.1GHz.

Why is Quarto taking a much longer time? Can you please help us with this? Our goal is to use one system for developing codes using both R and Python

Please Run this code in a quarto document inside a python chunk
quarto speed check.txt

[dragonstyle]

Without seeing the actual code that is triggering the issue, it is very hard to diagnose the reason. Can you share an example we could use to better understand the issue?

[cscheid]

Thanks for the test case. I can't reproduce the issue. Inside quarto, I get:

In the command line, I get:

$ python python-perf-test.py
/Users/cscheid/virtualenvs/homebrew-python3/lib/python3.10/site-packages/pandas/util/__init__.py:15: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing
/Users/cscheid/Desktop/daily-log/2022/09/20/python-perf-test.py:39: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_old['M'] = df_old['P'] * df_old['Q']
/Users/cscheid/Desktop/daily-log/2022/09/20/python-perf-test.py:53: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_new['M*Q'] = df_new['P'] * df_new['Q']
--- 2.9333550930023193 seconds ---

I suspect the environment you're running quarto is different from the other ones, and it's using different library versions, including ones with performance issues.

[hrushikul]

Thank you for your response @cscheid

I changed my default python environment and set up a new environment especially for running this code as you had suggested. But it still gave me similar results. My python console in the quarto document shows the output as:

In another iteration it showed the following output:

The above are the results. It takes more than 140 seconds in both cases whereas a normal python application like Jupyter notebook takes only around 20-25 sec for the execution of the same code. Following is the output on the Jupyter - notebook display:

I am using a system with 8 GB RAM and 2.10 GHz processor and windows working environment. I am wondering if you are getting the result in close to 3 seconds may have been using a really high-end system with high RAM and processor capacity or maybe a MAC environment. Can you please confirm? Because loading a CSV file with 600,000 rows and 40 columns alone take more than 3 seconds. I also tried the same code with default settings on 4 other laptops with configurations the same as mine. They are getting very similar results.

I have tried quite a few ways for a long time and nothing seems to be working. Can you please point me in the right direction? Am I missing out on something?

[cscheid]

You wrote:

The above are the results. It takes more than 140 seconds in both cases whereas a normal python application like Jupyter notebook takes only around 20-25 sec for the execution of the same code. Following is the output on the Jupyter - notebook display:

I continue to believe that this is not a quarto issue, especially since the warning you are getting doesn't come from quarto, it comes from pandas. As I said in my previous message,

I suspect the environment you're running quarto is different from the other ones, and it's using different library versions, including ones with performance issues.

You should start by printing out the versions of the libraries you're using.

[hrushikul]

Hi @cscheid

I did what you said. I updated the libraries to the latest updates and also fixed the warning from pandas. Still, I see a difference in the speeds of execution of the same script in the quarto document Vs (Jupyter notebook, pycharm, and anaconda spyder). All the python compilers work significantly faster than the quarto for the same code. [My hypothesis is python runs slow in quarto for large data frames]

This is the updated version of the code I am using:
quarto speed check.txt

Screenshot of Updated libraries in different compilers:
Jupyter notebook

Pycharm:

Updated library version in quarto document:

The following is the output time in seconds for different python compilers:
Jupyter notebook - time: 23.5 sec

Pycharm time: 23.5 sec

Anaconda Spyder time: 25.8 sec

Quarto document time: 111.6 sec

There clearly seems to be a difference between the working times of python in python compilers vs python in the quarto document. I strongly feel it may be because python runs slow in the quarto document when the data frames it is handling is large.

Can you please confirm the same?

[cscheid]

I still can't reproduce it. On my side, the quarto version:

and the version I run directly from python takes

$ python speed.py
1.4.4
1.23.2
done
--- 2.965824842453003 seconds ---

Your theory for what's happening unfortunately does not match with how quarto execution happens. Quarto runs python through jupyter. We literally don't run Python code: we take the python code in a .qmd file, prepare a jupyter notebook, and execute that instead. That's why I said before that this must be some configuration issue with the python environment that we are detecting. It cannot be "quarto executing the code differently" because quarto does not execute python code.

The next thing for you to try is to see if the libraries and implementation you're using are exactly the same. Try the following:

$ python
python
Python 3.10.6 (main, Aug 30 2022, 04:58:14) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> import sys
>>> import pandas
>>> sys.modules['pandas']
<module 'pandas' from '/Users/cscheid/virtualenvs/homebrew-python3/lib/python3.10/site-packages/pandas/__init__.py'>
>>> sys.modules['numpy']
<module 'numpy' from '/Users/cscheid/virtualenvs/homebrew-python3/lib/python3.10/site-packages/numpy/__init__.py'>
>>> platform.python_version()
'3.10.6'
>>> platform.uname()
uname_result(system='Darwin', node='Lydian.local', release='21.6.0', version='Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:23 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T6000', machine='arm64')

Check if the results of the module path match, as well as the details of the platform.

[hrushikul]

Hi @cscheid,

The path quarto was using was different from what jupyter notebook was using. The following is the previous path quarto was using:
Previous path Quarto

I used the following code to change the path:
Code for changing the python-path

The updated quarto path was the following:
Updated quarto path

The only thing is it is showing Python~1 [red underline] instead of Python310 as mentioned in the path. Is this the problem? If yes, how could we change it to the correct path?

The jupyter path also showed was same:

But after running the code, the time taken by quarto is still the same:
Quarto time with updated python path: 122 sec

It still didn't change much! Quarto is still taking a long time for the execution of the same code. I am unable to figure out the error. Please let me know your thoughts.

Another thing I wanted to point out is that PyCharm used the same python path and version as what quarto used earlier.
Pycharm python version

But still, the execution time of Pycharm for the same code is much faster than quarto [old python path]
PyCharm execution time: 16 sec

If the python path would've been the error, it should've made the PyCharm IDE work similarly to the quarto in terms of speed of the execution. Still, PyCharm works very similar to Jupyter in terms of speed execution despite having different python paths.

So, is this a Quarto issue in particular or something else?

[cscheid]

Check that the library versions of pandas in PyCharm, Jupyter, and quarto are the same. You haven't checked where Jupyter and PyCharm are getting Pandas from.

Also, do not use reticulate for changing the python path. Using that means that you will change the entire engine of execution away from jupyter, rendering the results incomparable.

You will need to figure out what it is in your environment that is causing quarto to find Python 3.9 instead of 3.10 (and, in addition, like I said above, make sure that the libraries are the same.)

[jthomasmock]

Hi @hrushikul as Carlos mentioned - there are a few things here that could be different across your speed testing.

You are executing Python code natively in Pycharm/Jupyter, but then you are choosing to switch to reticulate to execute the Python code in a Quarto document. Quarto can natively execute Python code via Jupyter kernel (which quite literally uses Jupyter to do ALL of the execution, Quarto doesn't have it's own 'engine'). Quarto can also use reticulate, but then there is the overhead of running Python code from a wrapper in R.

In your YAML, you can choose which engine to use:

engine: knitr # <- this is forcing R + Reticulate to execute
jupyter: python3 # <- this forces native execution in a python3 Jupyter kernel

On M1 Mac - I am getting ~3-5 seconds using either R + reticulate to execute your Python code or natively executing the Python code in Python with a Jupyter engine. I realize my environment is different than yours but to Carlos' point - this is not likely to be a Quarto issue but rather a difference in your environment regarding Jupyter kernels and/or reticulate.

A check you could do is run your example code in R alone with reticulate (option 1), in an RMarkdown file (using reticulate) (option 2) and also running your code in Quarto with jupyter: python3 (option 3).

These results would be an indication that reticulate might face slowdowns in your environment (regardless of whether it's used in Quarto or not).

Executing the code in a R script or RMarkdown gives us another indication that perhaps there is some issue you are facing with reticulate itself (which you could then move to the reticulate Github repo).

The expectation is that natively executing Python code in Quarto via a Jupyter kernel (instead of reticulate) gives you a similar speed to Jupyter alone since it literally IS Jupyter executing the code behind the scenes.

Examples below, executing around 3-5 seconds across the various methods with:

Python 3.10.6
Pandas 1.5.0
Numpy 1.23.2

Reticulate alone, from .R file

reticulate::source_python()

Use reticulate alone, from a .R file

reticulate-speed-test.R

library(reticulate)

source_python("py-speed-test.py")

Then a separate py-speed-test.py file.

py-speed-test.py

#################################################### CODE #########################################################
import time
import numpy as np
import pandas as pd
import os
import sys
from platform import python_version
from pandas._testing import rands_array
pd.set_option('display.max_columns',None)

python_version()

# creating a numbers dataframe
alphabets = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
x = pd.DataFrame(np.random.randint(100,10000,size=(600000,26)),columns=list(alphabets))
x.head()

# adding strings to numbers dataframe
N = (600000,14)
s_df = pd.DataFrame(rands_array(10, N))
s_df.shape


# combining number and string dataframe
df = pd.concat([x,s_df],axis=1)
df.shape

# write it to .csv file
path = os.getcwd().replace("\\",'/')
df.to_csv(path+'/quarto_file.csv',index=False)


start_time = time.time()

# reading the data
path = os.getcwd().replace('\\','/')
data = pd.read_csv(path+'/quarto_file.csv')

df_prep = data.copy()

# old data [slicing original data]
df_old = df_prep[df_prep.A>900]
df_old['M'] = df_old['P'] * df_old['Q']
old_group = df_old.groupby(['0']).agg({'G':'sum','Q':'sum','M':'sum','S':'sum'}).reset_index().rename(columns={'G':'old_G','Q':'old_Q','S':'old_S'})
old_group['old_M'] = old_group['M']/old_group['old_Q']
old_group = old_group.drop(['M'],axis=1)
old_group.shape
old_group.columns
old_group['old_G'] = old_group['old_G']/12
old_group['old_Q'] = old_group['old_Q']/12
old_group['old_S'] = old_group['old_S']/12
old_group

# new data [slicing original data]
df_new = df_prep[df_prep.Y>5000]
df_new.shape
df_new['M*Q'] = df_new['P'] * df_new['Q']

new_group = df_new.groupby(['0']).agg({'G':'sum','Q':'sum','M*Q':'sum','S':'sum'}).reset_index().rename(columns={'G':'new_G','Q':'new_Q','S':'new_S'})
new_group['new_M'] = new_group['M*Q']/new_group['new_Q']
new_group = new_group.drop(['M*Q'],axis=1)
new_group.shape
new_group['new_G'] = new_group['new_G']/4
new_group['new_Q'] = new_group['new_Q']/4
new_group['new_S'] = new_group['new_S']/4

# data checks [old data vs new data]
check = pd.merge(left=new_group,right=old_group,how='left',on=['0'])
check.columns
check.shape

check['%M change'] = 100*(new_group['new_M'] - old_group['old_M'])/ old_group['old_M']
check['%G change'] = 100*(new_group['new_G'] - old_group['old_G'])/ old_group['old_G']
check['%Q change'] = 100*(new_group['new_Q'] - old_group['old_Q'])/ old_group['old_Q']
check['%S change'] = 100*(new_group['new_S'] - old_group['old_S'])/ old_group['old_S']


print("--- %s seconds ---" % (time.time() - start_time))

RMarkdown - Python from R

This is RMarkdown to use reticulate - executing Python code from within R.

RMarkdown Example

This is RMarkdown to use reticulate - executing Python code from within R.

File name: rmd-speed.Rmd

---
output: html_document
engine: knitr
---

Quarto speed check

```{python}
#################################################### CODE #########################################################
import time
import numpy as np
import pandas as pd
import os
import sys
from platform import python_version
from pandas._testing import rands_array
pd.set_option('display.max_columns',None)

python_version()
```

```{python}
# creating a numbers dataframe
alphabets = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
x = pd.DataFrame(np.random.randint(100,10000,size=(600000,26)),columns=list(alphabets))
x.head()

# adding strings to numbers dataframe
N = (600000,14)
s_df = pd.DataFrame(rands_array(10, N))
s_df.shape


# combining number and string dataframe
df = pd.concat([x,s_df],axis=1)
df.shape

# write it to .csv file
path = os.getcwd().replace("\\",'/')
df.to_csv(path+'/quarto_file.csv',index=False)


start_time = time.time()

# reading the data
path = os.getcwd().replace('\\','/')
data = pd.read_csv(path+'/quarto_file.csv')

df_prep = data.copy()

# old data [slicing original data]
df_old = df_prep[df_prep.A>900]
df_old['M'] = df_old['P'] * df_old['Q']
old_group = df_old.groupby(['0']).agg({'G':'sum','Q':'sum','M':'sum','S':'sum'}).reset_index().rename(columns={'G':'old_G','Q':'old_Q','S':'old_S'})
old_group['old_M'] = old_group['M']/old_group['old_Q']
old_group = old_group.drop(['M'],axis=1)
old_group.shape
old_group.columns
old_group['old_G'] = old_group['old_G']/12
old_group['old_Q'] = old_group['old_Q']/12
old_group['old_S'] = old_group['old_S']/12
old_group

# new data [slicing original data]
df_new = df_prep[df_prep.Y>5000]
df_new.shape
df_new['M*Q'] = df_new['P'] * df_new['Q']

new_group = df_new.groupby(['0']).agg({'G':'sum','Q':'sum','M*Q':'sum','S':'sum'}).reset_index().rename(columns={'G':'new_G','Q':'new_Q','S':'new_S'})
new_group['new_M'] = new_group['M*Q']/new_group['new_Q']
new_group = new_group.drop(['M*Q'],axis=1)
new_group.shape
new_group['new_G'] = new_group['new_G']/4
new_group['new_Q'] = new_group['new_Q']/4
new_group['new_S'] = new_group['new_S']/4

# data checks [old data vs new data]
check = pd.merge(left=new_group,right=old_group,how='left',on=['0'])
check.columns
check.shape

check['%M change'] = 100*(new_group['new_M'] - old_group['old_M'])/ old_group['old_M']
check['%G change'] = 100*(new_group['new_G'] - old_group['old_G'])/ old_group['old_G']
check['%Q change'] = 100*(new_group['new_Q'] - old_group['old_Q'])/ old_group['old_Q']
check['%S change'] = 100*(new_group['new_S'] - old_group['old_S'])/ old_group['old_S']


print("--- %s seconds ---" % (time.time() - start_time))
```

Quarto: Python Native

This is Quarto, natively executing the Python code via a Jupyter kernel (no R involved)

Quarto Example w/ Jupyter kernel as engine

This is Quarto, natively executing the Python code via a Jupyter kernel (no R involved)

File name: qmd-speed-jupyter-kernel.qmd

---
format: html
jupyter: python3
---

Quarto speed check

```{python}
#################################################### CODE #########################################################
import time
import numpy as np
import pandas as pd
import os
import sys
from platform import python_version
from pandas._testing import rands_array
pd.set_option('display.max_columns',None)

python_version()
```

```{python}
# creating a numbers dataframe
alphabets = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
x = pd.DataFrame(np.random.randint(100,10000,size=(600000,26)),columns=list(alphabets))
x.head()

# adding strings to numbers dataframe
N = (600000,14)
s_df = pd.DataFrame(rands_array(10, N))
s_df.shape


# combining number and string dataframe
df = pd.concat([x,s_df],axis=1)
df.shape

# write it to .csv file
path = os.getcwd().replace("\\",'/')
df.to_csv(path+'/quarto_file.csv',index=False)


start_time = time.time()

# reading the data
path = os.getcwd().replace('\\','/')
data = pd.read_csv(path+'/quarto_file.csv')

df_prep = data.copy()

# old data [slicing original data]
df_old = df_prep[df_prep.A>900]
df_old['M'] = df_old['P'] * df_old['Q']
old_group = df_old.groupby(['0']).agg({'G':'sum','Q':'sum','M':'sum','S':'sum'}).reset_index().rename(columns={'G':'old_G','Q':'old_Q','S':'old_S'})
old_group['old_M'] = old_group['M']/old_group['old_Q']
old_group = old_group.drop(['M'],axis=1)
old_group.shape
old_group.columns
old_group['old_G'] = old_group['old_G']/12
old_group['old_Q'] = old_group['old_Q']/12
old_group['old_S'] = old_group['old_S']/12
old_group

# new data [slicing original data]
df_new = df_prep[df_prep.Y>5000]
df_new.shape
df_new['M*Q'] = df_new['P'] * df_new['Q']

new_group = df_new.groupby(['0']).agg({'G':'sum','Q':'sum','M*Q':'sum','S':'sum'}).reset_index().rename(columns={'G':'new_G','Q':'new_Q','S':'new_S'})
new_group['new_M'] = new_group['M*Q']/new_group['new_Q']
new_group = new_group.drop(['M*Q'],axis=1)
new_group.shape
new_group['new_G'] = new_group['new_G']/4
new_group['new_Q'] = new_group['new_Q']/4
new_group['new_S'] = new_group['new_S']/4

# data checks [old data vs new data]
check = pd.merge(left=new_group,right=old_group,how='left',on=['0'])
check.columns
check.shape

check['%M change'] = 100*(new_group['new_M'] - old_group['old_M'])/ old_group['old_M']
check['%G change'] = 100*(new_group['new_G'] - old_group['old_G'])/ old_group['old_G']
check['%Q change'] = 100*(new_group['new_Q'] - old_group['old_Q'])/ old_group['old_Q']
check['%S change'] = 100*(new_group['new_S'] - old_group['old_S'])/ old_group['old_S']


print("--- %s seconds ---" % (time.time() - start_time))
```