Skip to content

Comments

Remove our own tokio runtime and use the one provided by pyo3-async-runtimes#441

Merged
kylebarron merged 3 commits intomainfrom
kyle/remove-our-own-runtime
May 12, 2025
Merged

Remove our own tokio runtime and use the one provided by pyo3-async-runtimes#441
kylebarron merged 3 commits intomainfrom
kyle/remove-our-own-runtime

Conversation

@kylebarron
Copy link
Member

@kylebarron kylebarron commented May 9, 2025

cc @ion-elgreco ; any thoughts? You previously mentioned here that we should check for forked processes. I'm curious if you've considered using the runtime provided by pyo3-async-runtimes?

Closes #419

@ion-elgreco
Copy link

I believe the forked processes checking remains, pyo3-async also uses Tokio right?

Pyo3-async didn't make much sense for delta-rs because all methods in python are sync, and the rust code is async

@kylebarron
Copy link
Member Author

I believe the forked processes checking remains, pyo3-async also uses Tokio right?

pyo3-async-runtimes does use tokio, but I don't think it has as nice of a check as was here. I figure tokio will just silently stop working if the python user tries to multiprocess?

@ion-elgreco
Copy link

ion-elgreco commented May 9, 2025

I believe the forked processes checking remains, pyo3-async also uses Tokio right?

pyo3-async-runtimes does use tokio, but I don't think it has as nice of a check as was here. I figure tokio will just silently stop working if the python user tries to multiprocess?

It might deadlock or panic actually if you use fork

@kylebarron
Copy link
Member Author

Indeed this might fail when using fork-based multiprocessing, but we can wait to figure out how to document that when the first user complains.

@kylebarron kylebarron merged commit 8dc53c6 into main May 12, 2025
6 checks passed
@kylebarron kylebarron deleted the kyle/remove-our-own-runtime branch May 12, 2025 19:33
@oelhammouchi
Copy link

@kylebarron am I to understand that obstore is not fork-safe? I'm running into this issue right now.

@kylebarron
Copy link
Member Author

I think that yes, obstore is not fork safe. Do you have a suggestion of where to document that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use get_runtime from pyo3-async-runtimes instead of creating our own tokio runtime

3 participants