Very slow to import Client...

[edwardsnj]

Importing from atproto seems to take a long time. I'm on raspios bullseye, python 3.9. atproto v0.0.55. Yes, this is a low-powered place to run this, but this is extreme slowness.

Installed in ~/.local/... to ensure modules can be compiled etc... but this doesn't seem to help

% echo "from atproto import Client" | time -p python3
real 123.54
user 104.61
sys 1.20
% echo "from atproto import Client" | time -p python3
real 108.29
user 105.61
sys 0.98
% echo "from atproto import Client" | time -p python3
real 103.41
user 100.71
sys 1.21

Even going directly to the module doesn't seem to help:

% echo "from atproto_client.client.client import Client" | time -p python3
real 112.46
user 110.25
sys 1.08

What is it doing for 100+ seconds? How can I speed this up?

[MarshalX]

Wow... it takes up to 1 sec to me, but 100 is insane. All goes from this issue: pydantic/pydantic#9908

Now imagine that we have 400+ pydatic models

[edwardsnj]

OK, I have a hypothesis, will take me some time to test it. I'm guessing that the pydantic_core module (written in Rust, to speed it up) does not have a pre-compiled 32-bit ARM version of the module available. I'm on a Raspberry Pi Zero W. So it may be falling back to a pure python strategy, which is slooooow. I'm going to try to compile pydantic_core myself and see if that helps.

[edwardsnj]

As far as I can tell, my hypothesis is not correct - there is a compiled _pydantic_core module: _pydantic_core.cpython-39-arm-linux-gnueabihf.so and it appears to be loaded correctly. So its just very slow. While I love how quickly I was able to make a bluesky post client in python using this module, maybe its too heavyweight for my needs. I wonder what alternatives I have in python. I guess I can roll my own that just hits the web-service API to post.

[MarshalX]

That's very sad news 😞

[antont]

I wonder what alternatives I have in python. I guess I can roll my own that just hits the web-service API to post.

David Buchanan is making a minimalistic PDS in Python & C, I don't know if it could be helpful but maybe he has a lightweight atproto impl there (i didn't look) https://github.com/DavidBuchanan314/millipds

[antont]

apparently he uses a couple of his own libs for the AT stuff and the CBR code is indeed in C there https://github.com/DavidBuchanan314/dag-cbrrr/blob/main/src/cbrrr/_cbrrr.c

[MarshalX]

pydantic getting serious about import time now! they reported significant improvements in the new alpha version. feel free to test. more info: pydantic/pydantic#9908 (comment)

[MarshalX]

pydantic team did a very good job in the v2.11 release. the importing time and memory usage have been improved!

• their release with details: https://pydantic.dev/articles/pydantic-v2-11-release
• my comparison with models of atproto and bluesky: MarshalX/pydantic-memory-usage@f3f2d36

i highly recommend updating pydantic; gonna mark this as answered

[edwardsnj]

For my limited use-case, writing a bare-bones poster that hit the API directly was sufficient. Feel free to close the ticket.