Add More Multidict Functions C-API

[Vizonex]

What do these changes do?

The summary of #1185 should suffice but basically the pr was accidently merged then deleted luckily I had a backup of the original code on hand so that we can resume where we left off.

Are there changes in behavior for the user?

Changes mainly will involve everything explained in #1185 but given with some better details and other suggestions from other contributors will go into this one.

Related issue number

• New MultiDict C-API functions And Added Documentation #1185 it was accidently merged and then deleted, this will be the resumption to that pr to try and remedy that problem...

Checklist

• I think the code is well written
• Unit tests for the changes exist
• Documentation reflects the changes

[codspeed-hq]

CodSpeed Performance Report

Merging #1190 will not alter performance

_{Comparing Vizonex:capi (fd79456) with capi (636f265)}

Summary

✅ 245 untouched benchmarks

[asvetlov]

#1185 (review) review has a lot of comments that are not applied to this PR yet.

Could you please address it, or should I move these comments to the PR?

[Vizonex]

Tomorrow Or Later I will see about tracking down the gc/weakref bug.

[Vizonex]

@asvetlov I'm back. I woke up. I ran pre-commit install inside my env.

[Vizonex]

@asvetlov When I run pre-commit run --all-files here's what I get

Validate GitHub Workflows.......................................................Passed
Check GitHub Workflows set timeout-minutes......................................Passed
Validate ReadTheDocs Config.....................................................Passed
yamllint........................................................................Failed
- hook id: yamllint
- exit code: 1

.clang-format
  1:4       error    wrong new line character: expected \n  (new-lines)

.github/workflows/ci-cd.yml
  1:4       error    wrong new line character: expected \n  (new-lines)

.github/FUNDING.yml
  1:4       error    wrong new line character: expected \n  (new-lines)

.github/dependabot.yml
  1:4       error    wrong new line character: expected \n  (new-lines)

.codecov.yml
  1:4       error    wrong new line character: expected \n  (new-lines)

.pre-commit-config.yaml
  1:4       error    wrong new line character: expected \n  (new-lines)

.readthedocs.yml
  1:4       error    wrong new line character: expected \n  (new-lines)

.github/workflows/reusable-cibuildwheel.yml
  1:4       error    wrong new line character: expected \n  (new-lines)

.github/workflows/auto-merge.yml
  1:4       error    wrong new line character: expected \n  (new-lines)

.github/config.yml
  1:4       error    wrong new line character: expected \n  (new-lines)

.yamllint
  1:4       error    wrong new line character: expected \n  (new-lines)

.github/workflows/codeql.yml
  1:4       error    wrong new line character: expected \n  (new-lines)

.github/workflows/reusable-linters.yml
  1:4       error    wrong new line character: expected \n  (new-lines)

ruff check......................................................................Passed
ruff format.....................................................................Passed
there must be not top-level `__init__.py` in `tests/`.......(no files to check)Skipped
changelog filenames.........................................(no files to check)Skipped
Changelog files should use a non-broken :user:`name` role.......................Passed
clang-format....................................................................Passed
MyPy, for Python 3.13...........................................................Passed
MyPy, for Python 3.11...........................................................Passed

[Vizonex]

Ok so it appears all I have left is documentation according to the linter this should be an easy one to fix seems I had some spelling errors when I was writing late at night.

[Vizonex]

@asvetlov It's failing to built the wheels for mac-os for some strange reason but I will wait for you to get back before I do anything else...

[Vizonex]

@asvetlov Workflow seems to be ignore testcapi but I'm sure you'll find a way to get it to run.

[codecov]

Codecov Report

Attention: Patch coverage is 69.91525% with 71 lines in your changes missing coverage. Please review.

Project coverage is 96.12%. Comparing base (bfee715) to head (51bbc70).

Files with missing lines	Patch %	Lines
tests/test_capi.py	69.91%	69 Missing and 2 partials ⚠️

❌ Your patch check has failed because the patch coverage (0.43%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project check has failed because the head coverage (88.37%) is below the target coverage (100.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             capi    #1190      +/-   ##
==========================================
- Coverage   97.57%   96.12%   -1.46%     
==========================================
  Files          28       28              
  Lines        4125     4335     +210     
  Branches      736      738       +2     
==========================================
+ Hits         4025     4167     +142     
- Misses         52      119      +67     
- Partials       48       49       +1     

Flag	Coverage Δ
CI-GHA	96.12% <69.91%> (-1.46%)	⬇️
MyPy	79.62% <69.91%> (-0.62%)	⬇️
pytest	91.02% <0.43%> (-5.25%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[asvetlov]

Sorry, this evening I was busy preparing the 6.5.1 bugfix release.
Please let me review tomorrow.

[Vizonex]

So I have a theory on what caused MultiDict_New(...) to crash in the first place.
While playing around for a bit I discovered that using tp_alloc(XXX, 0) doesn't crash vs using the PyObject_GC_New method like you had before.

multidict/multidict/_multilib/capsule.h

Line 33 in ddbbf68

PyObject_GC_New(MultiDictObject, state->MultiDictType);

The problem may be tied to how Multidict is setup to first allocate memory.

multidict/multidict/_multidict.c

Line 911 in ddbbf68

{Py_tp_new, PyType_GenericNew},

Looking at PyType_GenericNew a bit deeper it turns out to be just doing this under the hood.

PyObject *
PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    return type->tp_alloc(type, 0);
}

Replacing PyObject_GC_New with PyType_GenericNew or type->tp_alloc(type, 0) prevents the crash.
So what I am going to do for now is replace what I have with it after we run the capi-tests first on all versions and systems and then we can try this incase the problem is somehow version related.

Sorry, this evening I was busy preparing the 6.5.1 bugfix release. Please let me review tomorrow.

My bad I'll wait for tomorrow, I apologize.

[asvetlov]

Replacing PyObject_GC_New with PyType_GenericNew or type->tp_alloc(type, 0) prevents the crash.

Yes, you are right. Better to call tp_alloc slot than PyType_GenericNew to prevent the case of changing tp_alloc slot in the future.

[Vizonex]

@asvetlov I'll leave the rest to you now, now that I've replaced the GC_New with tp_alloc All we need now is for the workflows to run the capi tests and will be all set to merge and then work on the rest of the c-api.

[asvetlov]

Ok.
The only thing that I'm not sure about is MultiDict_Equals.
In CPython, PyObject_RichCompare is used everywhere in public API, only PyUnicode has own comparison functions.
I doubt if we need MultiDict_Equals; a shorter API is better. PyObject_RichCompare should have almost the same performance as MultiDict_Equals; its usage is relatively unfrequent, I guess.

[Vizonex]

Ok. The only thing that I'm not sure about is MultiDict_Equals. In CPython, PyObject_RichCompare is used everywhere in public API, only PyUnicode has own comparison functions. I doubt if we need MultiDict_Equals; a shorter API is better. PyObject_RichCompare should have almost the same performance as MultiDict_Equals; its usage is relatively unfrequent, I guess.

Alright, makes sense I'll see about removing this one here in a bit.

[Vizonex]

@asvetlov I'm going to see about resolving a few conflicts today with this repo and test_multidict_benchmarks and then maybe after that we can see about merging and then we can work on adding in the rest of the api soon. Know that I will not be available on Saturday I may have a ton of stuff happening that day. After that, we should be fine.

[Vizonex]

@asvetlov I'll make sure your done with your benchmarks pr and then I will proceed.

[asvetlov]

I've done, benchmarks are stable now.

[Vizonex]

@asvetlov Sounds good let me sync what I need to sync

[Vizonex]

While I was looking how the new iterator I made I think I'm going to see if the http_writer.pyx module from aiohttp could be moved to C so that it has a lower-level footprint with only one function ever getting called. I'll make it into possibly a separate repo temporarily but it will utilize the new tech we worked on.

[asvetlov]

While I was looking how the new iterator I made I think I'm going to see if the http_writer.pyx module from aiohttp could be moved to C so that it has a lower-level footprint with only one function ever getting called. I'll make it into possibly a separate repo temporarily but it will utilize the new tech we worked on.

Maybe, let's see. Actually, Cython which utilizes multidict C API could be good enough

[Vizonex]

I have 1 script that can generate the cython-api but I haven't added it yet due to mypy hating my script. However it uses libclang and types-clang to get the job done. I was inspired by rust's bindgen tool to build my own.

• https://gist.github.com/Vizonex/01ec0c9d3e49069a5711e004163368c3

Here's my concept for optimizing the http-writer further with the new multidict c-api.
https://gist.github.com/Vizonex/44761bae90cbac5e1642a5ee097cef6c

[Vizonex]

@asvetlov I think I'm going to move some of the iterator functions over to something like MultidictIter since it comes with everything by default, this way users will be able to utilize the MultidictIter Object for using Next() and users will not have to Call to Free the position marker when iteration is over.

[Vizonex]

Can't wait to see 6.7 be released. Do we still want to add cython to it or should be cram that into 6.8?

[Vizonex]

Seems I added Cython in by accident but feel free to let me know if it shouldn't be autogenerated. A Handful of edits will likely fix that.

[Vizonex]

I added in my Cython script generator in because the __init__.pxd script it generates leaked in by accident. I have now added the tool that I've used to generate it unless anybody has any better ideas on how the cython portion should be maintained.

[Vizonex]

@asvetlov I wanted to provide cython support by the next update and I was wondering if I should go and document that portion of code or try moving away from trying to autogenerate that portion of code.

[Vizonex]

@asvetlov I ran some benchmarks in another pull request that is waiting for us to finish this up but if it inspires you to continue there's an almost 40% Performance increase in the _http_writer.pyx module now.
aio-libs/aiohttp#11320

[Vizonex]

@asvetlov Sorry for asking about this so suddenly but I am having problems adding the testcapi to the workflow and I was wondering if you had the time to help? I need to figure out a way to compile testcapi but using the current library rather than needing to fetch or have to point to a git url.

I guess the second thing I wanted to ask is if were going to resume this, I had some ideas with the new C-API outside of aiohttp that may involve yarl receiving more optimized functionality such as moving the URL object into Cython, the second project I had planned with this was a new msgspec conversion library using multidict for parsing server related things like payloads query parameters and transforming them to dataclasses.

[asvetlov]

Hi!
Sorry, I was partially available in the summer. I'll find time to review the current state of the PR this week.

[Vizonex]

Hi! Sorry, I was partially available in the summer. I'll find time to review the current state of the PR this week.

No problem thanks for letting me know. I did resume my IRL job recently so just a heads up but it should not restrain me from continuing to contribute here.