Skip to content

Docs: C API: Clarify what happens when null bytes are passed to PyUnicode_AsUTF8 #127458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jan 20, 2025
Merged

Docs: C API: Clarify what happens when null bytes are passed to PyUnicode_AsUTF8 #127458

Changes from 7 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
680651c
Document what happens when PyUnicode_AsUTF8() is given embedded null …
ZeroIntensity Nov 30, 2024
8bfd541
Suggest PyUnicode_AsUTF8AndSize for user input.
ZeroIntensity Nov 30, 2024
52e9117
Switch to a note instead of a warning.
ZeroIntensity Nov 30, 2024
1b393d4
Update Doc/c-api/unicode.rst
ZeroIntensity Dec 1, 2024
040608b
Update Doc/c-api/unicode.rst
ZeroIntensity Dec 1, 2024
6fb8cbe
Play with the wording a little bit.
ZeroIntensity Dec 15, 2024
3c7b6be
Add a reference.
ZeroIntensity Jan 13, 2025
0eac45f
Update Doc/c-api/unicode.rst
ZeroIntensity Jan 13, 2025
35e0783
Switch the wording away from 'user input'
ZeroIntensity Jan 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions Doc/c-api/unicode.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1035,6 +1035,15 @@ These are the UTF-8 codec APIs:

As :c:func:`PyUnicode_AsUTF8AndSize`, but does not store the size.

.. warning::
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A warning is a strong signal. Maybe a note is enough?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally had it as a note, but I think a warning is what we want here. I only discovered this quirk of PyUnicode_AsUTF8 because it caused a crash in the _interpreters module. Things that can potentially cause security issues should probably get a warning, right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right that notes about security are usually documented in red as warnings.


This function does not have any special behavior for
`null bytes <https://en.wikipedia.org/wiki/Null_character>`_ embedded within
*unicode*. As a result, strings containing null bytes will remain in the returned
string, which some C functions might interpret as the end of the string, leading to
truncation. When handling user input, it is recommended to use :c:func:`PyUnicode_AsUTF8AndSize`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that "user input" is useful here. I would prefer to say something like: "If truncation is an issue, ..."

instead.

.. versionadded:: 3.3

.. versionchanged:: 3.7
Expand Down
Loading