gh-67041: Allow to distinguish between empty and not defined URI components #123305

serhiy-storchaka · 2024-08-25T07:41:55Z

Changes in the urllib.parse module:

Add option ~~allow_none~~ missing_as_none in urlparse(), urlsplit() and urldefrag(). If it is true, represent not defined components as None instead of an empty string.
Add option keep_empty in urlunparse() and urlunsplit(). If it is true, keep empty non-None components in the resulting string. By default it is the same as the allow_none value for the result of the urlparse() and urlsplit() calls.
~~Add option keep_empty in the geturl() method of DefragResult, SplitResult, ParseResult and the corresponding bytes counterparts.~~

Issue: urllib.parse wrongly strips empty #fragment, ?query, //netloc #67041

…I components Changes in the urllib.parse module: * Add option allow_none in urlparse(), urlsplit() and urldefrag(). If it is true, represent not defined components as None instead of an empty string. * Add option keep_empty in urlunparse() and urlunsplit(). If it is true, keep empty non-None components in the resulting string. * Add option keep_empty in the geturl() method of DefragResult, SplitResult, ParseResult and the corresponding bytes counterparts.

serhiy-storchaka · 2024-11-27T11:28:17Z

It is now ready to review. The status of allow_none is now saved in the DefragResult, SplitResult and ParseResult objects, so in most cases there is no need to pass the keep_empty argument. geturl() no longer needs the keep_empty parameter.

Unfortunately, these objects now have __dict__ and no longer immutable. This is because non-empty __slots__ is not compatible with tuple subclasses. This is a separate complex issue. I'll try to find a solution of it, but it may be difficult.

The long term plan is to make allow_none True by default, and later deprecate allow_none=False. keep_empty=False can still be useful.

serhiy-storchaka · 2024-12-05T11:12:24Z

I am sorry, I forget to copy the _keep_empty attribute in copying/encoding/decoding methods. Now the PR is ready for review.

@orsenthil, @barneygale, could you please make a review?

merwok · 2025-11-17T02:11:24Z

I think that allow_none is not the correct name: this is not about allowing nones in the input (like how allow_fragments allows fragment in the input), but about controlling the data returned.

Possible alternatives:

use_none (short)
missing_as_none
use_none_for_empty

serhiy-storchaka · 2025-11-17T09:09:33Z

Maybe missing_as_empty? In future the behavior will be changed to use None for missing by default.

merwok · 2025-11-17T14:24:12Z

I think it should be missing_as_empty_string (short for return missing values as empty strings), which is long!

To change the default in the future, do you plan on adding a warning first?
I could see many projects not paying attention to the new param and getting surprised (or silent bugs) a few years after.

serhiy-storchaka · 2025-11-17T14:34:15Z

To change the default in the future, do you plan on adding a warning first?

On one hand, a warning will inform everyone about the change (it should be a FutureWarning). You will have to pass explicit True or False to silence it and get your behavior. This how we normally do. On other hand, the warning will unnecessary disturb those who are fine with any behavior. We will discuss this when the time come.

serhiy-storchaka · 2025-11-17T14:39:37Z

I think it should be missing_as_empty_string (short for return missing values as empty strings), which is long!

Currently, missing and empty component are not distinguishable. This parameter will allow to distinguish them. This PR adds also the keeps_empty parameter. So, "empty" refers not only to string, but to component. missing_component_as_empty_component, or simply missing_as_empty. Will it work?

merwok · 2025-11-17T19:24:10Z

Ah your’re right, there is empty component and empty string.

missing_as_none is good!

merwok · 2025-11-17T19:27:01Z

Doc/library/urllib.parse.rst


+   If *keep_empty* is true, empty strings are kept in the result (for example,
+   a ``?`` for an empty query), only ``None`` components are omitted.
+   This allows to restore the URL that was parsed with option


Suggested change

This allows to restore the URL that was parsed with option

This allows rebuilding a URL that was parsed with option

orsenthil · 2025-11-19T03:08:12Z

Doc/library/urllib.parse.rst

 or on combining URL components into a URL string.

-.. function:: urlparse(urlstring, scheme='', allow_fragments=True)
+.. function:: urlparse(urlstring, scheme=None, allow_fragments=True, *, missing_as_none=False)


I am really sorry, that I am contributing so late; but "missing_as_none=False" is confusing and not intuitive at all to me.

Pretty sure, others who have not participated are going to feel the same.

The function signature and term is not giving a signal on what it is meant to be.

Are you open to new name?

I am adding @gpshead, as one of the active developers in this area, to get his opinion too.

I'm fine with this name. Admittedly I have spent too much time in the past wrangling problems in this library, but the reason it still works for me despite that is that it is a common concept: do you represent the absence of a value distinctly from the base zero/empty version of that type or not? That is what None is for. and missing_as_none is at least explicit in name to indicate that some values may be None. I'm not going to call it pretty but it is "understandable enough" for me. I can't come up with anything that'd be meaningfully better rather than just alternately-understandable.

What are your thoughts about missing_as_empty with the opposite semantic? In future, None will be returned for not defined components by default, and you will need to specify missing_as_none=False or missing_as_empty=True to restore the current behavior.

I think that name should really be missing_as_empty_string at least (we can’t say just “empty”), meaning missing parts returned as empty strings.

What about use_none=False which is short and doesn’t try to be self-explanatory, so people need to read the docs?

bedevere-app · 2025-11-19T03:08:25Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

gpshead

overall I like this, comments are all about how to explain it in the docs.

gpshead · 2025-11-19T05:37:07Z

Doc/library/urllib.parse.rst

 or on combining URL components into a URL string.

-.. function:: urlparse(urlstring, scheme='', allow_fragments=True)
+.. function:: urlparse(urlstring, scheme=None, allow_fragments=True, *, missing_as_none=False)


I'm fine with this name. Admittedly I have spent too much time in the past wrangling problems in this library, but the reason it still works for me despite that is that it is a common concept: do you represent the absence of a value distinctly from the base zero/empty version of that type or not? That is what None is for. and missing_as_none is at least explicit in name to indicate that some values may be None. I'm not going to call it pretty but it is "understandable enough" for me. I can't come up with anything that'd be meaningfully better rather than just alternately-understandable.

gpshead · 2025-11-19T05:44:52Z

Doc/library/urllib.parse.rst

+   a ``?`` for an empty query), only ``None`` components are omitted.
+   This allows to restore the URL that was parsed with option
+   ``missing_as_none=True``.
+   By default, *keep_empty* is true if *parts* is the result of the


It is easy to miss this "footnote" about keep_empty not being a simple False default. I think the function signature above should be more clear that it has a non-trivial default value when not explicitly specified. something awkward with a fake descriptive name but indicative that people should read further for details keep_empty=_FALSE_UNLESS_PARTS_IS_A_URLSPLIT_RESULT or similar perhaps?

Pydoc will output urlunparse(components, *, keep_empty=['not specified']). Is it fine, or you need more descriptive name?

gpshead · 2025-11-19T05:46:55Z

Doc/library/urllib.parse.rst

+   a ``?`` for an empty query), only ``None`` components are omitted.
+   This allows to restore the URL that was parsed with option
+   ``missing_as_none=True``.
+   By default, *keep_empty* is true if *parts* is the result of the


same comment as above. make it obvious from the function signature that the default depends on the type of parts.

serhiy-storchaka

Thank you for your attention.

I would like to take this opportunity to discuss a certain design issue. To make this change more user friendly, we need to keep the status of missing_as_none in the result of urlsplit() and others. So the user will not need to specify keep_empty=True each time they use missing_as_none=True, and change their code again in future. There are two ways to do this:

Return different subclasses of SplitResult (and of other 5 classes). This approach already was used to differentiate string/bytes results. This will add 12 new classes.
Add a (hidden) boolean attribute to instances of SplitResult (and of other 5 classes).

In both cases there are some issues with copying, pickling, etc, which will be resolved in different ways. In distant future, after flipping the default behavior and deprecating and removing the current behavior (if we completely remove it), we may return to the current state of 6 named tuple classes.

What approach do you prefer?

serhiy-storchaka · 2025-11-19T07:47:11Z

Doc/library/urllib.parse.rst

 or on combining URL components into a URL string.

-.. function:: urlparse(urlstring, scheme='', allow_fragments=True)
+.. function:: urlparse(urlstring, scheme=None, allow_fragments=True, *, missing_as_none=False)


What are your thoughts about missing_as_empty with the opposite semantic? In future, None will be returned for not defined components by default, and you will need to specify missing_as_none=False or missing_as_empty=True to restore the current behavior.

serhiy-storchaka · 2025-11-19T07:54:18Z

Doc/library/urllib.parse.rst

+   a ``?`` for an empty query), only ``None`` components are omitted.
+   This allows to restore the URL that was parsed with option
+   ``missing_as_none=True``.
+   By default, *keep_empty* is true if *parts* is the result of the


Pydoc will output urlunparse(components, *, keep_empty=['not specified']). Is it fine, or you need more descriptive name?

bedevere-app bot mentioned this pull request Aug 25, 2024

urllib.parse wrongly strips empty #fragment, ?query, //netloc #67041

Open

serhiy-storchaka force-pushed the urllib-parse-allow-none branch from 7032015 to a60c9be Compare August 31, 2024 09:55

serhiy-storchaka added 3 commits September 2, 2024 15:14

Document the things.

a1dbfa6

Merge branch 'main' into urllib-parse-allow-none

b50b778

Preserve the status of allow_none in results.

eaa9ce6

serhiy-storchaka marked this pull request as ready for review November 27, 2024 11:16

bedevere-app bot added the awaiting core review label Nov 27, 2024

serhiy-storchaka requested a review from orsenthil November 27, 2024 11:19

serhiy-storchaka marked this pull request as draft November 28, 2024 09:14

bedevere-app bot removed the awaiting core review label Nov 28, 2024

Preserve _keep_empty in copying and encoding.

e5c31dd

serhiy-storchaka marked this pull request as ready for review December 5, 2024 11:10

bedevere-app bot added the awaiting core review label Dec 5, 2024

barneygale self-requested a review December 5, 2024 14:03

Merge branch 'main' into urllib-parse-allow-none

78bdc13

serhiy-storchaka requested a review from AA-Turner as a code owner November 14, 2025 18:58

Remove redundant import.

5846bf2

serhiy-storchaka added 2 commits November 17, 2025 18:22

Merge branch 'main' into urllib-parse-allow-none

b578c9d

Rename allow_none to missing_as_none.

7d59b7e

merwok reviewed Nov 17, 2025

View reviewed changes

orsenthil requested changes Nov 19, 2025

View reviewed changes

bedevere-app bot removed the awaiting core review label Nov 19, 2025

bedevere-app bot added the awaiting changes label Nov 19, 2025

orsenthil requested a review from gpshead November 19, 2025 03:11

gpshead reviewed Nov 19, 2025

View reviewed changes

gpshead requested a review from sethmlarson November 19, 2025 06:02

Address review comments.

d025fa8

serhiy-storchaka commented Nov 19, 2025

View reviewed changes

	This allows to restore the URL that was parsed with option
	This allows rebuilding a URL that was parsed with option

Uh oh!

gh-67041: Allow to distinguish between empty and not defined URI components #123305

Are you sure you want to change the base?

gh-67041: Allow to distinguish between empty and not defined URI components #123305

Uh oh!

Conversation

serhiy-storchaka commented Aug 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka commented Nov 27, 2024

Uh oh!

serhiy-storchaka commented Dec 5, 2024

Uh oh!

merwok commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka commented Nov 17, 2025

Uh oh!

merwok commented Nov 17, 2025

Uh oh!

serhiy-storchaka commented Nov 17, 2025

Uh oh!

serhiy-storchaka commented Nov 17, 2025

Uh oh!

merwok commented Nov 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-app bot commented Nov 19, 2025

Uh oh!

gpshead left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

serhiy-storchaka commented Aug 25, 2024 •

edited

Loading

merwok commented Nov 17, 2025 •

edited

Loading