gh-78319: add UTF8 marker per RFC #9436

gordonmessmer · 2018-09-20T03:30:43Z

This change implements RFC 6855 UTF8 APPEND per guidance from Sam Varshavchik:
https://bugs.python.org/issue34138

https://bugs.python.org/issue34138

Issue: imaplib RFC 6855 issue #78319

arhadthedev · 2023-02-13T18:02:19Z

@gordonmessmer tests are failing (see Details links under Some checks were not successful section). Would you mind to continue working on the PR, please?

bedevere-bot · 2023-02-14T04:31:32Z

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

gordonmessmer · 2023-02-14T05:19:23Z

I'll look at it again, yes. I'm getting odd results from updated tests, so some additional work is still needed...

gordonmessmer · 2023-02-14T07:06:36Z

Tests look like they're passing, but trying this on a live IMAP server fails, because the server actually receives:

LLFC3 APPEND INBOX UTF8 (~{63}\r\n

... when Sam suggested that it should receive:

LLFC3 APPEND INBOX NIL NIL UTF8 (~{63}\r\n

But that's probably a larger bug in the imaplib module, and not directly related to this change.

arhadthedev · 2023-02-14T07:10:22Z

@warsaw, @maxking (as active e-mail experts)

gordonmessmer · 2023-02-14T07:28:42Z

I've asked Sam to chime in. The RFC calls those arguments optional. Based on its behavior, it seems that Courier expects them to be specified as NIL when they are not provided, rather than left out entirely.

gordonmessmer · 2023-02-14T08:06:41Z

I added a commit for consideration. Transforming None to "NIL" is RFC-compliant (which is consistent with the intent of this PR). Tests pass, and this works with Courier IMAP.

gordonmessmer · 2024-12-15T01:30:56Z

Please let me know if there's anything I can do to help move this forward.

github-actions · 2025-04-13T06:03:19Z

This PR is stale because it has been open for 30 days with no activity.

python-cla-bot · 2025-04-18T09:54:41Z

All commit authors signed the Contributor License Agreement.

github-actions · 2025-09-21T06:03:24Z

This PR is stale because it has been open for 30 days with no activity.

serhiy-storchaka

LGTM. 👍

miss-islington-app · 2025-09-29T06:03:10Z

Thanks @gordonmessmer for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

Make UTF8 support for the IMAP APPEND command RFC 6855 compliant. (cherry picked from commit 408154d) Co-authored-by: Gordon Messmer <[email protected]>

bedevere-app · 2025-09-29T06:03:36Z

GH-139406 is a backport of this pull request to the 3.14 branch.

bedevere-app · 2025-09-29T06:03:42Z

GH-139407 is a backport of this pull request to the 3.13 branch.

…-139407) gh-78319: Fix implementation of IMAP APPEND UTF8 (GH-9436) Make UTF8 support for the IMAP APPEND command RFC 6855 compliant. (cherry picked from commit 408154d) Co-authored-by: Gordon Messmer <[email protected]>

bedevere-bot · 2025-09-29T07:02:03Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.x (tier-1) has failed when building commit 408154d.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/506/builds/11511) and take a look at the build logs.
Check if the failure is related to this commit (408154d) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/506/builds/11511

Failed tests:

test_pyrepl

Failed subtests:

test_timerfd_ns_TFD_TIMER_ABSTIME - test.test_os.TimerfdTests.test_timerfd_ns_TFD_TIMER_ABSTIME
test_repl_eio - test.test_pyrepl.test_unix_console.TestUnixConsoleEIOHandling.test_repl_eio

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_os.py", line 4644, in test_timerfd_ns_TFD_TIMER_ABSTIME
    self.assertLess(abs(next_expiration_ns - offset_ns),  limit_error)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 1062375 not less than 1000000


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 837, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed file <_io.FileIO name=8 mode='rb' closefd=True>
Warning -- Unraisable exception
Exception ignored while finalizing file <_io.FileIO name=10 mode='rb' closefd=True>:
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test
    regrtest_runner(result, test_func, runtests)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner
    test_result = test_func()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_func
    return run_unittest(test_mod, runtests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittest
    return _run_suite(tests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suite
    raise support.TestFailedWithDetails(err, errors, failures, stats=stats)


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1137, in __del__
    _warn("subprocess %s is still running" % self.pid,
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          ResourceWarning, source=self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: subprocess 1706007 is still running
Warning -- Unraisable exception
Exception ignored while finalizing file <_io.FileIO name=8 mode='rb' closefd=True>:
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test
    regrtest_runner(result, test_func, runtests)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner
    test_result = test_func()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_func
    return run_unittest(test_mod, runtests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittest
    return _run_suite(tests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suite
    raise support.TestFailedWithDetails(err, errors, failures, stats=stats)


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test
    regrtest_runner(result, test_func, runtests)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner
    test_result = test_func()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_func
    return run_unittest(test_mod, runtests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittest
    return _run_suite(tests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suite
    raise support.TestFailedWithDetails(err, errors, failures, stats=stats)


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1137, in __del__
    _warn("subprocess %s is still running" % self.pid,
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          ResourceWarning, source=self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: subprocess 1668708 is still running
Warning -- Unraisable exception
Exception ignored while finalizing file <_io.FileIO name=8 mode='rb' closefd=True>:
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test
    regrtest_runner(result, test_func, runtests)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner
    test_result = test_func()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_func
    return run_unittest(test_mod, runtests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittest
    return _run_suite(tests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suite
    raise support.TestFailedWithDetails(err, errors, failures, stats=stats)


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 837, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed file <_io.FileIO name=10 mode='rb' closefd=True>


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pyrepl/test_unix_console.py", line 362, in test_repl_eio
    _, err = proc.communicate(timeout=5)  # sleep for pty to settle
             ~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1219, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 2126, in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1266, in _check_timeout
    raise TimeoutExpired(
    ...<2 lines>...
            stderr=b''.join(stderr_seq) if stderr_seq else None)
subprocess.TimeoutExpired: Command '['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-E', '-S', '/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pyrepl/eio_test_script.py']' timed out after 5 seconds

…-139406) Make UTF8 support for the IMAP APPEND command RFC 6855 compliant. (cherry picked from commit 408154d) Co-authored-by: Gordon Messmer <[email protected]>

gordonmessmer added 2 commits September 19, 2018 20:24

issue34138: fix implementation of utf8 append

82660df

issue34138: fix implementation of utf8 append

085544f

gordonmessmer requested a review from a team as a code owner September 20, 2018 03:30

This comment was marked as resolved.

Sign in to view

the-knights-who-say-ni added the CLA not signed label Sep 20, 2018

bedevere-bot added the awaiting review label Sep 20, 2018

This comment was marked as resolved.

Sign in to view

brettcannon removed the CLA not signed label Mar 28, 2019

the-knights-who-say-ni added the CLA signed label Mar 28, 2019

ezio-melotti removed the CLA signed label Jul 13, 2022

Merge branch 'main' into master

463a07f

This comment was marked as resolved.

Sign in to view

arhadthedev changed the title ~~bpo-34138: add UTF8 marker per RFC~~ gh-78319: add UTF8 marker per RFC Feb 9, 2023

arhadthedev added stdlib Standard Library Python modules in the Lib/ directory topic-email labels Feb 9, 2023

Add NEWS for UTF8 IMAP APPEND fix.

5a92499

gordonmessmer force-pushed the master branch 3 times, most recently from 36fd655 to 41c490e Compare February 14, 2023 05:13

gordonmessmer force-pushed the master branch 2 times, most recently from c116b76 to 633ad44 Compare February 14, 2023 06:23

arhadthedev requested review from warsaw and removed request for a team February 14, 2023 08:10

github-actions bot added the stale Stale PR or inactive for long period of time. label Dec 15, 2024

Merge branch 'main' into master

94c5b1d

picnixz requested a review from serhiy-storchaka December 15, 2024 01:33

github-actions bot removed the stale Stale PR or inactive for long period of time. label Feb 27, 2025

github-actions bot added the stale Stale PR or inactive for long period of time. label Apr 13, 2025

github-actions bot removed the stale Stale PR or inactive for long period of time. label Jun 15, 2025

github-actions bot added the stale Stale PR or inactive for long period of time. label Sep 21, 2025

Merge branch 'main' into master

10993f6

serhiy-storchaka approved these changes Sep 29, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Sep 29, 2025

serhiy-storchaka added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Sep 29, 2025

serhiy-storchaka merged commit 408154d into python:main Sep 29, 2025
53 checks passed

github-project-automation bot moved this from Waiting Done to Done in CPython pull requests to have finally merged Sep 29, 2025

bedevere-app bot removed the awaiting merge label Sep 29, 2025

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Sep 29, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Sep 29, 2025

Uh oh!

gh-78319: add UTF8 marker per RFC #9436

gh-78319: add UTF8 marker per RFC #9436

Uh oh!

Conversation

gordonmessmer commented Sep 20, 2018 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

arhadthedev commented Feb 13, 2023

Uh oh!

bedevere-bot commented Feb 14, 2023

Uh oh!

gordonmessmer commented Feb 14, 2023

Uh oh!

gordonmessmer commented Feb 14, 2023

Uh oh!

arhadthedev commented Feb 14, 2023

Uh oh!

gordonmessmer commented Feb 14, 2023

Uh oh!

gordonmessmer commented Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gordonmessmer commented Dec 15, 2024

Uh oh!

github-actions bot commented Apr 13, 2025

Uh oh!

python-cla-bot bot commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 21, 2025

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app bot commented Sep 29, 2025

Uh oh!

bedevere-app bot commented Sep 29, 2025

Uh oh!

bedevere-app bot commented Sep 29, 2025

Uh oh!

bedevere-bot commented Sep 29, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Uh oh!

gordonmessmer commented Sep 20, 2018 •

edited by bedevere-bot

Loading

gordonmessmer commented Feb 14, 2023 •

edited

Loading

python-cla-bot bot commented Apr 18, 2025 •

edited

Loading