Wrap the Rust HTTP client with make_deferred_yieldable

[MadLittleMods]

Wrap the Rust HTTP client with make_deferred_yieldable so downstream usage doesn't need to use PreserveLoggingContext() or make_deferred_yieldable.

it seems like we should have some wrapper around it that uses make_deferred_yieldable(...) to make things right so we don't have to do this in the downstream code.

-- @MadLittleMods, #18357 (comment)

Spawning from wanting to remove PreserveLoggingContext() from the codebase and thinking that we shouldn't have to pollute all downstream usage with PreserveLoggingContext() or make_deferred_yieldable

Part of #18905 (Remove sentinel logcontext where we log in Synapse)

Dev notes

• Speed up MAS token introspection #18357
• Start and stop the Tokio runtime with the Twisted reactor #18691

---

Tokio runtime is not running -> #18903 (comment)

---

• #12514 Fix MemoryReactor.callWhenRunning not invoking callbacks if already started twisted/twisted#12514
• trial does not complain about left over callWhenRunning-scheduled calls in between tests twisted/twisted#3025

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)

[reivilibre]

OK to merge or otherwise would be happy to see the wrapperless approach if you have the energy for it

[MadLittleMods]

This is based on @reivilibre suggestion in #18903 (comment)

But please double-check that I didn't fumble the lifetimes, etc trying to kludge things together.

[sandhose]

Ideally we'd avoid unwrap as much as possible, especially when a function call can potentially raise. Unfortunately OnceCell::get_or_try_init is unstable so we can't avoid it here easily, but the call to make_deferred_yieldable can bubble up errorsa

Suggested change

	/// Given a deferred, make it follow the Synapse logcontext rules
	fn make_deferred_yieldable<'py>(
	py: Python<'py>,
	deferred: &Bound<'py, PyAny>,
	) -> Bound<'py, PyAny> {
	let make_deferred_yieldable = MAKE_DEFERRED_YIELDABLE.get_or_init(|| {
	let sys = PyModule::import(py, "synapse.logging.context").unwrap();
	let func = sys.getattr("make_deferred_yieldable").unwrap().unbind();
	func
	});
	make_deferred_yieldable
	.call1(py, (deferred,))
	.unwrap()
	.extract(py)
	.unwrap()
	}
	/// Given a deferred, make it follow the Synapse logcontext rules
	fn make_deferred_yieldable<'py>(
	py: Python<'py>,
	deferred: &Bound<'py, PyAny>,
	) -> PyErr<Bound<'py, PyAny>> {
	let make_deferred_yieldable = MAKE_DEFERRED_YIELDABLE.get_or_init(|| {
	let sys = PyModule::import(py, "synapse.logging.context").unwrap();
	let func = sys.getattr("make_deferred_yieldable").unwrap().unbind();
	func
	});
	make_deferred_yieldable
	.call1(py, (deferred,))?
	.extract(py)
	}

[MadLittleMods]

Had to adjust it a bit. Updated to this:

static MAKE_DEFERRED_YIELDABLE: OnceLock<pyo3::Py<pyo3::PyAny>> = OnceLock::new();

/// Given a deferred, make it follow the Synapse logcontext rules
fn make_deferred_yieldable<'py>(
    py: Python<'py>,
    deferred: &Bound<'py, PyAny>,
) -> PyResult<Bound<'py, PyAny>> {
    let make_deferred_yieldable = MAKE_DEFERRED_YIELDABLE.get_or_init(|| {
        let sys = PyModule::import(py, "synapse.logging.context").unwrap();
        let func = sys.getattr("make_deferred_yieldable").unwrap().unbind();
        func
    });

    make_deferred_yieldable.call1(py, (deferred,))?.extract(py)
}

[MadLittleMods]

I've confirmed this test fails without the changes to rust/src/http_client.rs

To reproduce:

1. Checkout this PR: git checkout madlittlemods/18357-rust-http-client-remove-preserve-logging-context
2. Use the old version of the Rust HTTP Client: git checkout develop -- rust/src/http_client.rs
3. Rebuild Rust: poetry install --extras all
4. Run the test: SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.synapse_rust.test_http_client.HttpClientTestCase.test_logging_context

SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.synapse_rust.test_http_client.HttpClientTestCase.test_logging_context
tests.synapse_rust.test_http_client
  HttpClientTestCase
    test_logging_context ...                                            [ERROR]

===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/virtualenvs/matrix-synapse-xCtC9ulO-py3.13/lib/python3.13/site-packages/twisted/internet/defer.py", line 1857, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "synapse/tests/synapse_rust/test_http_client.py", line 196, in do_request
    self._check_current_logcontext("competing")
  File "synapse/tests/synapse_rust/test_http_client.py", line 151, in _check_current_logcontext
    self.assertEqual(
  File "/virtualenvs/matrix-synapse-xCtC9ulO-py3.13/lib/python3.13/site-packages/twisted/trial/_synctest.py", line 444, in assertEqual
    super().assertEqual(first, second, msg)
  File "/usr/lib/python3.13/unittest/case.py", line 907, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.13/unittest/case.py", line 1273, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
twisted.trial.unittest.FailTest: 'sentinel' != 'competing'
- sentinel
+ competing
 : Expected LoggingContext(competing) but saw sentinel

tests.synapse_rust.test_http_client.HttpClientTestCase.test_logging_context
-------------------------------------------------------------------------------
Ran 1 tests in 0.251s

FAILED (errors=1)

[MadLittleMods]

This test is based off the tests we have in tests/util/test_logcontext.py

[MadLittleMods]

Perhaps we should just remove this as we should just assume that everything in Synapse does follow the Synapse logcontext rules unless otherwise stated.

For example:

synapse/synapse/logging/context.py

Lines 853 to 856 in d1c96ee

	Returns:
	Deferred which returns the result of func, or `None` if func raises.
	Note that the returned Deferred does not follow the synapse logcontext
	rules.

[MadLittleMods]

Created a Twisted PR to fix this flaw in the MemoryReactor -> twisted/twisted#12514

Seems to have buy-in (approved) so hopefully we don't need to deal with this in the future.

[sandhose]

Ideally we'd avoid unwrap as much as possible, especially when a function call can potentially raise. Unfortunately OnceCell::get_or_try_init is unstable so we can't avoid it here easily, but the call to make_deferred_yieldable can bubble up errorsa

Suggested change

	/// Given a deferred, make it follow the Synapse logcontext rules
	fn make_deferred_yieldable<'py>(
	py: Python<'py>,
	deferred: &Bound<'py, PyAny>,
	) -> Bound<'py, PyAny> {
	let make_deferred_yieldable = MAKE_DEFERRED_YIELDABLE.get_or_init(|| {
	let sys = PyModule::import(py, "synapse.logging.context").unwrap();
	let func = sys.getattr("make_deferred_yieldable").unwrap().unbind();
	func
	});
	make_deferred_yieldable
	.call1(py, (deferred,))
	.unwrap()
	.extract(py)
	.unwrap()
	}
	/// Given a deferred, make it follow the Synapse logcontext rules
	fn make_deferred_yieldable<'py>(
	py: Python<'py>,
	deferred: &Bound<'py, PyAny>,
	) -> PyErr<Bound<'py, PyAny>> {
	let make_deferred_yieldable = MAKE_DEFERRED_YIELDABLE.get_or_init(|| {
	let sys = PyModule::import(py, "synapse.logging.context").unwrap();
	let func = sys.getattr("make_deferred_yieldable").unwrap().unbind();
	func
	});
	make_deferred_yieldable
	.call1(py, (deferred,))?
	.extract(py)
	}

[sandhose]

I would make the make_deferred_yieldable logic part of create_deferred. I don't see a scenario where we don't want this

[MadLittleMods]

I think that makes sense 👍

Usually, we use make_deferred_yieldable(...) on third-party code (out of our control) that doesn't follow Synapse logcontext rules. But we control this code and can make the deferred do the right thing as we create it.

[reivilibre]

This looks reasonable to me, thanks :)

[MadLittleMods]

Thanks for the review and pointers @reivilibre and @sandhose 🦏