Skip to content

Fix deadlock in Measurements.close() and add test coverage - #282

Open
jaisinha77777 wants to merge 1 commit into
google-deepmind:mainfrom
jaisinha77777:fix-measurements-close-deadlock
Open

Fix deadlock in Measurements.close() and add test coverage#282
jaisinha77777 wants to merge 1 commit into
google-deepmind:mainfrom
jaisinha77777:fix-measurements-close-deadlock

Conversation

@jaisinha77777

Copy link
Copy Markdown
Contributor

Summary

While adding test coverage for concordia/utils/measurements.py (part of #205, no test file previously existed), I found that Measurements.close() deadlocks:

def close(self) -> None:
  with self._channels_lock:
    for channel in self._channels:
      self.close_channel(channel)   # tries to acquire _channels_lock again
    self._channels.clear()

def close_channel(self, channel: str) -> None:
  with self._channels_lock:
    del self._channels[channel]

_channels_lock is a plain threading.Lock (not reentrant), so close() blocks forever the moment it calls close_channel() while already holding the lock. Reproducible any time there's at least one open channel:

m = Measurements()
m.publish_datum('a', 1)
m.close()  # hangs forever

The loop also mutates self._channels while iterating over it, a second independent bug (RuntimeError: dictionary changed size during iteration) masked by the deadlock.

Separately, close_channel()'s docstring documents no-op behavior for a channel that doesn't exist ("If the channel doesn't exist yet, it will be created"), but the implementation (del self._channels[channel]) raises KeyError in that case.

There don't appear to be any current callers of close()/close_channel() in the repo, which is presumably why this has gone unnoticed.

Fix

  • close() now just clears the dict directly under the lock, without calling back into close_channel().
  • close_channel() uses dict.pop(channel, None) so it's idempotent on a missing channel, matching its docstring.

Test plan

  • Added concordia/utils/measurements_test.py, including a regression test that runs close() on a background thread and asserts it completes within a timeout (hangs/fails before this fix, passes after).
  • Added concordia/utils/async_measurements_test.py covering ReactiveMeasurements (capture/subscribe/dispose semantics, plus confirming close() doesn't deadlock on the subclass either).
  • python -m pytest concordia/utils/ -q -> all passing.
  • python -m pyink --check on all changed files -> clean.

Measurements.close() held _channels_lock while iterating over
_channels and calling close_channel() for each entry, which itself
tried to reacquire the same non-reentrant threading.Lock, deadlocking
whenever there was at least one open channel. The loop also mutated
_channels while iterating over it. close_channel() additionally
raised KeyError on a channel that was never published to, despite its
docstring documenting no-op behavior.

Adds measurements_test.py and async_measurements_test.py, covering
concordia/utils/measurements.py and async_measurements.py, which
previously had no test coverage. Part of google-deepmind#205.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant