Fix Windows Threading Issues by FrsECM · Pull Request #385 · Lightning-AI/LitServe

FrsECM · 2024-12-04T22:59:18Z

Before submitting

Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)

⚠️ How does this PR impact the user? ⚠️
As a user, i need to serve a model with multiple worker per device on a windows machine

What does this PR do?

This PR propose a fix to bug from Issue #384.

Uvicorn reference: Source

For example, with this simple server :

import litserve as ls
class SimpleLitAPI(ls.LitAPI):
    def setup(self, device):
        self.model1 = lambda x: x**2
    def decode_request(self, request):
        return request["input"] 
    def predict(self, x):
        squared = self.model1(x)
        output = squared
        return {"output": output}

    def encode_response(self, output):
        return {"output": output} 

# (STEP 2) - START THE SERVER
if __name__ == "__main__":
    # scale with advanced features (batching, GPUs, etc...)
    server = ls.LitServer(SimpleLitAPI(), accelerator="auto", max_batch_size=1,workers_per_device=2)
    server.run(port=8000)

Bellow, you'll find the difference between before and after the PR :

Before the PR

(litserve_win) PS > python .\dummy_server.py
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\site-packages\litserve\server.py:475: UserWarning: Windows does not support forking. Using threads api_server_worker_type will be set to 'thread'
  warnings.warn(
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
Swagger UI is available at http://0.0.0.0:8000/docs
INFO:     Started server process [35312]
INFO:     Started server process [35312]
INFO:     Waiting for application startup.
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Application startup complete.
Accept failed on a socket
socket: <asyncio.TransportSocket fd=608, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 8000)>
Traceback (most recent call last):
  File "C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\asyncio\proactor_events.py", line 841, in loop
    f = self._proactor.accept(sock)
  File "C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\asyncio\windows_events.py", line 563, in accept
    self._register_with_iocp(listener)
  File "C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\asyncio\windows_events.py", line 732, in _register_with_iocp      
    _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 87] Paramètre incorrect
Task exception was never retrieved
future: <Task finished name='Task-7' coro=<IocpProactor.accept.<locals>.accept_coro() done, defined at C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\asyncio\windows_events.py:577> exception=OSError(22, 'L’opération d’entrée/sortie a été abandonnée en raison de l’arrêt d’un thread ou à la demande d’une application', None, 995, None)>
Traceback (most recent call last):
  File "C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\asyncio\windows_events.py", line 580, in accept_coro
    await future
OSError: [WinError 995] L’opération d’entrée/sortie a été abandonnée en raison de l’arrêt d’un thread ou à la demande d’une application
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
Setup complete for worker 1.
Setup complete for worker 0.

After the PR

(litserve_win) PS > python .\dummy_server.py
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
C:\Users\F296849\AppData\Local\miniforge3\envs\iris2_win\lib\site-packages\litserve\server.py:475: UserWarning: Windows does not support forking. Using threads api_server_worker_type will be set to 'thread'
  warnings.warn(
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
Swagger UI is available at http://0.0.0.0:8000/docs
INFO:     Started server process [24816]
INFO:     Waiting for application startup.
INFO:     Started server process [24816]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Application startup complete.
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
Setup complete for worker 0.
uvloop is not installed. Falling back to the default asyncio event loop. Please install uvloop for better performance using `pip install uvloop`.
uvloop is not installed. Falling back to the default asyncio event loop.
Setup complete for worker 1.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

codecov · 2024-12-04T23:08:38Z

Codecov Report

Attention: Patch coverage is 66.66667% with 10 lines in your changes missing coverage. Please review.

Project coverage is 88%. Comparing base (7e01984) to head (2daba46).
Report is 1 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #385   +/-   ##
===================================
- Coverage    89%    88%   -0%     
===================================
  Files        37     37           
  Lines      2164   2184   +20     
===================================
+ Hits       1918   1928   +10     
- Misses      246    256   +10

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

aniketmaurya · 2024-12-04T23:26:27Z

hi @FrsECM, thank you so much for the PR! Could you also add additional information about why setting config.workers fixes this issue for reference?

FrsECM · 2024-12-04T23:48:47Z

hi @FrsECM, thank you so much for the PR! Could you also add additional information about why setting config.workers fixes this issue for reference?

Sure.
It's to make sure the code go through this portion of uvicorn :
Kludex/uvicorn@858f1c5

By default, the number of worker is set to 1 and it crashes when we have multiples worker per devices on windows.

FrsECM · 2024-12-05T00:41:11Z

I have to investigate a little more on another issue that cause [WinError 10022].

It seems that in some case (not completely reproductible), the socket is not ready to listen when we start uvicorn servers.
A workarround is to force listening before setting up uvicorn servers. I don't have any idea about why it behaves like that.
If you have any idea.

FrsECM · 2024-12-06T00:05:31Z

@aniketmaurya did you have a look on the PR ?

It seems to fix my problem on windows.

It doesn't fix issue number #372 but it allows to make multiple worker on windows.

aniketmaurya

LGTM! thanks for creating the fix 🚀

src/litserve/server.py

FrsECM · 2024-12-08T17:31:24Z

@aniketmaurya i got an idea for #372.

Unlike on Linux, i discovered that inference workers were stopped first instead of uvicorn's.

A fix is to join on inference worker on windows and to cleanly give a signal to uvicorn in order to end threads properly.
In order to do it, i need to keep uvicorn's server in a class variable.
It's because currently only workers are returned by method _start_server().

I renamed some variables to be a little more explicit about their content.

src/litserve/server.py

aniketmaurya · 2024-12-12T19:08:53Z

@FrsECM it seems like the Windows tests are stuck and have timed out.

FrsECM · 2024-12-12T19:10:46Z

Hum. Curious. On my windows machine I don’t have issues. I don’t know well the test stack. I'll try to have a look tomorrow. Envoyé à partir de Outlook pour iOS<https://aka.ms/o0ukef>

…

________________________________ De : Aniket Maurya ***@***.***> Envoyé : Thursday, December 12, 2024 8:09:16 PM À : Lightning-AI/LitServe ***@***.***> Cc : François Ponchon ***@***.***>; Mention ***@***.***> Objet : Re: [Lightning-AI/LitServe] Fix Windows Threading Issues (PR #385) @FrsECM<https://github.com/FrsECM> it seems like the Windows tests are stuck and have timed out. — Reply to this email directly, view it on GitHub<#385 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGG5F7H2TWURUOQHJ6OVXRT2FHNNZAVCNFSM6AAAAABTBG2HB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZZHAYDQMJZHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

FrsECM · 2024-12-13T09:29:26Z

@aniketmaurya i did a first try and suspected httpx>=0.28.0. But i was not up to date.

On python 3.10.15, i have no issues on my computer, every tests are running.

(litserve) PS C:\BUSCODE\packages\LitServe> python -m pytest
C:\Users\F296849\AppData\Local\miniforge3\envs\litserve\lib\site-packages\pytest_asyncio\plugin.py:207: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

  warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
==================================================================== test session starts ====================================================================
platform win32 -- Python 3.10.15, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\BUSCODE\packages\LitServe
configfile: pytest.ini
plugins: anyio-4.6.2.post1, asyncio-0.25.0, cov-6.0.0
asyncio: mode=strict, asyncio_default_fixture_loop_scope=None
collected 169 items

tests\e2e\test_e2e.py .............                                                                                                                    [  7%]
tests\test_auth.py ....                                                                                                                                [ 10%]
tests\test_batch.py ...........                                                                                                                        [ 16%]
tests\test_callbacks.py ....                                                                                                                           [ 18%]
tests\test_cli.py ..                                                                                                                                   [ 20%] 
tests\test_compression.py .                                                                                                                            [ 20%]
tests\test_connector.py s..ssssssss.                                                                                                                   [ 27%] 
tests\test_docker_builder.py ..                                                                                                                        [ 28%]
tests\test_examples.py ..........                                                                                                                      [ 34%]
tests\test_form.py ...                                                                                                                                 [ 36%]
tests\test_lit_server.py .........sssss..ss...........                                                                                                 [ 53%]
tests\test_litapi.py ..................                                                                                                                [ 64%]
tests\test_logger.py ........                                                                                                                          [ 69%]
tests\test_logging.py ...                                                                                                                              [ 71%] 
tests\test_loops.py .............                                                                                                                      [ 78%]
tests\test_middlewares.py ...                                                                                                                          [ 80%]
tests\test_pydantic.py .                                                                                                                               [ 81%]
tests\test_readme.py s                                                                                                                                 [ 81%] 
tests\test_schema.py ..                                                                                                                                [ 82%]
tests\test_simple.py .......                                                                                                                           [ 86%]
tests\test_specs.py ..................                                                                                                                 [ 97%]
tests\test_torch.py .s                                                                                                                                 [ 98%]
tests\test_utils.py ..                                                                                                                                 [100%]
================================================= 151 passed, 18 skipped, 50 warnings in 204.37s (0:03:24) ==================================================

On python 3.11.11, i have no issues on my computer, every tests are running.

(litserve3.11) PS C:\BUSCODE\packages\LitServe> python -m pytest
C:\Users\F296849\AppData\Local\miniforge3\envs\litserve3.11\Lib\site-packages\pytest_asyncio\plugin.py:207: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

  warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
==================================================================== test session starts ====================================================================
platform win32 -- Python 3.11.11, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\BUSCODE\packages\LitServe
configfile: pytest.ini
plugins: anyio-4.7.0, asyncio-0.25.0, cov-6.0.0, retry-1.6.3
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None
collected 169 items

tests\e2e\test_e2e.py .............                                                                                                                    [  7%]
tests\test_auth.py ....                                                                                                                                [ 10%]
tests\test_batch.py ...........                                                                                                                        [ 16%]
tests\test_callbacks.py ....                                                                                                                           [ 18%]
tests\test_cli.py ..                                                                                                                                   [ 20%]
tests\test_compression.py .                                                                                                                            [ 20%]
tests\test_connector.py s..ssssssss.                                                                                                                   [ 27%] 
tests\test_docker_builder.py ..                                                                                                                        [ 28%] 
tests\test_examples.py ..........                                                                                                                      [ 34%]
tests\test_form.py ...                                                                                                                                 [ 36%]
tests\test_lit_server.py .........sssss..ss...........                                                                                                 [ 53%]
tests\test_litapi.py ..................                                                                                                                [ 64%]
tests\test_logger.py ........                                                                                                                          [ 69%]
tests\test_logging.py ...                                                                                                                              [ 71%] 
tests\test_loops.py .............                                                                                                                      [ 78%]
tests\test_middlewares.py ...                                                                                                                          [ 80%]
tests\test_pydantic.py .                                                                                                                               [ 81%]
tests\test_readme.py s                                                                                                                                 [ 81%] 
tests\test_schema.py ..                                                                                                                                [ 82%]
tests\test_simple.py .......                                                                                                                           [ 86%]
tests\test_specs.py ..................                                                                                                                 [ 97%]
tests\test_torch.py .s                                                                                                                                 [ 98%]
tests\test_utils.py ..                                                                                                                                 [100%] 
================================================= 151 passed, 18 skipped, 48 warnings in 205.77s (0:03:25) ==================================================

FrsECM · 2024-12-13T09:42:17Z

@aniketmaurya I tried to increase the timeout. The usage of threading instead of processes and the absence of uvloop on windows makes it slower. It may also depend on the runner.
But i need your agreement to rerun the ci.

.github/workflows/ci-testing.yml

aniketmaurya · 2025-01-05T23:05:51Z

hi @FrsECM, Happy New Year! How is it going here? Please let me know if you need any help?

FrsECM · 2025-01-06T06:55:41Z

Hi ! Happy new year ! Unfortunately I’m unable to reproduce what happens on the stuck CICD locally. And I don’t have ressource to do this locally. (I can only use docker through wsl (eg. linux) with my company’s computer. For me the fix would be to increase the timeout since the issue was present before my proposal to fix multi worker problems. I suspect a problem with virtualization since my computer is not a VM. Envoyé à partir de Outlook pour iOS<https://aka.ms/o0ukef>

…

________________________________ De : Aniket Maurya ***@***.***> Envoyé : Monday, January 6, 2025 12:06:13 AM À : Lightning-AI/LitServe ***@***.***> Cc : François Ponchon ***@***.***>; Mention ***@***.***> Objet : Re: [Lightning-AI/LitServe] Fix Windows Threading Issues (PR #385) hi @FrsECM<https://github.com/FrsECM>, Happy New Year! How is it going here? Please let me know if you need any help? — Reply to this email directly, view it on GitHub<#385 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGG5F7AL7MTOLQDFOMNAV4T2JG3GLAVCNFSM6AAAAABTBG2HB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZRG44DCNZSHE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

src/litserve/server.py

RaiaN · 2025-01-23T21:26:28Z

This is still not fixed in latest litServe python package. Please fix it? It is such a trivial thing to fix for any dev. I have 100% reproducible example.

FrsECM · 2025-01-24T07:53:35Z

This is still not fixed in latest litServe python package. Please fix it? It is such a trivial thing to fix for any dev. I have 100% reproducible example.

The fix is implemented... Just i've been asked to check the CI that have been crashed before i started the implementation of the PR and i don't have hands on this. I suspect something with VMs.

Hope i'll get some support from maintainer about that.

FrsECM · 2025-02-05T08:47:27Z

This is still not fixed in latest litServe python package. Please fix it? It is such a trivial thing to fix for any dev. I have 100% reproducible example.

The fix is implemented... Just i've been asked to check the CI that have been crashed before i started the implementation of the PR and i don't have hands on this. I suspect something with VMs.

Hope i'll get some support from maintainer about that.

@aniketmaurya, @Borda could you have a look on the CICD stuck ?
As i said, it's not a commit from my PR that generated the issue, it started before my first commit.

2bbed42

For me the fix is to :

fix the CI to test on python 3.10 like before
change the timeout

Thanks,

FrsECM · 2025-02-28T18:16:53Z

@bhimrazy, is it possible to have a way to allow individual contributors to run pending checks manually ?
Thanks a lot.

bhimrazy · 2025-02-28T18:32:08Z

@bhimrazy, is it possible to have a way to allow individual contributors to run pending checks manually ? Thanks a lot.

Hi @FrsECM , I'm not sure about the access permissions on the CI, but I'll look into this issue over this weekend. I may need to test it in a separate PR since I don't have edit access to PRs either.

Thanks for your patience! 😊

bhimrazy · 2025-03-31T17:26:08Z

@bhimrazy, is it possible to have a way to allow individual contributors to run pending checks manually ? Thanks a lot.

Hi @FrsECM , I'm not sure about the access permissions on the CI, but I'll look into this issue over this weekend. I may need to test it in a separate PR since I don't have edit access to PRs either.

Thanks for your patience! 😊

Sorry, guys! I did start working on this issue but had to pause since it requires a windows device, and mine is currently under repair due to some part replacements. Hoping to get back to this issue soon!

Apologies again, for the late update—I thought I had already mentioned it (my bad)!

That said, if this is a priority and someone else wants to take it up, please feel free to go ahead.

…iple_workers

FrsECM · 2025-04-08T16:03:55Z

Just merged with last main branch. Can someone run pipelines ?
Thanks

…iple_workers

for more information, see https://pre-commit.ci

…FrsECM/LitServe into bugfix/windows_multiple_workers

for more information, see https://pre-commit.ci

FrsECM · 2025-04-29T15:35:13Z

I created a windows VM in order to understand a little more the issue...

I proposed a new way to handle KeyboardInterrupt on Windows that could have less impacts on tests.
Instead of joining on different process what can lock test in windows case, i handle now the KeyboardInterrupt Exception in Loops :

# Catch the main pid and  :
class LitLoop(_BaseLoop):
    def __init__(self):
        self._context = {}
        self.server_pid = os.getpid()

    ....
   # Kindly ask os to remove the server pid in case of keyboard interuption :
    def kill(self):
        with self._lock:
            try:
                print(f'Stop Server Requested - Kill parent pid [{self._server_pid}] from [{os.getpid()}]')
                os.kill(self._server_pid,signal.SIGTERM)
            except PermissionError:
                # Access Denied because pid already killed...
                return

Because on windows and macos the Ctrl+C is catched by Inference worker, we can just add an exception to handle the case :

        ...
        except KeyboardInterrupt:
            print(f"Keyboard Interruption - Kill server [{self.server_pid}]")
            self.kill()
            return

It works on macos and windows to kill server with Ctrl+C :

Could you please try to run the CICD to check if it positively affected tests ?
@Borda, @justusschock, @aniketmaurya

Thanks !

aniketmaurya

@FrsECM LGTM! Thank you for updating the PR and your patience on this.

src/litserve/server.py

src/litserve/loops/base.py

src/litserve/server.py

aniketmaurya · 2025-04-30T12:19:31Z

all good @FrsECM! Merging this after the CI turns green 🚀

aniketmaurya · 2025-04-30T14:02:10Z

merging this since tested and worked on GPUs

FrsECM requested review from Andrei-Aksionov, Borda, aniketmaurya, ethanwharris and lantiga as code owners December 4, 2024 22:59

aniketmaurya approved these changes Dec 6, 2024

View reviewed changes

src/litserve/server.py Outdated Show resolved Hide resolved

FrsECM changed the title ~~Fix bug on windows with uvicorn when multiple workers.~~ Fix Windows Threading Issues Dec 8, 2024

aniketmaurya reviewed Dec 8, 2024

View reviewed changes

src/litserve/server.py Outdated Show resolved Hide resolved

src/litserve/server.py Outdated Show resolved Hide resolved

aniketmaurya self-requested a review December 8, 2024 23:39

Borda approved these changes Dec 11, 2024

View reviewed changes

FrsECM requested a review from Borda December 13, 2024 09:48

aniketmaurya reviewed Dec 13, 2024

View reviewed changes

.github/workflows/ci-testing.yml Outdated Show resolved Hide resolved

justusschock approved these changes Jan 17, 2025

View reviewed changes

src/litserve/server.py Outdated Show resolved Hide resolved

src/litserve/server.py Outdated Show resolved Hide resolved

FrsECM requested a review from tchaton as a code owner January 24, 2025 07:47

Fix bug on windows with uvicorn when multiple workers.

8bc070d

Fix timeout for gpu tests

2d404c4

Borda mentioned this pull request Mar 4, 2025

[wip]: debug windows threading issue #441

Closed

4 tasks

Merge remote-tracking branch 'upstream/main' into bugfix/windows_mult…

0d487d7

…iple_workers

FrsECM and others added 8 commits April 29, 2025 09:47

Merge remote-tracking branch 'upstream/main' into bugfix/windows_mult…

7534794

…iple_workers

Fix KeyboardInterrupt Windows

bf15d9f

[pre-commit.ci] auto fixes from pre-commit.com hooks

c37229c

for more information, see https://pre-commit.ci

KeyboardInterrupt Windows - MultipleWorkers

8eebacf

Merge branch 'bugfix/windows_multiple_workers' of https://github.com/…

a3d0d33

…FrsECM/LitServe into bugfix/windows_multiple_workers

[pre-commit.ci] auto fixes from pre-commit.com hooks

84e15b1

for more information, see https://pre-commit.ci

Move pid detection to LitLoop for less intrusivity

261f866

[pre-commit.ci] auto fixes from pre-commit.com hooks

56e8718

for more information, see https://pre-commit.ci

Revert changes in CICD

75d1dda

FrsECM requested review from aniketmaurya and justusschock April 30, 2025 11:47

FrsECM mentioned this pull request Apr 30, 2025

[Windows] OSError when multiple workers #384

Closed

Remove lock (useless)

8d63735

aniketmaurya reviewed Apr 30, 2025

View reviewed changes

src/litserve/server.py Outdated Show resolved Hide resolved

src/litserve/loops/base.py Show resolved Hide resolved

aniketmaurya linked an issue Apr 30, 2025 that may be closed by this pull request

[Windows] OSError when multiple workers #384

Closed

aniketmaurya approved these changes Apr 30, 2025

View reviewed changes

src/litserve/server.py Outdated Show resolved Hide resolved

src/litserve/server.py Outdated Show resolved Hide resolved

Apply suggestions from code review

2daba46

aniketmaurya merged commit c7d8d2f into Lightning-AI:main Apr 30, 2025
20 of 21 checks passed

aniketmaurya linked an issue Apr 30, 2025 that may be closed by this pull request

[Windows] Server Hangs while closing #372

Closed

aniketmaurya mentioned this pull request Apr 30, 2025

[Windows] Server Hangs while closing #372

Closed

Conversation

FrsECM commented Dec 4, 2024 • edited by aniketmaurya Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

PR review

Did you have fun?

Uh oh!

codecov bot commented Dec 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

aniketmaurya commented Dec 4, 2024

Uh oh!

FrsECM commented Dec 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FrsECM commented Dec 5, 2024

Uh oh!

FrsECM commented Dec 6, 2024

Uh oh!

aniketmaurya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FrsECM commented Dec 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aniketmaurya commented Dec 12, 2024

Uh oh!

FrsECM commented Dec 12, 2024 via email

Uh oh!

FrsECM commented Dec 13, 2024

Uh oh!

FrsECM commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

aniketmaurya commented Jan 5, 2025

Uh oh!

FrsECM commented Jan 6, 2025 via email

Uh oh!

Uh oh!

Uh oh!

RaiaN commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FrsECM commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FrsECM commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FrsECM commented Feb 28, 2025

Uh oh!

bhimrazy commented Feb 28, 2025

Uh oh!

bhimrazy commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FrsECM commented Apr 8, 2025

Uh oh!

FrsECM commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aniketmaurya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aniketmaurya commented Apr 30, 2025

Uh oh!

aniketmaurya commented Apr 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

FrsECM commented Dec 4, 2024 •

edited by aniketmaurya

Loading

codecov bot commented Dec 4, 2024 •

edited

Loading

FrsECM commented Dec 4, 2024 •

edited

Loading

FrsECM commented Dec 8, 2024 •

edited

Loading

FrsECM commented Dec 13, 2024 •

edited

Loading

RaiaN commented Jan 23, 2025 •

edited

Loading

FrsECM commented Jan 24, 2025 •

edited

Loading

FrsECM commented Feb 5, 2025 •

edited

Loading

bhimrazy commented Mar 31, 2025 •

edited

Loading

FrsECM commented Apr 29, 2025 •

edited

Loading