Odd Issues When Running run.sh

[jmchilton]

A few of us have seen this issue and we haven't been able to figure what is going on.

@guerler captured the relevant log traces that match what I was seeing exactly. They are this on upload https://gist.github.com/guerler/c29e07ed9bf49e583dc14af19c44c407 possibly stemming from this https://gist.github.com/guerler/6d17b1be30ec5d3aa9c0192a091a342d on startup.

The workaround of skipping run.sh worked for me and @guerler but not for @jdavcs I think:

GALAXY_CONFIG_FILE=config/galaxy.yml uvicorn --app-dir lib --factory --port 8080 galaxy.webapps.galaxy.fast_factory:factory

[jdavcs]

I get the first trace: https://gist.github.com/guerler/c29e07ed9bf49e583dc14af19c44c407 regardless of whether I run uvicorn directly or gunicorn via run.sh. It does not prevent me from running galaxy though. It happens on uploads, but I did encounter it on other jobs too. I run Fedora 34.

[martenson]

Saw this on macos for upload and other jobs. Switching branches and recreating venv usually helped.

[mvdbeek]

I guess the big difference between uvicorn and gunicorn is the default preload setting.
Can you confirm that

gravity:
  app_server: gunicorn
  gunicorn:
    enable: true
    preload: false

fixes this ?

[jdavcs]

I've tested the above (ran via run.sh with and without it) - it doesn't fix the issue unfortunately.

[jarrah42]

FWIW I've been getting this issue for some time (running release_25.0 branch and dev). I just tried running without run.sh using the command line above, and the upload succeeded.

[mvdbeek]

Could someone on OSX please test #21031 (comment) ? @guerler @jmchilton ?

[jarrah42]

After getting it to upload once, I am unable to get it to work again. I've tried every combination I can think of, including the config file above, and running it directly with the uvicorn command. I've even tried removing and re-cloning the repo but to no avail.

[jmchilton]

Tried again this morning finally and run.sh works for me again on dev. Was that patch you mentioned part of gravity now @mvdbeek?

[mvdbeek]

No, not part of gravity ...

[jarrah42]

not for me unfortunately

[mvdbeek]

Could someone currently running into this issue post the contents of galaxy.yml, the operating system used, any sort of related proxy settings and the python version in question ?

[jarrah42]

I'm using the latest dev branch and have tried with no config, a copy of galaxy.yml.sample, and galaxy.yml.sample with gunicorn preload set to false (from above). I'm on Sequoia 17.5.1 using Python 3.13.2.

Here is the stack trace:

7.0.0.1:60814 - "PATCH /api/upload/resumable_upload/e0ef3a61e277457ca7061abed4b968eb HTTP/1.1" 204
[2025-10-28 08:53:47 -0400] [48262] [ERROR] Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
  |     yield
  |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/base.py", line 183, in __call__
  |     async with anyio.create_task_group() as task_group:
  |                ~~~~~~~~~~~~~~~~~~~~~~~^^
  |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 781, in __aexit__
  |     raise BaseExceptionGroup(
  |         "unhandled errors in a TaskGroup", self._exceptions
  |     ) from None
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    |     result = await app(  # type: ignore[func-returns-value]
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |         self.scope, self.receive, self.send
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |     )
    |     ^
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    |     return await self.app(scope, receive, send)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/fastapi/applications.py", line 1133, in __call__
    |     await super().__call__(scope, receive, send)
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/applications.py", line 113, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__
    |     raise exc
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__
    |     await self.app(scope, receive, _send)
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette_context/middleware/raw_middleware.py", line 94, in __call__
    |     await self.app(scope, receive, send_wrapper)
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/base.py", line 182, in __call__
    |     with recv_stream, send_stream, collapse_excgroups():
    |                                    ~~~~~~~~~~~~~~~~~~^^
    |   File "/Users/greg/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/contextlib.py", line 162, in __exit__
    |     self.gen.throw(value)
    |     ~~~~~~~~~~~~~~^^^^^^^
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
    |     raise exc
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/base.py", line 185, in __call__
    |     await response(scope, wrapped_receive, send)
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/base.py", line 229, in __call__
    |     await send({"type": "http.response.body", "body": chunk, "more_body": True})
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette_context/middleware/raw_middleware.py", line 83, in send_wrapper
    |     await send(message)
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 161, in _send
    |     await send(message)
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 500, in send
    |     output = self.conn.send(event=h11.Data(data=data))
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/h11/_connection.py", line 538, in send
    |     data_list = self.send_with_data_passthrough(event)
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/h11/_connection.py", line 571, in send_with_data_passthrough
    |     writer(event, data_list.append)
    |     ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/h11/_writers.py", line 65, in __call__
    |     self.send_data(event.data, write)
    |     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
    |   File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/h11/_writers.py", line 91, in send_data
    |     raise LocalProtocolError("Too much data for declared Content-Length")
    | h11._util.LocalProtocolError: Too much data for declared Content-Length
    +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.scope, self.receive, self.send
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/fastapi/applications.py", line 1133, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette_context/middleware/raw_middleware.py", line 94, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/base.py", line 182, in __call__
    with recv_stream, send_stream, collapse_excgroups():
                                   ~~~~~~~~~~~~~~~~~~^^
  File "/Users/greg/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
    raise exc
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/base.py", line 185, in __call__
    await response(scope, wrapped_receive, send)
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/base.py", line 229, in __call__
    await send({"type": "http.response.body", "body": chunk, "more_body": True})
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette_context/middleware/raw_middleware.py", line 83, in send_wrapper
    await send(message)
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 161, in _send
    await send(message)
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 500, in send
    output = self.conn.send(event=h11.Data(data=data))
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/h11/_connection.py", line 538, in send
    data_list = self.send_with_data_passthrough(event)
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/h11/_connection.py", line 571, in send_with_data_passthrough
    writer(event, data_list.append)
    ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/h11/_writers.py", line 65, in __call__
    self.send_data(event.data, write)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/Users/greg/Work/galaxy/.venv/lib/python3.13/site-packages/h11/_writers.py", line 91, in send_data
    raise LocalProtocolError("Too much data for declared Content-Length")
h11._util.LocalProtocolError: Too much data for declared Content-Length

galaxy.yml

[mvdbeek]

This should be fixed in #21201 but would appreciate confimation!