Repository corruption after server crash causes Weblate to fail reset/repair with 500 error

[lschaepers]

Describe the issue

After a server crash and restart, one Weblate component ended up in a broken repository state which cannot be recovered using the Weblate UI or API.

The Git repository inside Weblate appears to be corrupted (empty object files). After this happens, all repository operations fail, including Reset and discard, which returns a 500 Internal Server Error.

I already tried

• I've read and searched the documentation.
• I've searched for similar filed issues in this repository.

Steps to reproduce the behavior

Unfortunately, the issue is not reliably reproducible.

The problem occurred after a server crash and restart while (or so I assume) Weblate was performing
repository operations (merge/rebase due to diverged repositories).

After the crash, the repository inside Weblate was left in a corrupted state
(empty Git object files), which could not be recovered using the Weblate UI or API.

Edit / addendum: Just like @dylankiss we upgraded (from 5.14.3) to 5.15 before we fully restarted the weblate instance.

Expected behavior

Observed behavior

• Component shows repository errors:
  • Could not update repository
  • Could not merge repository
  • Could not push repository
• API endpoint /api/components///repository/ returns HTTP 500
• Repository Maintenance actions (Reset and discard / Reset and reapply) fail with Internal Server Error
• Weblate UI no longer shows Update / Push buttons

Relevant log output:

fatal: Invalid revision range ..origin/dev/app_1.0 (128)

error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
fatal: bad object HEAD

Python traceback during “Reset and discard”:

weblate.vcs.base.RepositoryError:
error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
fatal: bad object HEAD (128)

Expected behavior

When the repository is corrupted and HEAD is unreadable:

• Weblate should either:
  • detect this condition and mark the repository as irrecoverable
  • or make Reset and discard more robust by fully deleting and re-cloning the repository without relying on last_revision

Currently, Weblate attempts to read last_revision even though HEAD is invalid, which makes recovery impossible via the UI.

Screenshots

After trying to execute git operations in UI

Exception traceback

weblate-1   | [2025-12-19 08:05:52,713: INFO/354] pct/devapp_1-0: updating repository
weblate-1   | [2025-12-19 08:05:53,490: WARNING/354] Could not update the repository: RepositoryError: error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | fatal: cannot read existing object info d7c34d409253c6de07be6fffee0b9f729d18fb4e
weblate-1   | fatal: fetch-pack: invalid index-pack output
weblate-1   |  (128)
weblate-1   | 2025-12-19 08:05:53,490 INFO reaped unknown pid 26227 (exit status 141)
weblate-1   | 2025-12-19 08:05:53,490 INFO reaped unknown pid 26229 (exit status 128)
weblate-1   | 10.2.118.24 - - [19/Dec/2025:08:05:53 +0000] "POST /update/pct/devapp_1-0/ HTTP/1.1" 302 5 "https://weblate.example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0"
weblate-1   | 10.2.118.24 - - [19/Dec/2025:08:05:53 +0000] "GET /progress/pct/devapp_1-0/?info=1 HTTP/1.1" 302 5 "https://weblate.example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0"
weblate-1   | 10.2.118.24 - - [19/Dec/2025:08:05:53 +0000] "GET /projects/pct/devapp_1-0/ HTTP/1.1" 200 26159 "https://weblate.example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0"
weblate-1   | [2025-12-19 08:06:15,448: WARNING/25546] Could check push needed: RepositoryError: error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | fatal: Invalid revision range origin/dev/app_1.0..
weblate-1   |  (128)
weblate-1   | [2025-12-19 08:06:15,467: WARNING/25546] Could check push needed: RepositoryError: error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | fatal: Invalid revision range origin/dev/app_1.0..
weblate-1   |  (128)
weblate-1   | [2025-12-19 08:06:15,476: WARNING/25546] Could check merge needed: RepositoryError: error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | fatal: Invalid revision range ..origin/dev/app_1.0
weblate-1   |  (128)
weblate-1   | 10.2.118.24 - - [19/Dec/2025:08:06:15 +0000] "GET /js/git/pct/devapp_1-0/ HTTP/1.1" 200 4749 "https://weblate.example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0"
weblate-1   | [2025-12-19 08:06:17,051: INFO/MainProcess] Task weblate.utils.tasks.heartbeat[fefe7997-d656-4658-b5d9-d91a2025a55e] received
weblate-1   | [2025-12-19 08:06:31,430: ERROR/25546] Internal Server Error: /reset/pct/devapp_1-0/
weblate-1   | Traceback (most recent call last):
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/django/core/handlers/exception.py", line 55, in inner
weblate-1   |     response = get_response(request)
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/django/core/handlers/base.py", line 197, in _get_response
weblate-1   |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/django/contrib/auth/decorators.py", line 59, in _view_wrapper
weblate-1   |     return view_func(request, *args, **kwargs)
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/django/views/decorators/http.py", line 64, in inner
weblate-1   |     return func(request, *args, **kwargs)
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/weblate/trans/views/git.py", line 97, in reset
weblate-1   |     result = execute_locked(
weblate-1   |         request,
weblate-1   |     ...<6 lines>...
weblate-1   |         keep_changes="keep_changes" in request.POST,
weblate-1   |     )
weblate-1   |   File "/opt/python/cpython-3.14.2-linux-x86_64-gnu/lib/python3.14/contextlib.py", line 85, in inner
weblate-1   |     return func(*args, **kwds)
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/weblate/trans/views/git.py", line 33, in execute_locked
weblate-1   |     result = call(*args, **kwargs)
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/weblate/trans/models/component.py", line 234, in on_link_wrapper
weblate-1   |     return func(self, *args, **kwargs)
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/weblate/trans/models/component.py", line 2081, in do_reset
weblate-1   |     previous_head = self.repository.last_revision
weblate-1   |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/django/utils/functional.py", line 47, in __get__
weblate-1   |     res = instance.__dict__[self.name] = self.func(instance)
weblate-1   |                                          ~~~~~~~~~^^^^^^^^^^
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/weblate/vcs/base.py", line 335, in last_revision
weblate-1   |     return self.get_last_revision()
weblate-1   |            ~~~~~~~~~~~~~~~~~~~~~~^^
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/weblate/vcs/base.py", line 338, in get_last_revision
weblate-1   |     return self.execute(self._cmd_last_revision, needs_lock=False, merge_err=False)
weblate-1   |            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/weblate/vcs/base.py", line 306, in execute
weblate-1   |     self.last_output = self._popen(
weblate-1   |                        ~~~~~~~~~~~^
weblate-1   |         args,
weblate-1   |         ^^^^^
weblate-1   |     ...<5 lines>...
weblate-1   |         environment=environment,
weblate-1   |         ^^^^^^^^^^^^^^^^^^^^^^^^
weblate-1   |     )
weblate-1   |     ^
weblate-1   |   File "/app/venv/lib/python3.14/site-packages/weblate/vcs/base.py", line 276, in _popen
weblate-1   |     raise RepositoryError(process.returncode, errormessage)
weblate-1   | weblate.vcs.base.RepositoryError: error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | error: object file .git/objects/d7/c34d409253c6de07be6fffee0b9f729d18fb4e is empty
weblate-1   | fatal: bad object HEAD
weblate-1   |  (128)
weblate-1   | 10.2.118.24 - - [19/Dec/2025:08:06:31 +0000] "POST /reset/pct/devapp_1-0/ HTTP/1.1" 500 25599 "https://weblate.example.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0"

How do you run Weblate?

Docker container

Weblate versions

5.15

Weblate deploy checks

Additional context

• The issue occurred after a server crash / restart (likely during a Git operation such as merge or rebase).
• A new component with the same configuration works fine.
• Cloning the upstream Git repository manually shows no issues.
• The broken component repository had to be recreated to continue working.

[dylankiss]

We have the exact same issue after testing an upgrade to 5.15.1 on 2 components.

[drzraf]

Got the same problem after loosing my /var/lib/docker directory. Which manage.py / weblate cli command workaround did you used?

[lschaepers]

@drzraf What workaround are you referring to? Are you referring to "The broken component repository had to be recreated to continue working."?

[drzraf]

Actually got it via a mix of :

1. Update This will update the Weblate repository with the upstream changes. which recreate/repopulate the Git repository
2. Reset and reapply Reset the Weblate repository and reapply translations which flush the database into the repository.

I wish these could be better documented under the backup section of the documentation. These are all powerful and technical operations but it's not obvious at first glance how the data is going back & forth under the hood.