Incorrect SSH key permissions (0660) set by Weblate GUI causing Git operations to fail

[Autherain]

Describe the issue

When using Weblate's GUI to manage repositories, it creates SSH keys with incorrect permissions (0660) in the /app/data/ssh/ directory.

This causes SSH operations to fail with the following error:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0660 for '/app/data/ssh/id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/app/data/ssh/id_rsa": bad permissions
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0660 for '/app/data/ssh/id_ed25519' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/app/data/ssh/id_ed25519": bad permissions
git@gitlab.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
 (128) 

I already tried

• I've read and searched the documentation.
• I've searched for similar filed issues in this repository.

Steps to reproduce the behavior

Steps to Reproduce

1. Deploy Weblate 5.9 official Docker image in a Kubernetes cluster
2. Use Weblate GUI to manage repository settings
3. Attempt to perform Git operations using SSH
4. Observe the permission denied errors in logs

Note: Limited reproduction steps available as the exact trigger is not fully identified.

Expected behavior

Weblate should set the correct permissions (0600) when generating or managing SSH keys to ensure they work properly with Git operations.

Screenshots

No response

Exception traceback

How do you run Weblate?

Docker container

Weblate versions

5.9

Weblate deploy checks

Additional context

Current Workaround

To fix this issue, users need to manually access the container shell via kubectl and run:

chmod 600 /app/data/ssh/id_rsa
chmod 600 /app/data/ssh/id_ed25519

[nijel]

Something must have changed the permissions. Weblate creates the files correctly.

• Wide permissions (660) on a SSH key weblate#9995 (comment) explains what Weblate does
• Files are made group readable on the data volume docker#2439 shows some issues in Docker causing the permissions to be lost, but unfortunately gives no resolution to that

[RisingOpsNinja]

Check fsGroup and fsGroupChangePolicy settings in Pod's securityContext.

I'm not sure how you deployed weblate, but the weblate helm chart sets fsGroup and the Kubernetes cluster may sets fsGroupChangePolicy to Always by default, which causes the group permissions to be adjusted every time, when the volume is mounted (e.g. container restart).

Change fsGroupChangePolicy to OnRootMismatch should fix the issue in that case.

By default, Kubernetes recursively changes ownership and permissions for the contents of each volume to match the fsGroup specified in a Pod's securityContext when that volume is mounted.

fsGroupChangePolicy - fsGroupChangePolicy defines behavior for changing ownership and permission of the volume before being exposed inside a Pod. This field only applies to volume types that support fsGroup controlled ownership and permissions. This field has two possible values:

    OnRootMismatch: Only change permissions and ownership if the permission and the ownership of root directory does not match with expected permissions of the volume. This could help shorten the time it takes to change ownership and permission of a volume.
    Always: Always change permission and ownership of the volume when volume is mounted.

Source: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods

[Autherain]

That's a solution that i never saw coming to be honest ! Thank you very much :)

[Mart-Kuc]

Hi guys,

From my perspective, I don’t believe the issue is related to fsGroupChangePolicy. The default value is Always, meaning that every time a volume is mounted, Kubernetes recursively changes ownership and permissions of all files to match fsGroup (1000/weblate). But it has limitations and rules for this one.

If fsGroupChangePolicy is Always:

• All files and directories in the mounted volume will have their group ownership (chown) changed.
• Kubernetes does not automatically change file mode (chmod).
• However, if the volume is mounted as a shared volume (e.g., emptyDir, hostPath, or PVC with ReadWriteMany mode), the system ensures group members (fsGroup) have write access using the group’s write permission bit
• The only time permissions (chmod) are changed is for SetGid (g+s), and this applies only to directories.

If fsGroupChangePolicy is set to OnRootMismatch:

• If the root directory already belongs to the correct fsGroup, Kubernetes does nothing (chmod remains unchanged).
• If the root directory does not match fsGroup, Kubernetes will:
• Recursively change ownership (chown).
• Apply chmod g+s to directories.

TL;DR

• fsGroup changes only group ownership (chown).
• fsGroup does not change file permissions (chmod)
• Files created with 0600 stay 0600 even with fsGroupChangePolicy to default value Always

My question to @Autherain:
Are you using a PVC (CSI driver) with ReadWriteMany mode?

Of course, feel free to correct me if I’m wrong!

[RisingOpsNinja]

Hi,

fsGroup does change file permission if necessary. There may be more to it to trigger the issue actually, but OnRootMismatch fixed the issue for us persistently.

This way you can reproduce the behavior of fsGroup with your Weblate deployment:

Prerequisites: CSI driver has the capability of fsGroup (e.g. a hostpath PV has not). I have a PVC with accessMode ReadWriteOnce btw.

1. Check permission in weblate:

group has no access at all

weblate@weblate-9562d46645-sbd9g:/app/data/ssh$ ls -l id_ed25519 id_rsa
-rw------- 1 weblate weblate  399 Sep 11 07:05 id_ed25519
-rw------- 1 weblate weblate 3369 Jul  2  2024 id_rsa

2. Apply this pod manifest with fsGroup and fsGroupChangePolicy set to Always.

You may need to adjust the PVC name, so the Pod mounts it.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: busybox
  name: busybox
spec:
  restartPolicy: Never
  securityContext:
    fsGroup: 1000
    fsGroupChangePolicy: Always
  containers:
  - image: busybox
    name: busybox
    volumeMounts:
    - mountPath: /app/data
      name: weblate-data
  volumes:
  - name: weblate-data
    persistentVolumeClaim:
      claimName: weblate-data

4. Check permission again:

group is read and write now which ssh rejects

weblate@weblate-9562d46645-sbd9g:/app/data/ssh$ ls -l id_ed25519 id_rsa
-rw-rw---- 1 weblate weblate  444 Sep 11 07:05 id_ed25519
-rw-rw---- 1 weblate weblate 3286 Jul  2  2024 id_rsa

[sinnnie]

Happens on Weblate 5.9.2 and happened on previous versions as well, no clear consistency and no specific reason why, sometimes it shows up several times per day and other times no signs of it for months.