Merge branch 'master' into prune-timely-by-interval

Author: ThomasWaldmann

.coveragerc

@@ -1,56 +1,54 @@
 <!--
 Thank you for reporting an issue.
 
-*IMPORTANT* -  *before* creating a new issue please look around:
- - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
+*IMPORTANT* – Before creating a new issue, please look around:
+ - BorgBackup documentation: https://borgbackup.readthedocs.io/en/stable/index.html
  - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
- and
- - open issues in GitHub tracker: https://github.com/borgbackup/borg/issues
-  
+ - Open issues in the GitHub tracker: https://github.com/borgbackup/borg/issues
+
 If you cannot find a similar problem, then create a new issue.
 
 Please fill in as much of the template as possible.
 -->
 
-## Have you checked borgbackup docs, FAQ, and open GitHub issues?
+## Have you checked the BorgBackup docs, FAQ, and open GitHub issues?
 
 No
 
-## Is this a BUG / ISSUE report or a QUESTION?
+## Is this a bug/issue report or a question?
 
-Invalid
+Bug/Issue/Question
 
-## System information. For client/server mode post info for both machines.
+## System information. For client/server mode, post info for both machines.
 
-#### Your borg version (borg -V).
+#### Your Borg version (borg -V).
 
 #### Operating system (distribution) and version.
 
-#### Hardware / network configuration, and filesystems used.
+#### Hardware/network configuration and filesystems used.
 
-#### How much data is handled by borg?
+#### How much data is handled by Borg?
 
-#### Full borg commandline that lead to the problem (leave away excludes and passwords)
+#### Full Borg command line that led to the problem (leave out excludes and passwords).
 
 
 ## Describe the problem you're observing.
 
 #### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
 
-#### Include any warning/errors/backtraces from the system logs
+#### Include any warnings/errors/backtraces from the system logs
 
 <!--
 
-If this complaint relates to borg performance, please include CRUD benchmark
+If this complaint relates to Borg performance, please include CRUD benchmark
 results and any steps you took to troubleshoot.
-How to run benchmark: http://borgbackup.readthedocs.io/en/stable/usage/benchmark.html
+How to run the benchmark: https://borgbackup.readthedocs.io/en/stable/usage/benchmark.html
 
-*IMPORTANT* - Please mark logs and text output from terminal commands 
-or else GitHub will not display them correctly. 
+*IMPORTANT* – Please mark logs and terminal command output, otherwise GitHub will not display them correctly.
 An example is provided below.
 
 Example:
 ```
-this is an example how log text should be marked (wrap it with ```)
+this is an example of how log text should be marked (wrap it with ```)
 ```
 -->

.github/ISSUE_TEMPLATE.md

@@ -1,8 +1,8 @@
-Thank you for contributing code to Borg, your help is appreciated!
+Thank you for contributing code to Borg; your help is appreciated!
 
-Please, before you submit a pull request, make sure it complies with the
-guidelines given in our documentation:
+Before you submit a pull request, please make sure it complies with the
+guidelines in our documentation:
 
 https://borgbackup.readthedocs.io/en/latest/development.html#contributions
 
-**Please remove all above text before submitting your pull request.**
+**Please remove the text above before submitting your pull request.**

.github/PULL_REQUEST_TEMPLATE

@@ -1,5 +1,5 @@
 # https://black.readthedocs.io/en/stable/integrations/github_actions.html#usage
-# see also what we use locally, requirements.d/codestyle.txt - should be the same version here.
+# See also what we use locally in requirements.d/codestyle.txt — this should be the same version here.
 
 name: Lint
 

.github/workflows/black.yaml

@@ -84,6 +84,9 @@ jobs:
             - os: ubuntu-24.04
               python-version: '3.13'
               toxenv: py313-fuse3
+            - os: ubuntu-24.04
+              python-version: '3.14-dev'
+              toxenv: py314-fuse3
 
     env:
       TOXENV: ${{ matrix.toxenv }}
@@ -94,7 +97,7 @@ jobs:
     steps:
     - uses: actions/checkout@v4
       with:
-        # just fetching 1 commit is not enough for setuptools-scm, so we fetch all
+        # Just fetching one commit is not enough for setuptools-scm, so we fetch all.
         fetch-depth: 0
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v5
@@ -117,7 +120,60 @@ jobs:
         sudo apt-get install -y libfuse-dev fuse || true  # Required for Python llfuse module
         sudo apt-get install -y libfuse3-dev fuse3 || true  # Required for Python pyfuse3 module
         sudo apt-get install -y bash zsh fish  # for shell completion tests
-
+        sudo apt-get install -y rclone openssh-server curl
+    - name: Configure OpenSSH SFTP server (test only)
+      run: |
+        sudo mkdir -p /run/sshd
+        sudo useradd -m -s /bin/bash sftpuser || true
+        # Create SSH key for the CI user and authorize it for sftpuser
+        mkdir -p ~/.ssh
+        chmod 700 ~/.ssh
+        test -f ~/.ssh/id_ed25519 || ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519
+        sudo mkdir -p /home/sftpuser/.ssh
+        sudo chmod 700 /home/sftpuser/.ssh
+        sudo cp ~/.ssh/id_ed25519.pub /home/sftpuser/.ssh/authorized_keys
+        sudo chown -R sftpuser:sftpuser /home/sftpuser/.ssh
+        sudo chmod 600 /home/sftpuser/.ssh/authorized_keys
+        # Allow publickey auth and enable Subsystem sftp
+        sudo sed -i 's/^#\?PasswordAuthentication .*/PasswordAuthentication no/' /etc/ssh/sshd_config
+        sudo sed -i 's/^#\?PubkeyAuthentication .*/PubkeyAuthentication yes/' /etc/ssh/sshd_config
+        if ! grep -q '^Subsystem sftp' /etc/ssh/sshd_config; then echo 'Subsystem sftp /usr/lib/openssh/sftp-server' | sudo tee -a /etc/ssh/sshd_config; fi
+        # Ensure host keys exist to avoid slow generation on first sshd start
+        sudo ssh-keygen -A
+        # Start sshd (listen on default 22 inside runner)
+        sudo /usr/sbin/sshd -D &
+        # Add host key to known_hosts so paramiko trusts it
+        ssh-keyscan -H localhost 127.0.0.1 | tee -a ~/.ssh/known_hosts
+        # Start ssh-agent and add our key so paramiko can use the agent
+        eval "$(ssh-agent -s)"
+        ssh-add ~/.ssh/id_ed25519
+        # Export SFTP test URL for tox via GITHUB_ENV
+        echo "BORG_TEST_SFTP_REPO=sftp://sftpuser@localhost:22/borg/sftp-repo" >> $GITHUB_ENV
+    - name: Install and configure MinIO S3 server (test only)
+      run: |
+        set -e
+        arch=$(uname -m)
+        case "$arch" in
+          x86_64|amd64) srv_url=https://dl.min.io/server/minio/release/linux-amd64/minio; cli_url=https://dl.min.io/client/mc/release/linux-amd64/mc ;;
+          aarch64|arm64) srv_url=https://dl.min.io/server/minio/release/linux-arm64/minio; cli_url=https://dl.min.io/client/mc/release/linux-arm64/mc ;;
+          *) echo "Unsupported arch: $arch"; exit 1 ;;
+        esac
+        curl -fsSL -o /usr/local/bin/minio "$srv_url"
+        curl -fsSL -o /usr/local/bin/mc "$cli_url"
+        sudo chmod +x /usr/local/bin/minio /usr/local/bin/mc
+        export PATH=/usr/local/bin:$PATH
+        # Start MinIO on :9000 with default credentials (minioadmin/minioadmin)
+        MINIO_DIR="$GITHUB_WORKSPACE/.minio-data"
+        MINIO_LOG="$GITHUB_WORKSPACE/.minio.log"
+        mkdir -p "$MINIO_DIR"
+        nohup minio server "$MINIO_DIR" --address ":9000" >"$MINIO_LOG" 2>&1 &
+        # Wait for MinIO port to be ready
+        for i in $(seq 1 60); do (echo > /dev/tcp/127.0.0.1/9000) >/dev/null 2>&1 && break; sleep 1; done
+        # Configure client and create bucket
+        mc alias set local http://127.0.0.1:9000 minioadmin minioadmin
+        mc mb --ignore-existing local/borg
+        # Export S3 test URL for tox via GITHUB_ENV
+        echo "BORG_TEST_S3_REPO=s3:minioadmin:minioadmin@http://127.0.0.1:9000/borg/s3-repo" >> $GITHUB_ENV
     - name: Install Python requirements
       run: |
         python -m pip install --upgrade pip setuptools wheel
@@ -150,7 +206,7 @@ jobs:
         include:
             - os: macos-14
               python-version: '3.11'
-              toxenv: py311-none  # note: no fuse testing, due to #6099, see also #6196.
+              toxenv: py311-none  # Note: no FUSE testing due to #6099; see also #6196.
 
     env:
       # Configure pkg-config to use OpenSSL from Homebrew
@@ -163,7 +219,7 @@ jobs:
     steps:
     - uses: actions/checkout@v4
       with:
-        # just fetching 1 commit is not enough for setuptools-scm, so we fetch all
+        # Just fetching one commit is not enough for setuptools-scm, so we fetch all
         fetch-depth: 0
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v5
@@ -187,21 +243,21 @@ jobs:
       run: |
         python -m pip install --upgrade pip setuptools wheel
         pip install -r requirements.d/development.txt
-    - name: Install borgbackup
+    - name: Install BorgBackup
       env:
-        # we already have that in the global env, but something is broken and overwrites that.
-        # so, set it here, again.
+        # We already have this in the global environment, but something overrides it.
+        # So set it here again.
         PKG_CONFIG_PATH: "/opt/homebrew/opt/openssl@3.0/lib/pkgconfig:$PKG_CONFIG_PATH"
       run: |
         pip install -ve .
-    - name: run tox env
+    - name: Run pytest via tox
       env:
-        # we already have that in the global env, but something is broken and overwrites that.
-        # so, set it here, again.
+        # We already have this in the global environment, but something overrides it.
+        # So set it here again.
         PKG_CONFIG_PATH: "/opt/homebrew/opt/openssl@3.0/lib/pkgconfig:$PKG_CONFIG_PATH"
         XDISTN: "6"
       run: |
-        # do not use fakeroot, but run as root. avoids the dreaded EISDIR sporadic failures. see #2482.
+        # Do not use fakeroot; run as root. Avoids the dreaded sporadic EISDIR failures; see #2482.
         #sudo -E bash -c "tox -e py"
         tox --skip-missing-interpreters
     - name: Upload coverage to Codecov
@@ -215,7 +271,7 @@ jobs:
 
   windows:
 
-    if: false  # can be used to temporary disable build
+    if: false  # can be used to temporarily disable the build
     runs-on: windows-latest
     timeout-minutes: 120
     needs: linux

.github/workflows/ci.yml

@@ -31,7 +31,7 @@ jobs:
     - name: Checkout repository
       uses: actions/checkout@v4
       with:
-        # just fetching 1 commit is not enough for setuptools-scm, so we fetch all
+        # Just fetching one commit is not enough for setuptools-scm, so we fetch all.
         fetch-depth: 0
     - name: Set up Python
       uses: actions/setup-python@v5

.github/workflows/codeql-analysis.yml

@@ -2,14 +2,13 @@ MANIFEST
 docs/_build
 build
 dist
-external
-borg-env
 .tox
 src/borg/compress.c
 src/borg/crypto/low_level.c
-src/borg/hashindex.c
 src/borg/item.c
-src/borg/chunker.c
+src/borg/chunkers/buzhash.c
+src/borg/chunkers/buzhash64.c
+src/borg/chunkers/reader.c
 src/borg/checksums.c
 src/borg/platform/darwin.c
 src/borg/platform/freebsd.c
@@ -23,12 +22,9 @@ src/borg/_version.py
 *.pyd
 *.so
 .idea/
-.cache/
+.junie/
 .vscode/
-borg.build/
-borg.dist/
 borg.exe
 .coverage
 .coverage.*
 .vagrant
-.eggs

.gitignore

@@ -1,5 +1,5 @@
-E-mail addresses listed here are not intended for support, please see
-the `support section`_ instead.
+Email addresses listed here are not intended for support.
+Please see the `support section`_ instead.
 
 .. _support section: https://borgbackup.readthedocs.io/en/stable/support.html
 

AUTHORS

@@ -5,8 +5,8 @@ brew 'xxhash'
 brew 'openssl@3.0'
 
 # osxfuse (aka macFUSE) is only required for "borg mount",
-# but won't work on github actions' workers.
-# it requires installing a kernel extension, so some users
+# but won't work on GitHub Actions' workers.
+# It requires installing a kernel extension, so some users
 # may want it and some won't.
 
 #cask 'osxfuse'

Brewfile

@@ -1,6 +1,6 @@
-# stuff we need to include into the sdist is handled automatically by
+# The files we need to include in the sdist are handled automatically by
 # setuptools_scm - it includes all git-committed files.
-# but we want to exclude some committed files/dirs not needed in the sdist:
+# But we want to exclude some committed files/directories not needed in the sdist:
 exclude .editorconfig .gitattributes .gitignore .mailmap Vagrantfile
 prune .github
 include src/borg/platform/darwin.c src/borg/platform/freebsd.c src/borg/platform/linux.c src/borg/platform/posix.c