Question about Add support for diff-patching filter lists

[Nomes77]

When Add support for diff-patching filter lists is good and well implemented in the main releases, do you then welcome it if filterlist maintainers also implement it. Or is it not necessary?

Context:

• KOR-1: | 447GB/month
• NLD-0: | 80GB/month
• NOR-0: | 81GB/month
• RUS-0: | 4.187TB/month

[gorhill]

do you then welcome it if filterlist maintainers also implement it

Sure, if the filter list contains all that is needed according to spec, uBO will try to patch the list.

[Nomes77]

That is a quick reaction.
As you can see in our EasyDutchCDN we applied it.
May I propose then the following:

'0' is the contentURL, which is just the normal list
'1-3' are cdnURLs, which are all minified

0. https://easydutch-ubo.github.io/EasyDutch/EasyDutch.txt
1. https://easydutchcdn.pages.dev/EasyDutch.all.txt
2. https://cdn.jsdelivr.net/gh/EasyDutch-uBO/EasyDutchCDN@main/EasyDutch.all.txt
3. https://cdn.statically.io/gh/EasyDutch-uBO/EasyDutchCDN/main/EasyDutch.all.txt

(See: "Note on Data usage" in https://github.com/orgs/EasyDutch-uBO/discussions/44)

[krystian3w]

Maybe someone made script in bash/python for community - compatible with mechanisms like GitHub Actions.

[gorhill]

I've updated assets.dev.json with your new URLs, will see if this works. I am still addressing small issues. So for now only the dev build will try to fetch these new URLs, I will update the ones in assets.json once the new updater reaches stable.

[dportvine]

How to do it with RU AdList ?

[Nomes77]

To use EasyDutch as an example.

1. make the file tools/RuAdList-uBO.template:

! Title: RU AdList for uBO only
! Last modified: %timestamp%
! Expires: 2 days (фильтр обновляется автоматически каждые 2 дня)
! Diff-Path: %diffpath%#ruadlistubo
! Diff-Name: ruadlistubo
! Diff-Expires: 6 hours
! Homepage: https://forums.lanik.us/viewforum.php?f=102
!
! Комбинированная версия списков фильтров RU AdList против рекламы
! и экспериментальных списков RU AdList CSS Fixes и RU AdList JS Fixes
! На данный момент нормально работает только в uBlock Origin
!

%include ruadlist:advblock/adservers.txt%

%include ruadlist:advblock/first_level.txt%

%include ruadlist:advblock/general_block.txt%

%include ruadlist:advblock/general_hide.txt%

%include ruadlist:advblock/popup.txt%

%include ruadlist:advblock/specific_antisocial.txt%

%include ruadlist:advblock/specific_block.txt%

%include ruadlist:advblock/specific_hide.txt%

%include ruadlist:advblock/specific_special.txt%

%include ruadlist:advblock/thirdparty.txt%

%include ruadlist:advblock/whitelist.txt%

%include ruadlist:css-fixes-experimental.txt%

%include ruadlist:js-fixes-experimental.txt%

%include ruadlist:AWRL-non-sync.txt%

2. Add make-diffpatch.sh to the same folder
3. Add make-list.js to the same folder (please copy the file from uAssets, because I chanced some things for our specific (EasyDutch) needs.).
4. Add update-diffpatches.sh and change line 36 from git clone -q --single-branch --branch $PREVIOUS_VERSION --depth=1 https://github.com/EasyDutch-uBO/EasyDutch.git "$OLD_REPO" 2>/dev/null to git clone -q --single-branch --branch $PREVIOUS_VERSION --depth=1 https://github.com/easylist/ruadlist.git "$OLD_REPO" 2>/dev/null
5. Make a gh-pages branch in which you add a empty file called: RuAdList-uBO.min.txt
   Also add a empy file called version and make the folder patches by adding patches/README.md.
6. In the master branch add the following under .github/workflows/main.yml:

name: Deploy to gh-pages branch

on:
  push:
    branches:
      - master
    paths:
      - 'RuAdList-uBO.txt'
      - 'advblock/**'
      - 'css-fixes-experimental.txt'
      - 'js-fixes-experimental.txt'
      - 'AWRL-non-sync.txt'
      - 'tools/RuAdList-uBO.template'

# https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#example-only-cancel-in-progress-jobs-or-runs-for-the-current-workflow
concurrency: 
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read

jobs:
  publish:
    permissions:
      contents: write  # for Git to git push
    name: Publish lists
    runs-on: ubuntu-latest
    steps:
      - name: Clone RuAdList
        uses: actions/checkout@v3
        with:
          ref: gh-pages
      - name: Generate version string
        run: |
          TAGNAME=$(date -u "+%Y.%-m.%-d.")$(date -u "+%H*60+%M" | bc)
          echo "TAGNAME=$TAGNAME" >> $GITHUB_ENV
          echo "Version string is $TAGNAME"
      - name: Copy shell script from EasyList/RuAdList
        run: |
          TMPDIR=$(mktemp -d)
          git clone --depth=1 --single-branch --branch=master https://github.com/easylist/ruadlist.git $TMPDIR
          cp -v $TMPDIR/tools/make-diffpatch.sh .
          cp -v $TMPDIR/tools/update-diffpatches.sh .
      - name: Copy filter lists to gh-pages
        run: |
          TMPFILE=$(mktemp -d)
          git clone --depth=1 https://github.com/easylist/ruadlist.git $TMPFILE
          pushd $TMPFILE > /dev/null
          echo "*** RuAdList: Assembling RuAdList.txt"
          node ./tools/make-list.js in=./tools/RuAdList-uBO.template out=./RuAdList-uBO.min.txt minify=1
          popd > /dev/null
          cp $TMPFILE/RuAdList-uBO.min.txt RuAdList-uBO.min.txt
      - name: Patch last-updated field
        run: |
          DATE=$(date -Ru)
          for f in $(git diff --name-only); do
            STAT=$(git diff --numstat $f | sed -r '/^1\s+1\s+/d')
            if [[ -n $STAT ]]; then
              sed -ir "0,/^! Last updated: /s/^\(! Last updated: \)%timestamp%/\\1$DATE/" $f
            else
              git checkout -q $f
            fi
          done
      - name: Create new patch
        run: |
          chmod 755 ./make-diffpatch.sh
          ./make-diffpatch.sh "${{ env.TAGNAME }}"
      - name: Update existing patch
        run: |
          chmod 755 ./update-diffpatches.sh
          ./update-diffpatches.sh "${{ env.TAGNAME }}"
      - name: Commit changes (if any)
        run: |
          if [[ -z $(git diff --name-only --cached) ]]; then exit 0; fi
          git config user.name "GitHub-Actions"
          git config user.email "<>"
          git commit -m "Update modified filter lists to ${{ env.TAGNAME }}"
          git push origin gh-pages
          git tag ${{ env.TAGNAME }}
          git push origin ${{ env.TAGNAME }}

[gorhill]

Note that the branch does not have to be called gh-pages, it can be called whatever you want. I had gh-pages on my side because historically I had enabled "GitHub Pages" in repo settings, and that's the default name and I used that branch for distributing lists, but you do not need "GitHub Pages". just a different branch. It doesn't even have to be another branch, it can be a folder under the main repo. not a good idea in retrospect, a separate branch is better, but can be named whatever.

[krystian3w]

How to use this on only master branch?

[Nomes77]

I think just push to the master branch, so git push origin master in stead of git push origin gh-pages.
And of course put every file/folder I mentioned in point 5 in the master branch.

[gorhill]

Probably a disadvantage of using the master branch is that you always have to mind remote changes to the repo before pushing yours. Having another branch keeps all the remote changes confined to that branch, and also won't pollute the git history of the master branch, the git history of the content of the branch receiving the diff-able version of the list is not really of any interest, it could be wholly deleted and recreated in the future.

[gorhill]

I updated the bash scripts to encode the ! Diff-Name: value directly into the ! Diff_Path: value, so that there is no need for ! Diff-Name:. I feel this is a better approach than having to keep track of a separate field, and the name of a specific diff-block inside the patch fits well the semantic of the hash portion of a URL.

Modification to bash scripts and examples of the new way to encode diff-block name: f34a103

There need to be a transition, so for a while uBO itself will still use ! Diff-Name: if no hash portion can be found in ! Diff-Path:.

[dimisa-RUAdList]

.gitignore template - None?

[gorhill]

For uAssetsCDN, we don't have a .gitignore.

[dimisa-RUAdList]

Ok, done: https://github.com/dimisa-RUAdList/RUAdListCDN

What to do next?

[gorhill]

I will submit a pull request, this might take a while though, so probably tomorrow.

[JohnyP36]

In the uAssets tools/easylist/make-easylist.js is this code

    // Remove sublist header information: the importing list will provide its
    // own header.
    text = text.trim().replace(/^(?:!\s+[^\r\n]+?(?:\r\n|\n))+/s, '');

Which I also like to use to minify my filterlist.
The problem is that my sublist header info is

[uBlock Origin]
! Last modified: %timestamp%

I know I have to change the regex, but somehow I don't get it working.

Can some help me with this?

---

I'm sorry if this discussion is the wrong place to ask this question.

[Nomes77]

^(?:(?:\[|!\s+)[^\r\n]+?(?:\r\n|\n))+

[JohnyP36]

Thanks. I made this /^(?:(?:\[|!\s+[^0-9])[^\r\n]+?(?:\r\n|\n)){2}/s (used regex101)

[ghost]

I would suggest running these scripts through shellcheck, they may fail on spaces.

[krystian3w]

You means validator/compilator or fix comments by Grammarly (removal of the local English dialect or obvious typos)?

[ghost]

Bash variable quoting.

[gorhill]

Cool stuff. I am not a bash shell script guru, so I am just fumbling my way around with search engine without knowing whether what I am doing is the correct way to do it. This will be helpful.