ci(lint): expand jscpd to use branch comparison aws#5862

## Problem
Follow up to: aws#5833

A static threshold is useful, but if the threshold fails, we want to be
able to see what the new duplicated code is.

## Solution
Run `jscpd` on the current branch, and if it is above the set threshold,
we fail (same as before).

Now, we also filter the resulting report to contain only clones from the
files changed with respect to the base branch. Then, if the filtered
report is non-empty, i.e. one of the changed files has a clone, fail and
display the clones in CI.

Adding this extra step allows us to know when we are adding a clone that
was not otherwise in the codebase. It also allows us to see this clone
in CI, before the PR is merged, giving the developer a chance to remove
it.

As an example, see commit
(aws@a8e4860)
where a copy-pasted function is added, and notice that CI fails and
displays only the section of the report relevant to this clone. Then,
once we remove this work in the following commit
(aws@53d3973),
the check passes.

Included in this PR, is that it runs only on `pull_request` and no
longer on `push`.
## Notes
- In developing this, it became apparent that `jscpd` is not perfect and
was not able to detect the clone added in
(aws@e7357c0).
Will need to dig deeper into their algorithm (based on
https://en.wikipedia.org/wiki/Rabin%E2%80%93Karp_algorithm) and
implementation (https://github.com/kucherenko/jscpdto) to understand why
this is missed, and if it presents a larger problem.
- Also, if a clone already exists in a file that is changed in the PR,
this check will fail. In this case, the options are to ignore the check,
mark the case as an exception, or fix the clone. To encourage developers
to not repeatedly ignore the check, we still employ a static threshold
via aws#5833.
- If the branch hasn't been rebased up to date, it can pick up clones
from unrelated changes.

## Future Work
- We want to lower the threshold as much as possible by removing
duplicates and marking exceptions when necessary.
- More experience using this tool will help us find a `minLines`
argument that provides the desired sensitivity. This argument allows us
to ignore clones under `minLines` in length. Right now it is 3, which is
likely too granular, but better to start there and loosen as needed.
- Right now the CI output of the clones is very verbose. Digging through
this could be time consuming, especially when many clones are added.
Could be worth filtering it to provide a more ergonomic experience.

Author: Hweinstock

.github/workflows/copyPasteDetection.yml

@@ -0,0 +1,86 @@
+# github actions: https://docs.github.com/en/actions/use-cases-and-examples/building-and-testing/building-and-testing-nodejs
+# setup-node: https://github.com/actions/setup-node
+
+name: Copy-Paste Detection
+
+on:
+    pull_request:
+        branches: [master, feature/*, staging]
+
+jobs:
+    jscpd:
+        runs-on: ubuntu-latest
+        strategy:
+            matrix:
+                node-version: [18.x]
+        env:
+            NODE_OPTIONS: '--max-old-space-size=8192'
+
+        steps:
+            - uses: actions/checkout@v4
+              with:
+                  fetch-depth: 0
+
+            - name: Use Node.js ${{ matrix.node-version }}
+              uses: actions/setup-node@v4
+              with:
+                  node-version: ${{ matrix.node-version }}
+
+            - name: Fetch fork upstream
+              run: |
+                  git remote add forkUpstream https://github.com/${{ github.event.pull_request.head.repo.full_name }}  # URL of the fork
+                  git fetch forkUpstream  # Fetch fork
+
+            - name: Determine base and target branches for comparison.
+              run: |
+                  echo "CURRENT_BRANCH=${{ github.head_ref }}" >> $GITHUB_ENV
+                  echo "TARGET_BRANCH=${{ github.event.pull_request.base.ref }}" >> $GITHUB_ENV
+            - run: git diff --name-only origin/$TARGET_BRANCH forkUpstream/$CURRENT_BRANCH > diff_output.txt
+            - run: |
+                  npm install -g jscpd
+
+            - run: jscpd --config "$GITHUB_WORKSPACE/.github/workflows/jscpd.json"
+
+            - if: always()
+              uses: actions/upload-artifact@v4
+              with:
+                  name: unfiltered-jscpd-report
+                  path: ./jscpd-report.json
+
+            - name: Filter jscpd report for changed files
+              run: |
+                  if [ ! -f ./jscpd-report.json ]; then
+                    echo "jscpd-report.json not found"
+                    exit 1
+                  fi
+                  echo "Filtering jscpd report for changed files..."
+                  CHANGED_FILES=$(jq -R -s -c 'split("\n")[:-1]' diff_output.txt)
+                  echo "Changed files: $CHANGED_FILES"
+                  jq --argjson changed_files "$CHANGED_FILES" '
+                  .duplicates | map(select(
+                      (.firstFile?.name as $fname | $changed_files | any(. == $fname)) or
+                      (.secondFile?.name as $sname | $changed_files | any(. == $sname))
+                  ))
+                  ' ./jscpd-report.json > filtered-jscpd-report.json
+                  cat filtered-jscpd-report.json
+
+            - name: Check for duplicates
+              run: |
+                  if [ $(wc -l < ./filtered-jscpd-report.json) -gt 1 ]; then
+                    echo "filtered_report_exists=true" >> $GITHUB_ENV
+                  else
+                    echo "filtered_report_exists=false" >> $GITHUB_ENV
+                  fi
+            - name: upload filtered report (if applicable)
+              if: env.filtered_report_exists == 'true'
+              uses: actions/upload-artifact@v4
+              with:
+                  name: filtered-jscpd-report
+                  path: ./filtered-jscpd-report.json
+
+            - name: Fail and log found duplicates.
+              if: env.filtered_report_exists == 'true'
+              run: |
+                  cat ./filtered-jscpd-report.json
+                  echo "Duplications found, failing the check."
+                  exit 1

.github/workflows/jscpd.json

@@ -1,7 +1,9 @@
 {
     "pattern": "packages/**/*.ts",
-    "ignore": ["**node_modules**", "**dist**"],
+    "ignore": ["**node_modules**", "**dist**", "*/scripts/*"],
     "gitignore": true,
-    "threshold": 1.34,
-    "minLines": 15
+    "threshold": 3.0,
+    "minLines": 3,
+    "output": "./",
+    "reporters": ["json"]
 }

.github/workflows/node.js.yml

@@ -55,24 +55,6 @@ jobs:
             - run: npm run testCompile
             - run: npm run lint
 
-    jscpd:
-        needs: lint-commits
-        runs-on: ubuntu-latest
-        strategy:
-            matrix:
-                node-version: [18.x]
-        env:
-            NODE_OPTIONS: '--max-old-space-size=8192'
-        steps:
-            - uses: actions/checkout@v4
-            - name: Use Node.js ${{ matrix.node-version }}
-              uses: actions/setup-node@v4
-              with:
-                  node-version: ${{ matrix.node-version }}
-            - run: npm install jscpd
-            - name: Run jscpd
-              run: npx jscpd --config "$GITHUB_WORKSPACE/.github/workflows/jscpd.json"
-
     macos:
         needs: lint-commits
         name: test macOS