Explore using embeddings/similar to identify/track similar chunks/modules even when renamed

[0xdevalias]

I did some initial exploratory work for this in a script ages back; can't remember if it was in chatgpt-source-watch or udio-source-watch repo, and not sure if it ever got to being committed or if it's just somewhere locally still.

The general gist of this issue is that between webpack/similar builds, sometimes the chunk identifiers are renamed, which can mess up our diffing. Often times it's relatively easy to see/guess the renames based on looking at the diffs themselves (eg. in the _buildManifest.js / webpack.js files; but then it's a semi-manual process of renaming these to align so that the diffs look correctly (I believe I wrote some scripts to assist with this at some point also, probably alongside the one mentioned earlier, but similarly may not have been committed anywhere yet).

Similarly, sometimes the chunk identifiers themselves may not have changed, but the module identifiers and/or which chunk they are in may have moved around; causing similar issues with diffing/identifying what is actually new code vs just being moved around, etc.

The idea here is basically to use embeddings / similarity search / etc to compare the chunk files (which is what my initial script does), or the modules within them (which is a more recent idea I had for further enhancements to this) to find the closest match; which then allows us to infer in a programmatic/automated way whether its likely to have been renamed; after which we can handle it appropriately.

See Also

• Deobfuscating / Unminifying Obfuscated Web App / JavaScript Code (0xdevalias' gist)
  • Subsection: Fingerprinting Minified JavaScript Libraries
• Fingerprinting Minified JavaScript Libraries / AST Fingerprinting / Source Code Similarity / Etc (0xdevalias' gist)
• Integrate wakaru and/or webcrack JS unminimisers (and maybe humanify too) #2

[0xdevalias]

I'll try and find my older scripts later, but for now, a couple of initial references that may be useful, initially found from this issue:

• Add code search tool with RAG capabilities All-Hands-AI/openhands-aci#34
  • https://github.com/UKPLab/sentence-transformers

    • Sentence Transformers: Multilingual Sentence, Paragraph, and Image Embeddings using BERT & Co.

    • State-of-the-Art Text Embeddings

    • https://sbert.net/

      • Sentence Transformers (a.k.a. SBERT) is the go-to Python module for accessing, using, and training state-of-the-art text and image embedding models. It can be used to compute embeddings using Sentence Transformer models (quickstart) or to calculate similarity scores using Cross-Encoder models (quickstart). This unlocks a wide range of applications, including semantic search, semantic textual similarity, and paraphrase mining.

      • https://sbert.net/docs/sentence_transformer/usage/semantic_textual_similarity.html

        • Semantic Textual Similarity
          For Semantic Textual Similarity (STS), we want to produce embeddings for all texts involved and calculate the similarities between them. The text pairs with the highest similarity score are most semantically similar. See also the Computing Embeddings documentation for more advanced details on getting embedding scores.

      • https://sbert.net/docs/quickstart.html#cross-encoder

        • Cross Encoder
          Characteristics of Cross Encoder (a.k.a reranker) models:

          • Calculates a similarity score given pairs of texts.
          • Generally provides superior performance compared to a Sentence Transformer (a.k.a. bi-encoder) model.
          • Often slower than a Sentence Transformer model, as it requires computation for each pair rather than each text.
          • Due to the previous 2 characteristics, Cross Encoders are often used to re-rank the top-k results from a Sentence Transformer model.

      • https://sbert.net/examples/applications/computing-embeddings/README.html

        • Computing Embeddings

      • https://sbert.net/docs/sentence_transformer/usage/efficiency.html

        • Speeding up Inference
          Sentence Transformers supports 3 backends for computing embeddings, each with its own optimizations for speeding up inference

  • https://huggingface.co/spaces/mteb/leaderboard

    • MMTEB: Massive Multilingual Text Embedding Benchmark
      The MMTEB leaderboard compares text embedding models on 1000+ languages.

[0xdevalias]

I did some initial exploratory work for this in a script ages back; can't remember if it was in chatgpt-source-watch or udio-source-watch repo, and not sure if it ever got to being committed or if it's just somewhere locally still.

Looks like it was in udio-source-watch, and that it is still only local, not committed/pushed anywhere. Here are the seemingly relevant scripts/bits.

requirements.txt:

numpy==1.26.4
scikit-learn==1.4.2

scripts/text_similarity_checker.py

#!/usr/bin/env python

# TODO: It would be interesting to see how this TfidfVectorizer + cosine_similarity method compares with using difflib's SequenceMatcher + ratio methods:
#   https://docs.python.org/3/library/difflib.html#sequencematcher-objects
#   See also:
#     https://docs.python.org/3/library/difflib.html#difflib.get_close_matches
#       Return a list of the best “good enough” matches. word is a sequence for which close matches are desired (typically a string), and possibilities is a list of sequences against which to match word (typically a list of strings).

# TODO: This ChatGPT chat has some examples of how to calculate this sort of thing in JavaScript:
#   https://chatgpt.com/c/7fef26fd-0531-4079-b508-43904ff3e089
#   See also:
#     https://github.com/NaturalNode/natural/
#       https://naturalnode.github.io/natural/
#       https://blog.logrocket.com/natural-language-processing-node-js/
#     https://winkjs.org/
#       https://winkjs.org/wink-nlp/bm25-vectorizer.html
#         BM25 is a major improvement over the classical TF-IDF based algorithms. The weights for a specific term (i.e. token) is computed using the BM25 algorithm.
#       https://github.com/winkjs/wink-nlp
#       https://github.com/winkjs/wink-nlp-utils
#         https://winkjs.org/wink-nlp-utils/
#       https://github.com/winkjs/wink-distance

import argparse
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import os

def read_file(file_path):
    with open(file_path, 'r', encoding='utf-8') as file:
        return file.read()

def calculate_similarities(main_file, other_files):
    documents = [read_file(main_file)] + [read_file(f) for f in other_files]
    tfidf_vectorizer = TfidfVectorizer()
    tfidf_matrix = tfidf_vectorizer.fit_transform(documents)
    main_doc_matrix = tfidf_matrix[0:1]
    similarities = cosine_similarity(main_doc_matrix, tfidf_matrix[1:])
    return list(zip(other_files, similarities.flatten()))

def main():
    parser = argparse.ArgumentParser(description="Calculate cosine similarity between a main file and a list of other files.")
    parser.add_argument("main_file", type=str, help="The main file to compare.")
    parser.add_argument("other_files", nargs='+', type=str, help="A list of other files to compare against the main file.")
    args = parser.parse_args()

    # Filter out the main file early if it's accidentally included in other_files
    filtered_files = [f for f in args.other_files if f != args.main_file]

    if not os.path.isfile(args.main_file):
        print(f"Error: '{args.main_file}' does not exist or is not a file.")
        return

    for file_path in filtered_files:
        if not os.path.isfile(file_path):
            print(f"Error: '{file_path}' does not exist or is not a file.")
            return

    results = calculate_similarities(args.main_file, filtered_files)
    sorted_results = sorted(results, key=lambda x: x[1], reverse=True)

    # for other_file, similarity in sorted_results:
    #     print(f"Similarity between {args.main_file} and {other_file}: {similarity:.4f}")

    print(f"Comparing against: {args.main_file}")
    for other_file, similarity in sorted_results:
        print(f"{other_file}: {similarity:.4f}")

if __name__ == "__main__":
    main()

scripts/rename-chunk.sh

#!/usr/bin/env zsh

# Check if the correct number of arguments is provided
if [ "$#" -ne 2 ]; then
  echo "Usage: $0 <old_file> <new_file>"
  exit 1
fi

old_file=$1
new_file=$2

# Check if both arguments are regular files
if [ ! -f "$old_file" ]; then
  echo "Error: $old_file is not a file."
  exit 1
fi

if [ ! -f "$new_file" ]; then
  echo "Error: $new_file is not a file."
  exit 1
fi

# Check if the old file is tracked by Git
if ! git ls-files --error-unmatch "$old_file" &> /dev/null; then
  echo "Error: $old_file is not tracked by Git."
  exit 1
fi

echo "Starting the file renaming process..."

# Temporarily rename the new file to preserve it
mv $new_file $new_file.new

# Use git mv to rename the old file to the new file's original name
git mv $old_file $new_file

# Restore the originally new file from its temporary name
mv $new_file.new $new_file

echo "File renaming complete. $old_file has been renamed to $new_file."

useful-commands.md

This isn't the full file, just some relevant looking snippets from it:

See how similar a chunk file is to other chunk files (to find potential chunkID churn):

```bash
npm run chunk:check 1793

# chunk-check () { ./scripts/text_similarity_checker.py unpacked/_next/static/chunks/${1}.js unpacked/_next/static/chunks/*.js | head -n 5; }; chunk-check 1793
```

Rename a chunk file that changed due to chunkID churn:

```bash
npm run chunk:rename 7073 1793

# chunk-rename () { ./scripts/rename-chunk.sh unpacked/_next/static/chunks/${1}.js unpacked/_next/static/chunks/${2}.js; }; chunk-rename 7073 1793
```