Simple diff analysis based on strings and identifiers

[veilm]

This is an idea for a type of analysis of code diffs. This issue is just for tracking notes and ideas

Example input 1

+            guidance: function (e) {
+              var t = e.isSearchable,
+                n = e.isMulti,
+                r = e.isDisabled,
+                i = e.tabSelectsValue;
+              switch (e.context) {
+                case "menu":
+                  return "Use Up and Down to choose options"
+                    .concat(
+                      r
+                        ? ""
+                        : ", press Enter to select the currently focused option",
+                      ", press Escape to exit the menu",
+                    )
+                    .concat(
+                      i
+                        ? ", press Tab to select the option and exit the menu"
+                        : "",
+                      ".",
+                    );

Here the extraction might be

guidance
e
t
isSearchable
n
isMulti
isDisabled
i
tabSelectsValue
context
"menu"
"Use Up and Down to choose options"
concat
r
""
", press Enter to select the currently focuse doption"
", press Escape to exit the menu"
", press Tab to select the option and exit the menu"
"."

Of course, this gives you far less information than the original, but I think it could be a good trade-off in cases where you want to look at the diff a little bit but don't have time to see everything.

Example input 2

Since common generic ones like e, t, n, "", and "." would show up frequently in any context, they would have already been seen in the past, and therefore filtered out. You'd more focus on the e.g. "Use Up and Down to choose options", with some kind of convenient way to jump back to see it in-context in the code.

For input like the following:

+      var a = n(72843);
+      function s(e, t) {
+        for (var n = 0; n < t.length; n++) {
+          var r = t[n];
+          (r.enumerable = r.enumerable || !1),
+            (r.configurable = !0),
+            "value" in r && (r.writable = !0),
+            Object.defineProperty(e, (0, a.Z)(r.key), r);
+        }
+      }
+      function l(e, t, n) {
+        return (
+          t && s(e.prototype, t),
+          n && s(e, n),
+          Object.defineProperty(e, "prototype", { writable: !1 }),
+          e
+        );
+      }

, none of the names or strings would probably be new, and so you wouldn't see it at all. This is intended, because I can't gleam any conclusions from looking at it, and thus would prefer not to see it

Glenn's comments

https://twitter.com/_devalias/status/1770284997385277554

I think given the size of a lot of the JS files, and the diffs themselves; it would probably end up being a LOT of strings; which might be confusing when removed from the rest of the context of the surrounding code.

For large diffs I think it'd be a lot, but strings and names are a subset of the raw diff, so it should still be less work than a full manual analysis. The idea is to just visually filter through them until you see a name/string that looks interesting on their own, which could lead to something good in-context.

It should be fairly easy to prototype a script using babel parser and babel traverse though.
You would add a rule or couple to the traverse so that it matches on whatever strings are called in the AST; and then output them to console or a file or similar.

Haven't worked with Babel but some relevant docs seem to be

• https://babeljs.io/docs/babel-parser
• https://www.npmjs.com/package/%40babel/traverse

Are there other AST parsers too? Would something like TreeSitter work? I'd generally prefer to avoid node.js if it's not required

Then you would just diff that output file of strings between one build and the next.
If code moves around between builds it might introduce it’s own form of noise (but maybe git diff —color-moved would handle that still anyway)

I haven't seen enough diffs to exactly anticipate how these would look like but there might be different solutions like color-moved that could work depending on how it goes

I also noticed you liked some of my tweets about my more generalised diff minimiser; which would reduce the noise of things a fair bit overall as well.
I still need to polish that and commit/upload it; been super busy lately and haven’t had a chance to yet.

Related:

• Explore AST based diff tools, diff minimisation, etc #3

Feel free to open an issue on the ChatGPT Source watch repo about the string extractor idea + link back to these tweets/copy the relevant info in.
I’d be happy to give some more pointers about it and/or include it in the repo if you wanted to work on it.

Yeah, I want to make a prototype and see if it will kind of work. I'm still not sure on the implementation, though; the most efficient system might be to integrate with a text editor, which makes it harder to be replicable

[veilm]

According to a random post I read on Hacker News, difftastic uses tree-sitter too, so I suppose it would be general enough for this use case
https://tree-sitter.github.io/tree-sitter/

[0xdevalias]

Since common generic ones like e, t, n, "", and "." would show up frequently in any context, they would have already been seen in the past, and therefore filtered out. You'd more focus on the e.g. "Use Up and Down to choose options"

@michaelskyba For the variable names like e, t, n, etc; if you're using an AST parser they'll show as a different node type to proper "" strings, so they'll be pretty easy to avoid.

---

with some kind of convenient way to jump back to see it in-context in the code.

@michaelskyba An easy way to connect from those strings back to the context in the code would definitely be interesting. Almost feels like something you could do with monaco editor (the core of vscode), and symbol links/similar:

• https://github.com/microsoft/monaco-editor
• https://code.visualstudio.com/docs/editor/editingevolved#_go-to-symbol

I know @j4k0xb has done some cool stuff with using monaco as the web IDE for webcrack:

• https://github.com/j4k0xb/webcrack

And I think @pionxzh 's wakaru also uses monaco (or plans to for the newer IDE; I can't remember exactly off the top of my head):

• https://github.com/pionxzh/wakaru

As a first exploration/PoC, I would probably consider the contextual linking back to be out of scope though, as it has the ability to quickly become a complicated rabbithole/distraction from the core features.

---

For large diffs I think it'd be a lot, but strings and names are a subset of the raw diff, so it should still be less work than a full manual analysis. The idea is to just visually filter through them until you see a name/string that looks interesting on their own, which could lead to something good in-context.

@michaelskyba Mm, makes sense.

---

Are there other AST parsers too? Would something like TreeSitter work? I'd generally prefer to avoid node.js if it's not required

@michaelskyba Yeah, there are definitely lots of AST parsers out there. Technically you could probably use TreeSitter or similar for this.

I'm curious what your reasoning for wanting to avoid using node is though? Pretty much all of the scripts and automation in this repo currently are currently written in node/JS/TS; for many use cases it's a pretty useful tool.

If you want to go deep on this subject, there's heaps of stuff on my gists, eg.:

• https://gist.github.com/0xdevalias
  • https://gist.github.com/0xdevalias/d8b743efb82c0e9406fc69da0d6c6581#deobfuscating--unminifying-obfuscated-web-app-code
  • https://gist.github.com/0xdevalias/8c621c5d09d780b1d321bfdb86d67cdd#reverse-engineering-webpack-apps
  • etc

---

According to a random post I read on Hacker News, difftastic uses tree-sitter too, so I suppose it would be general enough for this use case

@michaelskyba Yup, tree-sitter is pretty generic.

---

Yeah, I want to make a prototype and see if it will kind of work. I'm still not sure on the implementation, though; the most efficient system might be to integrate with a text editor, which makes it harder to be replicable

@michaelskyba If you're happy to use node and babel parser/etc, then I can definitely help you get started with it. There are already some examples within this repo (albeit not using traverse I don't think):

• https://github.com/search?q=repo%3A0xdevalias%2Fchatgpt-source-watch%20babel%20parser&type=code
  • https://github.com/0xdevalias/chatgpt-source-watch/blob/1c127295d3aa2781b933a56d96bfb386380eb029/scripts/extract-webpack-urls.js

There are a few 'AST Explorer' tools that are super useful when working with AST's as well:

• https://astexplorer.net/
• https://ast.sxzz.moe/

I would personally start by writing it as a standalone script that just writes to the console or a file, that way you can quickly 'prove the concept' and see if it will be beneficial. Then later on, if you wanted to enhance the UX (user experience) of using it by integrating it into a text editor / IDE / GUI / etc, that could be a path for further exploration.

[0xdevalias]

Here's a quick/dirty example that I hacked together with ChatGPT + some patterns from existing code. I haven't deeply tested it works properly, but from a quick skim through + basic test, it looks like it should probably work, or be close to doing so:

Example String Extraction Code

#!/usr/bin/env node

const fs = require('fs');
const path = require('path');
const parser = require('@babel/parser');
const traverse = require('@babel/traverse').default;

let DEBUG = false; // Initial debug state

function displayUsage(scriptName) {
  console.log(`Usage: ${scriptName} <file-path>
Options:
  --debug        Enable debug mode.
  -h, --help     Display this usage information.`);
}

function parseCommandLineArguments() {
  const scriptName = path.basename(process.argv[1]);
  const args = process.argv.slice(2);

  if (args.includes('-h') || args.includes('--help')) {
    displayUsage(scriptName);
    process.exit(0);
  }

  const debugFlagIndex = args.indexOf('--debug');
  if (debugFlagIndex !== -1) {
    DEBUG = true;
    args.splice(debugFlagIndex, 1);
  }

  if (args.length !== 1) {
    displayUsage(scriptName);
    console.error('\nError: Invalid number of arguments provided.');
    process.exit(1);
  }

  const filePath = args[0];

  let code;
  try {
    code = fs.readFileSync(filePath, 'utf8');
  } catch (err) {
    console.error(`Error reading file: ${filePath}\n${err.message}`);
    process.exit(1);
  }

  return { code };
}

function parseCodeToAST(code) {
  return parser.parse(code, {
    sourceType: 'module',
    plugins: [
      // Add any necessary plugins here
    ],
  });
}

function extractStrings(ast) {
  const strings = [];
  traverse(ast, {
    StringLiteral: ({ node }) => {
      strings.push(node.value);
    },
    TemplateLiteral: ({ node }) => {
      node.quasis.forEach((element) => {
        strings.push(element.value.cooked);
      });
    },
  });
  return strings;
}

function main() {
  const { code } = parseCommandLineArguments();
  const ast = parseCodeToAST(code);
  const strings = extractStrings(ast);

  if (DEBUG) {
    console.log('[DEBUG] Extracted Strings:', strings);
  } else {
    console.log(strings.join('\n'));
  }
}

main();

The main parts to pay attention to are parseCodeToAST and extractStrings; the logic within them might need some tweaking/etc though; and it could no doubt be extended to better capture the context they're being extracted from/etc to make it easier to jump back to them later, etc.

Example Output

⇒ ./scripts/extract-js-strings.js unpacked/_next/static/chunks/webpack.js

define cannot be used indirect
object
function
object
static/chunks/
1092ac6e
9784a43c
queryString
c029043f
tsub-middleware
30750f44
303a24d8
420d684a
schemaFilter
auto-track
legacyVideos
65fd0ec4
sso
5be645cc
remoteMiddleware
ajs-destination
0d9e9cd9
b9f778a8
adb5c70d
.
660fa56f7d9f8841
f366c884f1a0069d
5c208462c867bb6d
11bb38d51776e36c
3ff460de12e80a25
1695899797a18685
c39541320838d2e1
7f0de228a3562cc0
35c9d0cb5854cadd
1aac68555afd4d6b
035f577202ad7345
fd18bd86faeaa7b0
e9a991b898301078
361ecd58fa0ac9f5
797361ea99a892a2
8596a5c250b2c4f7
b4746c7a1f512b5a
36f791c3f2dbf0ec
29104c892767faf2
415799798aaeeffc
04ab16a240f4ce97
32ab3d24800a1a80
3421eea06995d5df
0f0eee9f9d7a9a3a
df174924c6968fe4
c865be216348dd51
2f6ad9ce262d966a
4c4cb925af3442d0
c63200a5c100c707
46b523e60584f44f
739643ec58c60aab
2fad09c9cd727ecc
a08151d3efed095c
0c0ee75dab4b3a1d
f0c19e9fbc4a2762
1ce750acd0106d6e
b80d67713515dff2
d028af8c55d386c9
b41a5c7bc5017269
94b9b8a3298ce71f
352bbad25795ea42
60463b33da33b8b8
6ef5e476dad7ed16
2ae7c3fec20da7cb
73ef26c15091fb86
eecba8f575f628a2
0445a5d12846771b
9dc2787a40cce737
1a42259f0c3d91c1
6c7d9ca4e6244388
70979731ae1a472e
c3edeef63a58e362
270e8342cc6087e0
a671d3e99d9a3215
ee7e1452e33739e6
b0eb71c1fc6253b5
4b3fe5232d753e7c
2177f28b937c71b0
583ec3a0c8764140
018b8b2946dc8b20
10400e67574b5a9a
abf37f1db8be5c89
3a60974229840ffd
c682389becf401d9
f235c81abfebf195
6d3d8ed11b0e9709
367b294ff3d2e4a0
5fa4b187af7d697f
42809fbb8b2f331e
01c9e3f110f368ae
c6c0f36a9a82d8dd
fa6fc1112abad268
87e726622393959c
84cbf69a0bf382da
edae42c864b47324
bc4f77e47caaeac6
cbbca09eae978a72
79c8284e0121e7e4
d018f6e9a9ebea16
16a1a63dd7d6adcb
207481a6c8e707c7
5b368a3b1f211414
32ab39c72c4d4509
3b0e463179d66d5e
6abd0ae9c08306a9
36bb6cb6e7d3b8e4
4ed8f6b71d8bfe92
aead845693cd969c
1f34ff95cbccd78d
d9cd8bc88bf780e8
92961eb1ec331066
550cbb4354158933
.js
static/css/
f3d0bad461ffb80b
944a9aa5c00c7d0e
d92eaad6fd257c73
.css
object
return this
object
_N_E:
script
src
data-webpack
script
utf-8
nonce
data-webpack
/
anonymous
timeout
undefined
Module
__esModule
undefined
nextjs#bundler
https://cdn.oaistatic.com/_next/
link
stylesheet
text/css
load
load
missing
Loading CSS chunk
 failed.
(
)
CSS_CHUNK_LOAD_FAILED
/
anonymous
link
data-href
href
stylesheet
style
data-href
load
missing
Loading chunk
 failed.
(
:
)
ChunkLoadError
chunk-

While the above example shows that the concept works, running it on a larger file ends up with a LOT more output content (eg. 57k lines in this case):

⇒ ./scripts/extract-js-strings.js unpacked/_next/static/chunks/pages/_app.js | wc -l
   57025

Obviously most of that wouldn't change between builds though, so would end up being a lot less all in all.

---

Edit: Committed the above basic script in 210a870 for future reference/usage.

[0xdevalias]

Then you would just diff that output file of strings between one build and the next.
If code moves around between builds it might introduce it’s own form of noise (but maybe git diff —color-moved would handle that still anyway)

I haven't seen enough diffs to exactly anticipate how these would look like but there might be different solutions like color-moved that could work depending on how it goes

Something like this could also be useful for reducing the 'noise' of code that moves around:

• https://github.com/orgs/community/discussions/9632#discussioncomment-8769709

  • As an alternative, you can use VS Code to review while highlighting moved code. It's available behind an experimental flag. It also works in githubs online version. So, simply press . in an open PR and review there.

  • https://code.visualstudio.com/updates/v1_82#_moved-code-detection

    • This iteration we polished the moved code detection feature. It can be enabled with "diffEditor.experimental.showMoves": true or in the diff editor context menu. When enabled, code blocks that are moved from one location to a different location in the same file are detected and arrows are drawn to indicate where the code blocks moved to.

      Code moves are also detected when they are slightly modified. The Compare button can be used to compare the block before and after the move.

[veilm]

An easy way to connect from those strings back to the context in the code would definitely be interesting. Almost feels like something you could do with monaco editor (the core of vscode), and symbol links/similar:

Yeah, possibly; that's the kind of thing I was thinking could be useful as a text editor integration. But I think that would make sense as a wrapper around the main script. The output itself could maybe have the line numbers to the left of each finding, which would be parsed by the wrapper

I'm curious what your reasoning for wanting to avoid using node is though? Pretty much all of the scripts and automation in this repo currently are currently written in node/JS/TS; for many use cases it's a pretty useful tool.

I guess I just personally find the JS/npm ecosystem pretty annoying compared to e.g. Python or Go. It's not a dealbreaker though

If you want to go deep on this subject, there's heaps of stuff on my gists

Yeah, looks useful, I linked to it as part of my notes for the ChatGPT frontend

Here's a quick/dirty example that I hacked together with ChatGPT + some patterns from existing code

Cool, that's pretty good

I still want to look at Tree-sitter in case I like it more, but this kind of API from @babel/traverse looks fine enough

function extractStrings(ast) {
  const strings = [];
  traverse(ast, {
    StringLiteral: ({ node }) => {
      strings.push(node.value);
    },
    TemplateLiteral: ({ node }) => {
      node.quasis.forEach((element) => {
        strings.push(element.value.cooked);
      });
    },
  });
  return strings;
}

[veilm]

Looks like building .so language files from source is deprecated in py-tree-sitter, but there are no official python bindings for the javascript grammar. Welp, that's annoying...

The third-party Go bindings in https://github.com/smacker/go-tree-sitter look more promising maybe

[veilm]

Yeah, go-tree-sitter seems to work. Counting lines should be possible by looking at sourceCode.

[veilm]

From the testing I've done to this point, I think would prefer go-tree-sitter to Babel, so I'll continue developing the formatting here. It'd still be useful to have your extract-js-strings.js for reference, though.

[j4k0xb]

Another approach I used in the past is renaming every variable and its references etc to a so the diff can detect actual changes pretty reliably:

and ignore all name changes caused by minifying:

I wonder if its possible to get all the diff positions and then render it using the original names

[veilm]

I wonder if its possible to get all the diff positions and then render it using the original names

That sounds like it could indeed save time on skipping some changes, but I'd expect some minification diffs to be structurally different too, such that the same values are written differently, even if the actual name is the same. But I expect @0xdevalias would have a more informed perspective

In other news, I got a basic byte tracker that checks for newlines, to map each name properly:

1:e
1:t
1:n
2:"use strict"
3:a
4:r
4:n
5:s
6:s
6:Object
6:assign
7:Object
7:assign
7:bind
8:e
9:t
9:t
9:arguments
9:length
9:t

[j4k0xb]

afaik minifiers should produce the exact same structure for unmodified regions of the source

[veilm]

afaik minifiers should produce the exact same structure for unmodified regions of the source

Oh, that's more convenient, then. Does the formatter (biome/prettier) have the same property?