ci: "Failed to watch node_modules/…/package.json using fs.watch() … ENOSPC"

[justinmk3]

Problem

The Linux Unit Tests CI jobs show these (non-fatal, currently) errors indicating that something is crawling files in node_modules and trying to watch package.json

[File Watcher (parcel)] Inotify limit reached (ENOSPC) (path: /codebuild/output/src4054755577/src/github.com/aws/aws-toolkit-vscode/node_modules)
  file_editAwsFile telemetry
[File Watcher (node.js)] Failed to watch /codebuild/output/src4054755577/src/github.com/aws/aws-toolkit-vscode/node_modules/@types/readline-sync/package.json for changes using fs.watch() (Error: ENOSPC: System limit for number of file watchers reached, watch '/codebuild/output/src4054755577/src/github.com/aws/aws-toolkit-vscode/node_modules/@types/readline-sync/package.json')
[File Watcher (node.js)] Failed to watch /codebuild/output/src4054755577/src/github.com/aws/aws-toolkit-vscode/node_modules/@types/responselike/package.json for changes using fs.watch() (Error: ENOSPC: System limit for number of file watchers reached, watch '/codebuild/output/src4054755577/src/github.com/aws/aws-toolkit-vscode/node_modules/@types/responselike/package.json')
[File Watcher (node.js)] Failed to watch /codebuild/output/src4054755577/src/github.com/aws/aws-toolkit-vscode/node_modules/@types/retry/package.json for changes using fs.watch() (Error: ENOSPC: System limit for number of file watchers reached, watch '/codebuild/output/src4054755577/src/github.com/aws/aws-toolkit-vscode/node_modules/@types/retry/package.json')
[File Watcher (node.js)] Failed to watch /codebuild/output/src4054755577/src/github.com/aws/aws-toolkit-vscode/node_modules/@types/semver/package.json for changes using fs.watch() (Error: ENOSPC: System limit for number of file watchers reached, watch '/codebuild/output/src4054755577/src/github.com/aws/aws-toolkit-vscode/node_modules/@types/semver/package.json')
Unable to load and parse grammar for scope source.ssmjson from file:///codebuild/output/src4054755577/src/github.com/aws/aws-toolkit-vscode/packages/core/syntaxes/SSMJSON.tmLanguage { fileOperationResult: 1, options: { preferUnbuffered: true } }
    ✔ emits when opened by user (488ms)

Expected behavior

Toolkit/Q code should not create lots of file watchers. #2382 (Good news: this may be a CI-only issue; see "Analysis" below.)

This issue is currently not "fatal" (doesn't fail CI or interfere with tests ... yet). But:

1. it may point to a problem that affects actual users (on Linux if not other OSes)
2. it adds noise to the CI logs
3. it may interfere with tests in the future

Analysis

• The …/aws-toolkit-vscode/node_modules/…/package.json paths indicate that this may be happening from the test runner or other tooling, rather than the Toolkit application code. Because that isn't the "test workspace", that's the source tree.
  • This suggests that this may be a CI-only problem.
  • No effect from disabling all calls to createFileSystemWatcher() in our code.
  • No effect from overriding fs.watch.
• Our watchedFiles modules suffers from the fact that its "exclude" feature doesn't actually prevent watchers from being created. it only skips the events at runtime.
• The docs for createFilesystemWatcher indicate that recursive patterns for a workspace are supposedly handled by vscode to avoid duplicate watchers ?
• This may be a linux-only issue? see:
  • Error: ENOSPC: System limit for number of file watchers reached, watch '/snap/code' microsoft/vscode#162567
  • "To find out a culprit which is wasting your watchers you can use https://github.com/fatso83/dotfiles/tree/master/utils/scripts#inotify-consumers"

[justinmk3]

This is becoming much more frequent, for example #5944

Potential solution: add "**/node_modules/**" to the files.watcherExclude vscode setting, before running tests in CI.