Skip to content

src: map file symlinks for old node versions #292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
flakey5 merged 1 commit into from
Feb 21, 2025
Merged

src: map file symlinks for old node versions #292

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 42 additions & 2 deletions scripts/build-r2-symlinks.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
#!/usr/bin/env node
'use strict';

import { join } from 'node:path';
import { writeFile } from 'node:fs/promises';
import { basename, dirname, join } from 'node:path';
import { readFile, writeFile } from 'node:fs/promises';
import {
HeadObjectCommand,
ListObjectsV2Command,
Expand Down Expand Up @@ -35,6 +35,14 @@ const CACHED_DIRECTORIES_OUT = join(
'cachedDirectories.json'
);

const FILE_SYMLINKS = join(
import.meta.dirname,
'..',
'src',
'constants',
'fileSymlinks.json'
);

if (!process.env.CF_ACCESS_KEY_ID) {
throw new TypeError('CF_ACCESS_KEY_ID missing');
}
Expand Down Expand Up @@ -118,6 +126,38 @@ const cachedDirectories = {
},
};

// Some older versions of Node exist in `nodejs/release/` and have folders
// with symlinks to them. For example, node-v0.1.100.tar.gz lives under
// `nodejs/release/`, but there's also `nodejs/release/v0.1.100/node-v0.1.100.tar.gz`
// which is just a symlink to it.
// Let's add these to our cached directories.
const fileSymlinks = JSON.parse(await readFile(FILE_SYMLINKS, 'utf8'));

for (const file of Object.keys(fileSymlinks)) {
Copy link
Member

@ovflowd ovflowd Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be cleaned up a bit? Maybe instead if else, we can have different loops just to separate concerns. Performance here should not be a driving factor.

Copy link
Member Author

@flakey5 flakey5 Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really sure separating it into two different loops would look much better,

for (const file of Object.keys(fileSymlinks)) {
  const directory = `${dirname(file)}/`

  if (!(directory in cachedDirectories)) {
    continue;
  }

  const actualFile = await headFile(client, fileSymlinks[file]);

  cachedDirectories[directory].files.push({
    ...actualFile,
    name: basename(file)
  })
}

for (const file of Object.keys(fileSymlinks)) {
  const directory = `${dirname(file)}/`

  if (!(directory in cachedDirectories)) {
    continue;
  }

  const actualFile = await headFile(client, fileSymlinks[file]);

  const contents = await listDirectory(client, directory);
  contents.files.push({
    ...actualFile,
    name: basename(file),
  })

  cachedDirectories[directory] = contents;
}

// Stat the actual file so we can get it's size, last modified
const actualFile = await headFile(client, fileSymlinks[file]);

const directory = `${dirname(file)}/`;

if (directory in cachedDirectories) {
// Directory was already cached, let's just append the file to the result
cachedDirectories[directory].files.push({
...actualFile,
name: basename(file),
});
} else {
// List the directory that the symlink is in so we can append the symlink to
// what's actually there
const contents = await listDirectory(client, directory);
contents.files.push({
...actualFile,
name: basename(file),
});

cachedDirectories[directory] = contents;
}
}

await writeFile(CACHED_DIRECTORIES_OUT, JSON.stringify(cachedDirectories));

/**
Expand Down
1 change: 1 addition & 0 deletions src/constants/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,6 @@ Various constants used throughout the worker
- [cachedDirectories.json](./cachedDirectories.json) - Directories that have their listing result cached because they have symlinks in them. These are updated by the [build-r2-symlinks](../scripts/build-r2-symlinks.mjs) script.
- [docsDirectory.json](./docsDirectory.json) - The contents of `nodejs/docs/` in the `dist-prod` bucket. This is updated by the [build-r2-symlinks](../scripts/build-r2-symlinks.mjs) script.
- [files.ts](./files.ts) - Constants related to files the worker serves.
- [fileSymlinks.ts](./fileSymlinks.json) - Manually updated mapping of file symlinks that exist in the `dist-prod` bucket.
- [latestVersions.json] - Map of `latest-*` directories to their actual latest versions. This is updated by the [build-r2-symlinks](../scripts/build-r2-symlinks.mjs) script.
- [limits.ts](./limits.ts) - Hardcap limits that the worker shouldn't exceed.
Loading
Loading