Automatically regenerate media sizes

[jmikrut]

This is a commonly asked topic and it just came up in Discord today, so I figured I would share a quick script that the team and I use for our Payload projects when we need something like WP's "Regenerate Thumbnails" plugin. Basically if you update your imageSizes and want to automatically re-run the image transformations for each one of your uploads, this can be super handy.

const payload = require('payload');
const path = require('path');

require('dotenv').config();

const { PAYLOAD_SECRET_KEY, MONGO_URL } = process.env;

payload.init({
  secret: PAYLOAD_SECRET_KEY,
  mongoURL: MONGO_URL,
  local: true,
});

const regenerateMediaSizes = async () => {
  try {
    const media = await payload.find({
      collection: 'media',
      depth: 0,
      limit: 300,
    });
    await Promise.all(media.docs.map(async (mediaDoc) => {
      try {
        const staticDir = path.resolve(__dirname, '../media');

        await payload.update({
          collection: 'media',
          id: mediaDoc.id,
          data: mediaDoc,
          filePath: `${staticDir}/${mediaDoc.filename}`,
          overwriteExistingFiles: true,
        });

        console.log(`Media ${mediaDoc.alt || mediaDoc.id} regenerated successfully`);
      } catch (err) {
        console.error(`Media ${mediaDoc.alt || mediaDoc.id} failed to regenerate`);
        console.error(err);
      }
    }));
  } catch (err) {
    console.log('Unable to find documents with payload');
    console.error(err);
    process.exit(0);
  }

  console.log('Media size regeneration completed!');
  process.exit(0);
};

regenerateMediaSizes();

If you save this file as regenerateMediaSizes.js, then all you need to do to run it is node ./regenerateMediaSizes.js and boom.

Related: #434

[fabioquarantini]

How to delete old unused thumbnails?

[ChristopherNowlan]

I had to move payload init function into the regenerateMediaSizes async function. Because my config file was within the src directory I had to update the command to be cross-env PAYLOAD_CONFIG_PATH=dist/payload.config.js node ./regenerateMediaSizes.js

const payload = require('payload');
const path = require('path');

require('dotenv').config();

const { PAYLOAD_SECRET_KEY, MONGO_URL } = process.env;

const regenerateMediaSizes = async () => {
  try {

  await payload.init({
        secret: PAYLOAD_SECRET_KEY,
        mongoURL: MONGODB_URL,
        local: true,
    });

    const media = await payload.find({
      collection: 'media',
      depth: 0,
      limit: 300,
    });

    await Promise.all(media.docs.map(async (mediaDoc) => {
      try {
        const staticDir = path.resolve(__dirname, '../media');

        await payload.update({
          collection: 'media',
          id: mediaDoc.id,
          data: mediaDoc,
          filePath: `${staticDir}/${mediaDoc.filename}`,
          overwriteExistingFiles: true,
        });

        console.log(`Media ${mediaDoc.alt || mediaDoc.id} regenerated successfully`);
      } catch (err) {
        console.error(`Media ${mediaDoc.alt || mediaDoc.id} failed to regenerate`);
        console.error(err);
      }
    }));
  } catch (err) {
    console.log('Unable to find documents with payload');
    console.error(err);
    process.exit(0);
  }

  console.log('Media size regeneration completed!');
  process.exit(0);
};

regenerateMediaSizes();

[andriasang]

@jmikrut : Is this script compatible with the cloud storage plugin? I removed the "filePath" section of the update call and it seems that every image in my collection is being iterated over but the sizes area is just empty objects:

sizes: { w400: {}, w768: {}, w1024: {} },

[arieltonglet]

@andriasang I've managed to make a version of this script that works with plugin-cloud-storage – building on top of @ChristopherNowlan script and with a huge help from @jmikrut on Discord.

The main point to get it working is: get the media url, fetch it, and pass it as a File in a file prop instead of filePath

Important to note as well: Check if you have the latest config file in your dist folder, to get the updated imageSizes. To make sure of that, I added a build to the script. Its more of a local development problem than a production one, but just in case:

yarn build && cross-env PAYLOAD_CONFIG_PATH=dist/payload.config.js node ./regenerateMediaSizes.js

const payload = require("payload");
const path = require("path");

require("dotenv").config();

const { PAYLOAD_SECRET, MONGODB_URI } = process.env;

const regenerateMediaSizes = async () => {
  try {
    await payload.init({
      secret: PAYLOAD_SECRET,
      mongoURL: MONGODB_URI,
      local: true,
    });

    const media = await payload.find({
      collection: "media",
      depth: 0,
      limit: 300,
    });

    await Promise.all(
      media.docs.map(async (mediaDoc) => {
        try {
          await fetch(mediaDoc.url)
            .then((response) => response.blob())
            .then(async (blob) => {
              const arrayBuffer = await blob.arrayBuffer();
              const buffer = await Buffer.from(arrayBuffer, "binary");

              const file = {
                data: buffer,
                mimetype: blob.type,
                name: mediaDoc.filename,
                size: mediaDoc.filesize,
              };

              await payload.update({
                collection: "media",
                id: mediaDoc.id,
                data: mediaDoc,
                file,
                overwriteExistingFiles: true,
                contentType: blob.type,
              });

              console.log(
                `Media ${mediaDoc.id} (${mediaDoc.filename}) regenerated successfully`
              );
            });
        } catch (err) {
          console.error(
            `Media ${mediaDoc.id} (${mediaDoc.filename}) failed to regenerate`
          );
          console.error(err);
        }
      })
    );
  } catch (err) {
    console.log("Unable to find documents with payload");
    console.error(err);
    process.exit(0);
  }

  console.log("Media size regeneration completed!");
  process.exit(0);
};

regenerateMediaSizes();

[linobino1]

When I use @jmikrut's script*, images get lost in a random way. Often it works as expected, but sometimes the original file is removed during the process.

* adapted to work with payload 1.6.27 async payload.init()

[richadr]

try with overwriteExistingFiles: false

[johnrisby]

Hi, is there a problem running this under the current Payload (I'm on 1.15.7) ? I'm getting errors about not finding the collection. I've spent a while trying various things but still not working. If it does work, can someone explain where it should be run from etc please. Thanks

[danpii]

@ChristopherNowlan thanks for your help, really appreciated! Yes, I am running locally. I tried your recommendation, but I still get the same error:

Unable to find documents with payload
/Users/danielpanagiotopulos/Documents/_Work/WebStorm/playgrounds/payload-test/payload-0/src/payload.config.ts:1
import path from 'path'
^^^^^^

SyntaxError: Cannot use import statement outside a module

[ChristopherNowlan]

No worries. Change to
/dist/payload.config.js

[ChristopherNowlan]

Because I got the same issue when I tried to run it through src/payload.config.ts

[danpii]

@ChristopherNowlan Awesome! That worked! I had always run the scipt directly, but now with a npm command and /dist/payload.config.js it's running! Was that the reason why you have 'media' at root level?

For future me, and maybe others (@johnrisby @isamardzija ) here a little how to, this is how it works for me now:
Tested with Payload v2.11.0 on localhost

• media folder and regenerateMediaSizes.js on payload root level (same where package.json lies)
• use regenerateMediaSizes.js from @ChristopherNowlan - see his comment
• run yarn build
• add this npm command to "scripts" in package.json
  • "regenerateMediaSizes": "cross-env PAYLOAD_CONFIG_PATH=dist/payload.config.js node ./regenerateMediaSizes.js"
• then: npm run regenerateMediaSizes

info: when you change 'imageSizes' in your Media collection you have to run yarn build again

[ChristopherNowlan]

@danpii you could add the regenerate script to end of build script. That way it runs every time you build.

[58bits]

Here's a version based on @arieltonglet 's excellent script - also for plugin-cloud-storage .

In this version only one image is processed at a time. For concurrency (with rate limits) and a larger number of files we'd use https://github.com/sindresorhus/p-limit

This is a Typescript file, that can be placed anywhere, but must be run via ts-node in the root of the Payload project (at the same level as the .env file)

For example... ts-node src/scripts/regenerateMediaSizes.ts

/* eslint-disable @typescript-eslint/ban-ts-comment */
// @ts-nocheck

import payload from 'payload'

// eslint-disable-next-line
require('dotenv').config()

const { PAYLOAD_SECRET, DATABASE_URI } = process.env

const regenerateMediaSizes = async (): Promise<void> => {
  await payload.init({
    secret: PAYLOAD_SECRET,
    mongoURL: DATABASE_URI,
    local: true
  })

  const media = await payload.find({
    collection: 'media',
    depth: 0,
    limit: 100
  })

  if (media != null && media.totalDocs > 0) {
    payload.logger.info(`Found ${media.totalDocs} media files.`)
    for (let index = 0; index < media.docs.length; index++) {
      const mediaDoc = media.docs[index]
      try {
        await fetch(mediaDoc.url)
          .then(async (response) => await response.blob())
          .then(async (blob) => {
            const arrayBuffer = await blob.arrayBuffer()
            const buffer = Buffer.from(arrayBuffer, 'binary')

            const file = {
              data: buffer,
              mimetype: blob.type,
              name: mediaDoc.filename,
              size: mediaDoc.filesize
            }

            await payload.update({
              collection: 'media',
              id: mediaDoc.id,
              data: mediaDoc,
              file,
              overwriteExistingFiles: true,
              contentType: blob.type
            })

            payload.logger.info(
              `Media ${mediaDoc.id} (${mediaDoc.filename}) regenerated successfully`
            )
          })
      } catch (err) {
        payload.logger.error(`Media ${mediaDoc.id} (${mediaDoc.filename}) failed to regenerate`)
        console.error(err)
      }
    }
  } else {
    payload.logger.info('No media files found.')
  }

  payload.logger.info('Done!')
  process.exit(0)
}

void regenerateMediaSizes()

[tsemachh]

For Payload 3 we used:

import { getPayload } from 'payload'
import * as dotenv from 'dotenv'
import path from 'node:path'
import { fileURLToPath } from 'node:url'
import { importConfig } from 'payload/node'
dotenv.config() // Load environment variables from .env

const __dirname: string = path.dirname(fileURLToPath(import.meta.url))
const payloadConfigPath: string = path.join(__dirname, '../payload.config.ts')
const awaitedConfig = await importConfig(payloadConfigPath)
const payload = await getPayload({ config: awaitedConfig })

// eslint-disable-next-line
import 'dotenv/config'

const regenerateMediaSizes = async (): Promise<void> => {
  const media = await payload.find({
    collection: 'media',
    depth: 0,
    limit: 500,
  })
  if (media != null && media.totalDocs > 0) {
    payload.logger.info(`Found ${media.totalDocs} media files.`)
    for (let index = 0; index < media.docs.length; index++) {
      const mediaDoc = media.docs[index]
      const req = {
        headers: new Map(), // Or use any object literal
      } as PayloadRequestWithData
      req.headers.set('origin', process.env.PAYLOAD_URL)
      try {
        await payload.update({
          collection: 'media',
          id: mediaDoc.id,
          data: mediaDoc,
          overwriteExistingFiles: true,
          req,
        })

        payload.logger.info(
          `Media ${mediaDoc.id} (${mediaDoc.filename}) successfully and new copy created.`,
        )
      } catch (err) {
        payload.logger.error(`Media ${mediaDoc.id} (${mediaDoc.filename}) failed to regenerate`)
        console.error(err)
      }
    }
  } else {
    payload.logger.info('No media files found.')
  }

  payload.logger.info('Done!')
  process.exit(0)
}

void regenerateMediaSizes()

[alextreppass]

For Payload 2 I used something similar to the above approach - specifying file and overwrite existing

However, it fails to regenerate a certain image size we have 😭

      {
        name: 'hero',
        // ≈1.414:1
        width: 2048,
        height: 1448,
        position: 'centre',
      },

The other sizes were ok.

I ended up having to dump all the affected ids, and manually go to the media entry in our running app admin page, and, one by one:

• Hit 'Edit Image'
• Hit 'Apply changes' (even though no changes are made)
• Hit Save

The middle part of this seemed necessary else the sizes would not regenerate

[rvanderfeer]

I'm currently using this as a migration with Payload v3, version 3.33.0
Seems less than ideal but it does seem to do the trick.
I tried some of the solutions provided above but none worked for me as typings seem to have changed in the latest Payload v3.

import { MigrateDownArgs, MigrateUpArgs } from '@payloadcms/db-mongodb'
import { Payload, UploadCollectionSlug, UploadConfig } from 'payload'
import fs from 'fs'
import path from 'path'

async function regenerateImageSizes(payload: Payload, collectionSlug: UploadCollectionSlug) {
	// Get the collection config to access upload settings
	const collectionConfig = payload.collections[collectionSlug].config
	const uploadConfig = collectionConfig.upload as UploadConfig

	// Get all documents that have files
	const { docs } = await payload.find({
		collection: collectionSlug,
		depth: 0,
		limit: 1000, // Adjust as needed
	})

	console.log(`Found ${docs.length} documents to process`)

	// Update each document to trigger regeneration
	for (const doc of docs) {
		if (doc.filename) {
			// Only process documents with files
			console.log(`Processing document ${doc.id}...`)

			try {
				// Get the file path from PayloadCMS upload directory
				const { staticDir } = uploadConfig
				const filePath = path.join(staticDir!, doc.filename!)

				// Check if file exists
				if (!fs.existsSync(filePath)) {
					console.error(`File not found: ${filePath}`)
					continue
				}

				// Read the file as buffer
				const fileBuffer = fs.readFileSync(filePath)

				// Re-upload the file to trigger image processing
				await payload.update({
					collection: collectionSlug,
					id: doc.id,
					data: {
						alt: doc.alt?.replace(/\.[\w]+$/, ''),
					},
					file: {
						data: fileBuffer,
						size: fs.statSync(filePath).size,
						mimetype: doc.mimeType!,
						name: doc.filename,
					}
				})

				console.log(`Successfully regenerated sizes for document ${doc.id}`)
			} catch (error) {
				console.error(`Error processing document ${doc.id}:`, error)
			}
		}
	}

	console.log('Image regeneration complete!')
}

export async function up({ payload }: MigrateUpArgs): Promise<void> {
	// Collection containing your uploads
	const MEDIA_COLLECTIONS = ['media', 'brandbook'] as UploadCollectionSlug[]

	// Process each collection
	for (const collectionSlug of MEDIA_COLLECTIONS) {
		await regenerateImageSizes(payload, collectionSlug).catch(console.error)
	}
}

export async function down({}: MigrateDownArgs): Promise<void> {
	console.log('No down migration needed for image regeneration')
}

[rvanderfeer]

I don't have any hands-on experience with PayloadCMS+S3 but I think you will need to use the aws-sdk package to retrieve the file, I think doc.filename should contain the 'key' to the file in the S3 bucket. So s3.getObject() should retrieve that and you should then be able to assign the s3 response's file buffer to the payload.update() method.

[camilo220]

Unfortunately, I'm still quite new to TypeScript and PayloadCMS, and I haven't managed to fully resolve this yet. However, I'll leave a draft migration script that successfully triggers image regeneration within PayloadCMS but does not upload the new files to S3. As a result, the images' new sizes are created and stored in MongoDB, but the files themselves are missing from the S3 bucket.

My current hypothesis is that I'm missing some configuration to enable the migration to properly utilize the Payload S3 storage plugin.

import { MigrateUpArgs, MigrateDownArgs } from '@payloadcms/db-mongodb';
import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';
import { UploadCollectionSlug } from 'payload';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import fetch from 'node-fetch';

export async function up({ payload }: MigrateUpArgs): Promise<void> {
  const s3 = new S3Client({
    region: process.env.S3_REGION,
    credentials: {
      accessKeyId: process.env.S3_ACCESS_KEY_ID!,
      secretAccessKey: process.env.S3_SECRET_ACCESS_KEY!,
    },
  });

  const collectionSlug: UploadCollectionSlug = 'blog-media';
  const { docs } = await payload.find({
    collection: collectionSlug,
    depth: 0,
    limit: 1000,
  });

  console.log(`🔍 Found ${docs.length} documents in '${collectionSlug}'`);

  for (const doc of docs) {
    if (!doc.filename) continue;

    const prefix = 'cms-data/blog-media';

    const originalFilename = doc.filename.replace(/-\d+x\d+(?=\.\w+$)/, '');
    const key = `${prefix}/${originalFilename}`;
    console.log(`Original file name: ${originalFilename}`);
    console.log(`key: ${key}`);


    try {
      const command = new GetObjectCommand({
        Bucket: process.env.S3_BUCKET,
        Key: key,
      });

      const signedUrl = await getSignedUrl(s3, command, { expiresIn: 3600 });
      const fileResponse = await fetch(signedUrl);

      if (!fileResponse.ok) {
        throw new Error(`Error fetching from S3: ${fileResponse.statusText}`);
      }

      const fileBuffer = Buffer.from(await fileResponse.arrayBuffer());
      
      console.log(`DOC: ${doc.filename}`);

      await payload.update({
        collection: collectionSlug,
        id: doc.id,
        data: { alt: doc.alt || '' },
        overwriteExistingFiles: true,
        file: {
          data: fileBuffer,
          mimetype: doc.mimeType || 'application/octet-stream',
          size: fileBuffer.length,
          name: originalFilename
        },
      });

      console.log(`✅ Regenerated sizes for: ${originalFilename}`);
    } catch (error) {
      console.error(`❌ Failed to regenerate ${originalFilename}:`, error);
    }
  }

  console.log('🎉 Image regeneration complete!');
}

export async function down({}: MigrateDownArgs): Promise<void> {
  console.log('ℹ️ No rollback needed');
}

[nachomglz]

@camilo220 I don't know if you were able to solve this, but I just ran a script that updated each media (with no data updates, just use the update method for the media) and it re-generates the sizes and uploads the new images and everything, it ends up being easier than initially thought haha.

[nachomglz]

It's actually weird, because sometimes it does update the files in S3 and sometimes it doesn't...

[camilo220]

@nachomglz I wasn’t able to get the script above working, as you mentioned, it behaved differently each time I ran it. In the end, I used this script I found here:

https://www.reddit.com/r/PayloadCMS/comments/1it3ta6/payload_image_size_regeneration

Let me know how it works for you, and apologies for the delay in getting back to you.