Encoding issue on binary data when using rewrites in AWS

[ipc-zpg]

Originally reported to SST here: sst/sst#5317

When using a rewrite in NextJS in a AWS deployment, binary data appears to be being encoded incorrectly. When running locally the issue is not present.

In my case, this is causing images to be corrupted.

Tested with SST version 3.4.56, and version 3.5.0

next.config.ts

import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  rewrites: async () => {
    return {
      beforeFiles: [],
      afterFiles: [],
      fallback: [
        {
          source: "/google_logo",
          destination: "https://www.google.com/images/branding/googlelogo/2x/googlelogo_dark_color_272x92dp.png",
        },
      ]
    }
  }
};

export default nextConfig;

sst.config.ts

// eslint-disable-next-line @typescript-eslint/triple-slash-reference
/// <reference path="./.sst/platform/config.d.ts" />

export default $config({
  app(input) {
    return {
      name: "ipc-test",
      removal: input?.stage === "production" ? "retain" : "remove",
      protect: ["production"].includes(input?.stage),
      home: "aws",
    };
  },
  async run() {
    new sst.aws.Nextjs("MyWeb");
  },
});

Testing:

$ curl https://www.google.com/images/branding/googlelogo/2x/googlelogo_dark_color_272x92dp.png -o real.png
$ curl  http://localhost:3000/google_logo -o local.png
$ curl  https://d29t9gko77stcb.cloudfront.net/google_logo -o cloudfront.png

$ ls -la *
.rw-r--r-- ianchristian wheel  12 KB Wed Jan  8 08:36:35 2025   cloudfront.png
.rw-r--r-- ianchristian wheel 6.9 KB Wed Jan  8 08:33:40 2025   local.png
.rw-r--r-- ianchristian wheel 6.9 KB Wed Jan  8 08:31:20 2025   real.png

$ md5sum *
efa5b41728d0f52920fd230782066ee4  cloudfront.png
82381fe256523aa6a6061db820818e66  local.png
82381fe256523aa6a6061db820818e66  real.png

$ hexdump -C  real.png | head -n 3
00000000  89 50 4e 47 0d 0a 1a 0a  00 00 00 0d 49 48 44 52  |.PNG........IHDR|
00000010  00 00 02 20 00 00 00 b8  08 03 00 00 00 ed fd a7  |... ............|
00000020  29 00 00 03 00 50 4c 54  45 00 00 00 00 00 00 00  |)....PLTE.......|

$ hexdump -C  cloudfront.png | head -n 3
00000000  ef bf bd 50 4e 47 0d 0a  1a 0a 00 00 00 0d 49 48  |...PNG........IH|
00000010  44 52 00 00 02 20 00 00  00 ef bf bd 08 03 00 00  |DR... ..........|
00000020  00 ef bf bd ef bf bd ef  bf bd 29 00 00 03 00 50  |..........)....P|

Update

I have found that I can fix this partly by turning on lambda streaming:

open-next.config.ts

export default {
  default: {
    override: {
      wrapper: "aws-lambda-streaming"
    }
  }

However, this introduces 2 issues:

• content type is application/json for everything
• transfer speed seems awful - 8 seconds for a 1.3MB JPEG.

For content-type, I 'fixed' this by setting the header in middleware before the rewrite happens, but that's a nasty work around.

Update 2

A second work around that doesn't affect performance in the same was the first update does, is to specifically set the content type.

To prove/test this:

middleware.ts

import { NextRequest, NextResponse } from "next/server";

export async function middleware(request: NextRequest) {
  const pathname = request.nextUrl.pathname;
  if (pathname.endsWith("ct_fixed")) {
    console.log("fixing content type");
    request.headers.set("Content-Type", "image/png");
  } else {
    console.log("NOT fixing content type");
  }
  return NextResponse.next({ request });
}

next.config.ts

import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  rewrites: async () => {
    return {
      beforeFiles: [],
      afterFiles: [],
      fallback: [
        {
          source: "/google_logo",
          destination: "https://www.google.com/images/branding/googlelogo/2x/googlelogo_dark_color_272x92dp.png",
        },
        {
          source: "/google_logo_ct_fixed",
          destination: "https://www.google.com/images/branding/googlelogo/2x/googlelogo_dark_color_272x92dp.png",
        },
      ]
    }
  }
};

export default nextConfig;

Testing:

❯ curl  https://d29t9gko77stcb.cloudfront.net/google_logo -o cloudfront-1.png
❯ curl  https://d29t9gko77stcb.cloudfront.net/google_logo_ct_fixed -o cloudfront-2.png
❯ ls -la
.rw-r--r-- ianchristian wheel  12 KB Thu Jan  9 09:56:39 2025  cloudfront-1.png
.rw-r--r-- ianchristian wheel 6.9 KB Thu Jan  9 09:56:52 2025  cloudfront-2.png

Notice here that cloudfront-1 (not content-type fixed) has a size of 12kb due to extra encoding, and cloudfront-2 (content-type fixed) has the correct size, and is not corrupted.

[conico974]

@ipc-zpg could you test with #698 (comment)

[ipc-zpg]

@conico974 I'll try to get this tested today or tomorrow. Thanks so much for a speedy commit!

[ipc-zpg]

@conico974 - I've tried to test, but hit limitations in my knowledge (I'm mostly an infra guy!).

While I've added the package using your suggestion (pnpm add https://pkg.pr.new/@opennextjs/aws@698), I think because I'm using SST which is defining it's own version of things, it's getting ignored.

The test looks great to me - but does introduce a dependancy on no one updating the logo on the website, I'm not sure if that's something worth worrying about though?

[sommeeeer]

you probably need a custom buildCommand on the sst.aws.Nextjs component. something like this:

new sst.aws.Nextjs('MyWeb', {
  buildCommand:
    'pnpx https://pkg.pr.new/@opennextjs/aws@698 build',
});

you probably want to add it as a dev dependency btw, if you need to add it. this is only required if you want the typing in open-next.config.ts. for the deploy its not needed as a dependency as long as you use SST

[ipc-zpg]

I can now confirm that this has worked! Thanks for helping me figure out SST as well.

I'll leave this issue open as the PR is still open - however everything looks great to me.

All the best