fix: status code to avoid navigation with empty props (#60968)

### What?
Currently, when a middleware is active and the path results in a 404,
the server responds with a 200 status code to data requests. We propose
a change to ensure that a 404 status code is returned in these
situations, as expected, solving production issues for the navigation
router.

### Why?
The client data requests (identified with `_next/data` path &
`x-nextjs-data` header) get answered the status code 200. The code below
is executed when there is a version skew for the data requests (e.g
prefetch in production).


https://github.com/vercel/next.js/blob/4125069840ca98981f0e7796f55265af04f3e903/packages/next/src/server/lib/router-server.ts#L217-L227

The version skew consists in the client side version identified with the
buildId in `__NEXT_DATA__ ` tag to be obsolete compared to the server
version (`BUILD_ID` file).

In the case of prefetching, this leads to the code above being executed,
therefore the `prefetch-reducer.ts` handles the response as valid and
sets it in its cache. Which ultimately triggers a navigation with empty
prop, resulting in erroneous behaviours reported in issues and in our
production websites:

https://github.com/vercel/next.js/blob/4125069840ca98981f0e7796f55265af04f3e903/packages/next/src/client/components/router-reducer/reducers/prefetch-reducer.ts#L54-L74

By switching the response to a 404, we trigger this code in the fetch
(`fetchServerResponse`). This change prompts a hard navigation
(mpaNavigation), effectively refreshing the client version and
resynching it with the server version.

https://github.com/vercel/next.js/blob/4125069840ca98981f0e7796f55265af04f3e903/packages/next/src/client/components/router-reducer/fetch-server-response.ts#L125-L134

### How?
We simply update the status code to 404 here:

https://github.com/vercel/next.js/blob/4125069840ca98981f0e7796f55265af04f3e903/packages/next/src/server/lib/router-server.ts#L223

Closes: #60785
Closes: #59295
Fixes: #47516

---------

Co-authored-by: JJ Kasper <[email protected]>

Author: Lezzio

packages/next/src/server/lib/router-server.ts

@@ -180,7 +180,7 @@ export async function initialize(opts: {
         removePathPrefix(invokePath, config.basePath) === '/404'
       ) {
         res.setHeader('x-nextjs-matched-path', parsedUrl.pathname || '')
-        res.statusCode = 200
+        res.statusCode = 404
         res.setHeader('content-type', 'application/json')
         res.end('{}')
         return null

packages/next/src/shared/lib/router/router.ts

@@ -323,30 +323,17 @@ async function withMiddlewareEffects<T extends FetchDataOutput>(
     return null
   }
 
-  try {
-    const data = await options.fetchData()
+  const data = await options.fetchData()
 
-    const effect = await getMiddlewareData(
-      data.dataHref,
-      data.response,
-      options
-    )
+  const effect = await getMiddlewareData(data.dataHref, data.response, options)
 
-    return {
-      dataHref: data.dataHref,
-      json: data.json,
-      response: data.response,
-      text: data.text,
-      cacheKey: data.cacheKey,
-      effect,
-    }
-  } catch {
-    /**
-     * TODO: Revisit this in the future.
-     * For now we will not consider middleware data errors to be fatal.
-     * maybe we should revisit in the future.
-     */
-    return null
+  return {
+    dataHref: data.dataHref,
+    json: data.json,
+    response: data.response,
+    text: data.text,
+    cacheKey: data.cacheKey,
+    effect,
   }
 }
 
@@ -2403,7 +2390,7 @@ export default class Router implements BaseRouter {
                   locale,
                 }),
                 hasMiddleware: true,
-                isServerRender: this.isSsr,
+                isServerRender: false,
                 parseJSON: true,
                 inflightCache: this.sdc,
                 persistCache: !this.isPreview,

test/e2e/middleware-general/test/index.test.ts

@@ -104,14 +104,12 @@ describe('Middleware Runtime', () => {
   function runTests({ i18n }: { i18n?: boolean }) {
     it('should work with notFound: true correctly', async () => {
       const browser = await next.browser('/ssr-page')
-      await browser.eval('window.beforeNav = 1')
       await browser.eval('window.next.router.push("/ssg/not-found-1")')
 
       await check(
         () => browser.eval('document.documentElement.innerHTML'),
         /This page could not be found/
       )
-      expect(await browser.eval('window.beforeNav')).toBe(1)
 
       await browser.refresh()
       await check(