How to request Nextjs app page from a Server a parse the HTML?

[alamenai]

Summary

I have a Next.js application and I want to request a page from a server and parse the content to access a specific element, however when I try to access that page the app returns the content for 1s then an error appears:

Using Thunder extension

As you see in the image above, this is just test page that's called proto:

I thought this issue maybe from Thunder but when I tried the Next.js website it works as expected:

My application uses Redux and Mapbox as third parties.

When I open the app in the browser that error does not appear:

I just want to know the reason behind the issue. Is that related to the JavaScript bundle?

Additional information

No response

Example

No response

[icyJoseph]

Could it be that the JS runtime, or rather browser baked into the Thunder Client, doesn't support some features? I see you are calling localhost:3000, so I can assume your server is in dev mode? that has a bunch of JS that might not be compatible with all browsers.

Try with build and start.

[alamenai]

My page is a server comopnent:

import { getAllBuildings } from "@/actions/buildings"
import { MapViewer } from "@/features/viewer/components"
import { Suspense } from "react"

type SearchParams = {
  street: string
  house_number: string
  zip_code: string
  city: string
  image: boolean
}

export default async function Home({ searchParams }: Readonly<{ searchParams: SearchParams }>) {
  let buildings = null

  const address = {
    street: searchParams.street as string,
    houseNumber: searchParams.house_number as string,
    zipCode: searchParams.zip_code as string,
    city: searchParams.city as string,
  }

  if (address.street && address.houseNumber && address.zipCode && address.city) {
    buildings = await getAllBuildings(address)
  }

  return (
    <Suspense>
      <div className='w-screen h-screen z-50'>
       <p>TESTING</p>
      </div>
    </Suspense>
  )
}

[icyJoseph]

What does the Map component do?

[alamenai]

Well, it loads a map, add some custom layers, then if the image=true in the URL it will take screenshot and return the image, my goal is to get the image when I request the URL:

http://localhost:3001/?street=Latsch&house_number=1b&zip_code=92637&city=Weiden&image=true

Component

"use client"

import { getEnv } from "@/actions/env"
import { takeScreenshot } from "@/utils/screenshot"
import mapboxgl, { LngLatLike } from "mapbox-gl"
import "mapbox-gl/dist/mapbox-gl.css"
import Image from "next/image"
import { useSearchParams } from "next/navigation"
import React, { useEffect, useRef, useState } from "react"

import { get3DModels, getFootPrints, getParcels } from "../actions"
import { createBuildingModelsLayers } from "./layers/3d-building-model"
import { createFootPrintLayers } from "./layers/foot-print"
import { createParcelLayers } from "./layers/parcel"

export const defaultLocation: [number, number] = [10.4515, 51.1657]
export const defaultZoom = 10
export const defaultPitch = 45
export const defaultBearing = 45

// eslint-disable-next-line @typescript-eslint/no-explicit-any
export const MapViewer = ({ buildings }: { buildings: any }) => {
  const searchParams = useSearchParams()

  const shouldTakeScreenshot = searchParams.get("image") === "true"

  const urlZoom = searchParams.get("zoom")

  const urlPitch = searchParams.get("pitch")

  const urlBearing = searchParams.get("bearing")

  const mapContainer = useRef<HTMLDivElement>(null)

  const mapRef = useRef<mapboxgl.Map | null>(null)

  const [isMapLoaded, setIsMapLoaded] = useState(false)

  const [isParcelsLoaded, setIsParcelsLoaded] = useState(false)

  const [isFootprintsLoaded, setIsFootprintsLoaded] = useState(false)

  const [isModelsLoaded, setIsModelsLoaded] = useState(false)

  const [screenshot, setScreenshot] = useState("")

  const [isFullyLoaded, setIsFullyLoaded] = useState(false)

  const footPrintLayersRef = useRef<{
    sourceIds: string[]
    layerIds: string[]
    labelSourceIds: string[]
    labelIds: string[]
    areaLabelSourceIds: string[]
    areaLabelIds: string[]
  }>({
    sourceIds: [],
    layerIds: [],
    labelSourceIds: [],
    labelIds: [],
    areaLabelSourceIds: [],
    areaLabelIds: [],
  })

  useEffect(() => {
    getEnv()
      .then((env) => {
        const token = env.NEXT_PUBLIC_DOCKER_MAPBOX_ACCESS_TOKEN!
        initializeMap(token)
      })
      .catch((error) => {
        console.error("Failed to initialize map:", error)
      })

    return () => mapRef.current?.remove()
    // eslint-disable-next-line react-hooks/exhaustive-deps
  }, [])

  useEffect(() => {
    if (!isMapLoaded || !buildings) return

    let cleanupFunction: () => void

    const firstFootPrintGeometrylayer =
      footPrintLayersRef.current.layerIds.length > 0 ? footPrintLayersRef.current.layerIds[0] : undefined

    getParcels(buildings)
      .then((parcels) => {
        const { bounds, cleanup } = createParcelLayers(mapRef.current!, parcels, firstFootPrintGeometrylayer)
        if (!bounds.isEmpty()) {
          mapRef.current?.fitBounds(bounds, {
            maxZoom: 20,
            pitch: Number(urlPitch) || defaultPitch,
            bearing: Number(urlBearing) || defaultBearing,
          })
        }
        cleanupFunction = cleanup
        setIsParcelsLoaded(true)
      })
      .catch(console.error)

    return () => cleanupFunction?.()
  }, [buildings, isMapLoaded, urlPitch, urlBearing])

  useEffect(() => {
    if (!isMapLoaded || !buildings) return

    let cleanupFunction: () => void

    getFootPrints(buildings).then((footPrints) => {
      const { layerIds, cleanup } = createFootPrintLayers(mapRef.current!, footPrints)
      footPrintLayersRef.current = {
        sourceIds: layerIds.map((layer) => layer.sourceId),
        layerIds: layerIds.map((layer) => layer.layerId),
        labelSourceIds: [],
        labelIds: [],
        areaLabelSourceIds: [],
        areaLabelIds: [],
      }
      cleanupFunction = cleanup
      setIsFootprintsLoaded(true)
    })

    return () => cleanupFunction?.()
  }, [buildings, isMapLoaded])

  useEffect(() => {
    if (!mapRef || !isMapLoaded || !buildings) return

    let layerIds: string[] = [] // Store the layer IDs

    get3DModels(buildings)
      .then((models) => createBuildingModelsLayers(mapRef, models))
      .then((ids) => {
        layerIds = ids
        setIsModelsLoaded(true)
      })
      .catch(console.error)

    return () => {
      if (!mapRef.current?.style) return
      layerIds.forEach((id) => {
        if (mapRef.current?.getLayer(id)) {
          mapRef.current.removeLayer(id)
        }
      })
    }
  }, [buildings, isMapLoaded])

  const initializeMap = (token: string) => {
    if (!mapContainer.current) return

    const lng = buildings?.[0].longitude
    const lat = buildings?.[0].latitude

    const initialCenter = lng && lat ? [parseFloat(lng), parseFloat(lat)] : defaultLocation

    mapboxgl.accessToken = token

    mapRef.current = new mapboxgl.Map({
      container: mapContainer.current,
      style: "mapbox://styles/mapbox/satellite-streets-v12",
      center: initialCenter as LngLatLike,
      zoom: (() => {
        if (urlZoom) return parseFloat(urlZoom)
        return initialCenter === defaultLocation ? defaultZoom : 20
      })(),
      pitch: Number(urlPitch) || defaultPitch,
      antialias: true,
      bearing: Number(urlBearing) || defaultBearing,
    })

    mapRef.current.on("load", () => {
      setIsMapLoaded(true)
    })

    window.mapRef = mapRef
  }

  // Modify your existing useEffect for screenshot
  useEffect(() => {
    const allLayersLoaded = isMapLoaded && isParcelsLoaded && isFootprintsLoaded && isModelsLoaded

    if (shouldTakeScreenshot && allLayersLoaded && mapRef.current) {
      // Mark the canvas as loaded
      const canvas = mapRef.current.getCanvas()
      canvas.setAttribute("data-loaded", "true")

      setIsFullyLoaded(true)
    }
  }, [isMapLoaded, isParcelsLoaded, isFootprintsLoaded, isModelsLoaded, shouldTakeScreenshot])

  useEffect(() => {
    if (isFullyLoaded && shouldTakeScreenshot) {
      takeScreenshot(mapRef).then((dataUrl) => {
        setScreenshot(dataUrl)
      })
    }
  }, [isFullyLoaded, shouldTakeScreenshot])

  return (
    <>
      <div ref={mapContainer} className='h-screen w-screen z-50' />
      {shouldTakeScreenshot && screenshot ? <Image src={screenshot} fill alt='' className='rounded-2xl z-50' /> : null}
    </>
  )
}

Though, I tried to wait using timeout ( I expected the issue is due the timing ) but I still get this HTML from Postman:

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charSet="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <link rel="stylesheet" href="/_next/static/css/app/layout.css?v=1738855464921"
        data-precedence="next_static/css/app/layout.css" />
    <link rel="stylesheet" href="/_next/static/css/app/page.css?v=1738855464921"
        data-precedence="next_static/css/app/page.css" />
    <link rel="preload" as="script" fetchPriority="low" href="/_next/static/chunks/webpack.js?v=1738855464921" />
    <script src="/_next/static/chunks/main-app.js?v=1738855464921" async=""></script>
    <script src="/_next/static/chunks/app-pages-internals.js" async=""></script>
    <script src="/_next/static/chunks/app/page.js" async=""></script>
    <link rel="apple-touch-icon" sizes="180x180" href="/favicons/apple-touch-icon.png" />
    <link rel="icon" type="image/png" sizes="32x32" href="/favicons/favicon-32x32.png" />
    <link rel="icon" type="image/png" sizes="16x16" href="/favicons/favicon-16x16.png" />
    <title>credium | Map Viewer</title>
    <meta name="next-size-adjust" />
    <script src="/_next/static/chunks/polyfills.js" noModule=""></script>
</head>

<body class="__variable_d65c78 antialiased">
    <div class="w-screen h-screen z-50">
        <div class="h-screen w-screen z-50" data-custom=""></div>
    </div>
    <script src="/_next/static/chunks/webpack.js?v=1738855464921" async=""></script>
    <script>
        (self.__next_f=self.__next_f||[]).push([0]);self.__next_f.push([2,null])
    </script>
    <script>
        self.__next_f.push([1,"1:HL[\"/_next/static/media/a34f9d1faa5f3315-s.p.woff2\",\"font\",{\"crossOrigin\":\"\",\"type\":\"font/woff2\"}]\n2:HL[\"/_next/static/css/app/layout.css?v=1738855464921\",\"style\"]\n3:HL[\"/_next/static/css/app/page.css?v=1738855464921\",\"style\"]\n0:D{\"name\":\"r2\",\"env\":\"Server\"}\n"])
    </script>
    <script>
        self.__next_f.push([1,"4:I[\"(app-pages-browser)/../../../node_modules/next/dist/client/components/app-router.js\",[\"app-pages-internals\",\"static/chunks/app-pages-internals.js\"],\"\"]\n8:I[\"(app-pages-browser)/../../../node_modules/next/dist/client/components/layout-router.js\",[\"app-pages-internals\",\"static/chunks/app-pages-internals.js\"],\"\"]\n9:I[\"(app-pages-browser)/../../../node_modules/next/dist/client/components/render-from-template-context.js\",[\"app-pages-internals\",\"static/chunks/app-pages-internals.js\"],\"\"]\nd:I[\"(app-pages-browser)/../../../node_modules/next/dist/client/components/error-boundary.js\",[\"app-pages-internals\",\"static/chunks/app-pages-internals.js\"],\"\"]\n5:D{\"name\":\"\",\"env\":\"Server\"}\n6:D{\"name\":\"Home\",\"env\":\"Server\"}\n7:D{\"name\":\"RootLayout\",\"env\":\"Server\"}\na:D{\"name\":\"NotFound\",\"env\":\"Server\"}\na:[[\"$\",\"title\",null,{\"children\":\"404: This page could not be found.\"}],[\"$\",\"div\",null,{\"style\":{\"fontFamily\":\"system-ui,\\\"Segoe UI\\\",Roboto,Helvetica,Arial,sans-serif,\\\"Apple Color Emoji\\\",\\\"Segoe UI Emoji\\\"\",\"height\":\"100vh\",\"textAlign\":\"center\",\"display\":\"flex\",\"flexDirection\":\"column\",\"alignItems\":\"center\",\"justifyContent\":\"center\"},\"children\":[\"$\",\"div\",null,{\"children\":[[\"$\",\"style\",null,{\"dangerouslySetInnerHTML\":{\"__html\":\"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}\"}}],[\"$\",\"h1\",null,{\"className\":\"next-error-h1\",\"style\":{\"display\":\"inline-block\",\"margin\":\"0 20px 0 0\",\"padding\":\"0 23px 0 0\",\"fontSize\":24,\"fontWeight\":500,\"verticalAlign\":\"top\",\"lineHeight\":\"49px\"},\"children\":\"404\"}],[\"$\",\"div\",null,{\"style\":{\"display\":\"inline-block\"},\"children\":[\"$\",\"h2\",null,{\"style\":{\"fontSize\":14,\"fontWeight\":400,\"lineHeight\":\"49px\",\"margin\":0},\"children\":\"This page could not be found.\"}]}]]}]}]]\n7:[\"$\",\"html\",null,{\"lang\":\"en\",\"children\":[[\"$\",\"head\",null,{\"children\":[[\"$\",\"link\",null,{\"rel\":\"apple-touch-icon\",\"sizes\":\"180x180\",\"href\":\"/favicons/apple-touch-icon.p"])
    </script>
    <script>
        self.__next_f.push([1,"ng\"}],[\"$\",\"link\",null,{\"rel\":\"icon\",\"type\":\"image/png\",\"sizes\":\"32x32\",\"href\":\"/favicons/favicon-32x32.png\"}],[\"$\",\"link\",null,{\"rel\":\"icon\",\"type\":\"image/png\",\"sizes\":\"16x16\",\"href\":\"/favicons/favicon-16x16.png\"}]]}],[\"$\",\"body\",null,{\"className\":\"__variable_d65c78 antialiased\",\"children\":[\"$\",\"$L8\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\"],\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"errorScripts\":\"$undefined\",\"template\":[\"$\",\"$L9\",null,{}],\"templateStyles\":\"$undefined\",\"templateScripts\":\"$undefined\",\"notFound\":\"$a\",\"notFoundStyles\":[]}]}]]}]\nb:D{\"name\":\"rQ\",\"env\":\"Server\"}\nb:null\nc:D{\"name\":\"\",\"env\":\"Server\"}\ne:[]\n0:[\"$\",\"$L4\",null,{\"buildId\":\"development\",\"assetPrefix\":\"\",\"urlParts\":[\"\",\"?street=Latsch\u0026house_number=1b\u0026zip_code=92637\u0026city=Weiden\u0026image=true\"],\"initialTree\":[\"\",{\"children\":[\"__PAGE__?{\\\"street\\\":\\\"Latsch\\\",\\\"house_number\\\":\\\"1b\\\",\\\"zip_code\\\":\\\"92637\\\",\\\"city\\\":\\\"Weiden\\\",\\\"image\\\":\\\"true\\\"}\",{}]},\"$undefined\",\"$undefined\",true],\"initialSeedData\":[\"\",{\"children\":[\"__PAGE__\",{},[[\"$L5\",\"$L6\",[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/css/app/page.css?v=1738855464921\",\"precedence\":\"next_static/css/app/page.css\",\"crossOrigin\":\"$undefined\"}]]],null],null]},[[[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/css/app/layout.css?v=1738855464921\",\"precedence\":\"next_static/css/app/layout.css\",\"crossOrigin\":\"$undefined\"}]],\"$7\"],null],null],\"couldBeIntercepted\":false,\"initialHead\":[\"$b\",\"$Lc\"],\"globalErrorComponent\":\"$d\",\"missingSlots\":\"$We\"}]\n"])
    </script>
    <script>
        self.__next_f.push([1,"f:I[\"(app-pages-browser)/./features/viewer/components/index.tsx\",[\"app/page\",\"static/chunks/app/page.js\"],\"MapViewer\"]\nc:[[\"$\",\"meta\",\"0\",{\"name\":\"viewport\",\"content\":\"width=device-width, initial-scale=1\"}],[\"$\",\"meta\",\"1\",{\"charSet\":\"utf-8\"}],[\"$\",\"title\",\"2\",{\"children\":\"credium | Map Viewer\"}],[\"$\",\"meta\",\"3\",{\"name\":\"next-size-adjust\"}]]\n5:null\n6:[\"$\",\"div\",null,{\"className\":\"w-screen h-screen z-50\",\"children\":[\"$\",\"$Lf\",null,{\"buildings\":[{\"buildingId\":\"DEBYvAAAAAAqm5qR\",\"buildingFootprintGeometry\":\"https://credium-api.azure-api.net/dev/geometry?GeomId=DEBYvAAAAAAqm5qR\u0026GeomType=0\",\"parcelInformation\":{\"parcels\":[]},\"longitude\":12.1061343,\"latitude\":49.6754415}]}]}]\n"])
    </script>
</body>

</html>

Preview

As you see on the top, the body contains only the divs but because the image depends on the screenshot state it does not rendered.

[alamenai]

I guess, this is because the page relies on client-rendering ( CSR ) or some browser-specific features that are not available in a server-side environment.

[alamenai]

The only solution I suggested for the back-end developer is to use pupeeter and access the image component.

[alamenai]

Hi @icyJoseph , I made a simple repro where I understand the reason behind the issue. IMHO, the issue is due the useEffect timing issues:

const TestComponent = ()=>{

 const [test, setTest] = useState("default")

  useEffect(() => {
    setTimeout(() => {
      setTest("visible")
    }, 2000)
  }, [])

  return test === "visible" && <p>{test}</p>
}

When I request this component from Postman, I got empty page, because the page is returned before the test has visible value.

[icyJoseph]

Right, in other words, your page doesn't support disabled JavaScript. As said before, these baked in clients, won't necessarily support a whole bunch of things.

[alamenai]

Thank you @icyJoseph for you support.