TRSS Instability Issues

[steelhead31]

Here are my notes regarding the periodic instability issues being encountered by trss..

trss_log_excerpt.txt

I've attached the log excerpt, I've used to investigate...

Around 18/12/2025 05:37 failures started occurring when accessing data-backed pages in the AQA Test Tools service. The webpage remained up, but no data was returned.

Symptoms:

• Some pages (for example /output/test, /deepHistory, /testPerPlatform) initially loaded.
• Data-heavy sections failed shortly after page load.
• Browsers received 502 Bad Gateway errors for multiple API calls.
• The site appeared unstable or partially broken rather than fully offline.

What the Logs Show

Nginx (Frontend / Reverse Proxy)

• Nginx was running normally and accepting client requests.
• Requests to frontend routes returned HTTP 200 with very low latency (0–1 ms).
• Requests to backend API endpoints (/api/*) consistently failed.

Typical error:

connect() failed (111: Connection refused) while connecting to upstream

Nginx returned:

502 Bad Gateway

The upstream configured for these requests was:

http://172.21.0.5:3001/

Root Cause

The backend API service listening on port 3001 was not running or not accepting connections.

Key points:

• The IP address was reachable, but the TCP connection was refused.
• This means nothing was listening on port 3001 at the time.
• Nginx itself was healthy and behaving correctly.
• The issue was not caused by nginx, the browser, or general networking.

This strongly indicates that the API container or process:

• Crashed
• Exited due to an unhandled error
• Was killed (for example OOM kill)
• Restarted and failed to come back up

Likely Contributing Factors

Based on surrounding context and earlier MongoDB errors:

• The API service likely depends on MongoDB.
• A MongoDB connectivity issue may have caused the API process to exit.
• The API did not recover automatically, leaving nginx pointing at a dead upstream.

Impact

• All /api/* endpoints were unavailable.
• Pages relying on live API data failed.
• Users experienced intermittent or broken functionality.
• MongoDB logged client disconnects as a downstream effect.

[andrew-m-leonard]

Has the "TRSSBackend" and/or "TRSSFrontend" systemd service stopped when this occurs?
Could we try and make it auto-restartable ?

Suspect "TRSSFrontend" stops listening, as backend.js looks like it handles all exceptions, but frontend.js perhaps not so.

Suggestions by co-pilot:

1. Instead of app.listen(...) (which hides the server instance), explicitly create the HTTP server and attach error handlers
2. Ensure the process restarts automatically (systemd or PM2)
3. Add error handling to server instance of app.listen()

I suspect a belt&braces approach would be 2 & 3.

[andrew-m-leonard]

PR to add Express server error logging: https://github.com/adoptium/aqa-test-tools/pull/1147/files

[jiekang]

Maybe more error logging/handling for the backend would help too.

[jiekang]

A very simple local test crashes:

Run Mongodb.
Run TRSS Backend
Run TRSS Client

Kill Mongodb
Wait for next monitorBuild cycle. TRSS backend crashes:

11:39:16 AM - error: Exception in database query:  message=connect ECONNREFUSED 127.0.0.1:27017, stack=MongoServerSelectionError: connect ECONNREFUSED 127.0.0.1:27017
    at Timeout._onTimeout (/home/jkang/Work/src/github.com/adoptium/aqa-test-tools/TestResultSummaryService/node_modules/mongodb/lib/core/sdam/topology.js:438:30)
    at listOnTimeout (node:internal/timers:588:17)
    at process.processTimers (node:internal/timers:523:7), name=MongoServerSelectionError, type=Single, setName=null, maxSetVersion=null, maxElectionId=null, , stale=false, compatible=true, compatibilityError=null, logicalSessionTimeoutMinutes=null, heartbeatFrequencyMS=10000, localThresholdMS=15, commonWireVersion=27
/home/jkang/Work/src/github.com/adoptium/aqa-test-tools/TestResultSummaryService/EventHandler.js:87
                    ...task,
                       ^

ReferenceError: task is not defined
    at EventHandler.monitorBuild (/home/jkang/Work/src/github.com/adoptium/aqa-test-tools/TestResultSummaryService/EventHandler.js:87:24)

Node.js v22.21.1
[nodemon] app crashed - waiting for file changes before starting...

[jiekang]

I added some logging and error handling for the backend: #1148

However now there are dangling/waiting HTTP requests in the scenario where MongoDB queries fail...

[jiekang]

Note: Today while querying TRSS apis like https://trss.adoptium.net/api/getTotals?buildName=openjdk23-pipeline&url=https://ci.adoptium.net/job/build-scripts&buildNum=113, I was getting intermittent 503's "Service Temporarily Unavailable"

E.g. monitor network requests in the browser when repeatedly visiting https://trss.adoptium.net/ with a hard refresh to prevent any caching.

[jiekang]

Error logs also mention something like:

12:24:21 AM - error: Exception in BuildProcessor: message=Cannot read properties of null (reading 'startsWith'), stack=TypeError: Cannot read properties of null (reading 'startsWith')

This is noted and analyzed in this issue: #891

[jiekang]

TRSS backend experienced an OOM, there may be a leak of strings. Needs further analysis.

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

[jiekang]

I have been trying to replicate this issue on a local instance... I've only run it for a 8 hour period

My experience so far is:

• The backend is memory stable at 50mb, except for when there are "streaming" type jobs running
• When there are, I've seen memory go up to 500 mb, but my local instance hasn't gone beyond that. It does eventually GC and bring it down to 50mb too...
• I have heap dumps across a few hours on my local instance. The primary culprit taking memory are strings of the console output of jobs which is a part of the "streaming" type jobs. Some of the strings are as big as 12mb, but I haven't been able to determine why. The data in the larger string seems to be gibberish, whereas all the smaller strings are not.

[jiekang]

trss.adoptium.net backend OOM again:

<--- Last few GCs --->

[109:0x20eb88a0] 346311996 ms: Scavenge 7968.7 (8205.7) -> 7968.5 (8216.5) MB, 9.8 / 0.0 ms  (average mu = 0.636, current mu = 0.782) allocation failure; 
[109:0x20eb88a0] 346312036 ms: Scavenge 7975.1 (8216.5) -> 7974.8 (8217.7) MB, 15.4 / 0.0 ms  (average mu = 0.636, current mu = 0.782) allocation failure; 
[109:0x20eb88a0] 346314225 ms: Scavenge 7976.2 (8217.7) -> 7975.5 (8240.2) MB, 2181.8 / 0.0 ms  (average mu = 0.636, current mu = 0.782) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

[jiekang]

There are also instances of:

TypeError: Cannot read properties of undefined (reading '_id')
    at module.exports (/usr/src/app/routes/getTestById.js:7:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

TypeError: Cannot read properties of undefined (reading 'buildName')
    at module.exports (/usr/src/app/routes/getHistoryPerTest.js:7:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[steelhead31]

I've made some changes to the container setup..

Both of these changes made in docker-compose.yml

New Container Added:

memwatch : monitoring memory usage of aqa-test-tools-server-1, logging data to /home/jenkins/trss/aqa-test-tools/dumps

  memwatch:
    profiles:
      - prod
    image: alpine:3.20
    container_name: aqa-test-tools-server-memwatch
    depends_on:
      - server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./dumps:/dumps
    environment:
      TARGET_CONTAINER: aqa-test-tools-server-1
      OUT_FILE: /dumps/aqa-test-tools-server-1_mem.csv
      INTERVAL_SECONDS: "10"
    command: >
      sh -lc '
        apk add --no-cache docker-cli >/dev/null;
        out="$$OUT_FILE";
        if [ ! -f "$$out" ]; then
          echo "utc_ts,mem_used,mem_limit,mem_percent" >> "$$out";
        fi
        while true; do
          ts=$$(date -u +"%Y-%m-%dT%H:%M:%SZ");
          line=$$(docker stats --no-stream --format "{{.MemUsage}},{{.MemPerc}}" "$$TARGET_CONTAINER" 2>/dev/null || true);
          if [ -n "$$line" ]; then
            memusage=$${line%%,*}; perc=$${line##*,};
            used=$${memusage%% /*}; limit=$${memusage##*/ };
            echo "$$ts,$$used,$$limit,$$perc" >> "$$out";
          fi
          sleep "$$INTERVAL_SECONDS";
        done
      '
    restart: unless-stopped

And some additional logging Added To Aqa-Test-Tools-Server-1 container :

NODE_OPTIONS: "--max-old-space-size=8192 --report-on-fatalerror --report-uncaught-exception --report-directory=/dumps --report-filename=trss-report-.%p.%t.json"