feat: reduce memory usage by manuel-rw · Pull Request #4740 · homarr-labs/homarr

manuel-rw · 2025-12-23T20:45:15Z

Homarr

Thank you for your contribution. Please ensure that your pull request meets the following pull request:

Builds without warnings or errors (pnpm build, autofix with pnpm format:fix)
Pull request targets dev branch
Commits follow the conventional commits guideline
No shorthand variable names are used (eg. x, y, i or any abbrevation)
Documentation is up to date. Create a pull request here.

TODO:

Rebase to dev
Split into multiple PRs?
What to do with documentation?

potential fix for #3759 , code by @AartSchinkel

packages/auth/configuration.ts

deepsource-io · 2025-12-23T20:47:12Z

Here's the code health analysis summary for commits 99f2842..345ac1e. View details on DeepSource ↗.

Analysis Summary

Analyzer	Status	Summary	Link
JavaScript	❌ Failure	❗ 2 occurences introduced	View Check ↗

💡 If you’re a repository administrator, you can configure the quality gates from the settings.

Co-authored-by: AartSchinkel <aartschinkel@iservecloud.com> Co-authored-by: Meier Lukas <meierschlumpf@gmail.com>

azy2k9 · 2025-12-24T12:52:58Z

According to cursor ai, you could refactor /packages/integrations/src/base/creator.ts to use dynamic imports for the integrations to ensure only the ones used by the end user are loaded into memory. If someone wants to try this and benchmark the code would look like:

import type { IntegrationKind } from "@homarr/definitions";

import type { Integration, IntegrationInput } from "./integration";

export const createIntegrationAsync = async <TKind extends keyof typeof integrationCreators>(
  integration: IntegrationInput & { kind: TKind },
) => {
  if (!(integration.kind in integrationCreators)) {
    throw new Error(
      `Unknown integration kind ${integration.kind}. Did you forget to add it to the integration creator?`,
    );
  }

  const creatorLoader = integrationCreators[integration.kind];
  const creator = await creatorLoader();

  // factories are an array, to differentiate in js between class constructors and functions
  if (Array.isArray(creator)) {
    return (await creator[0](integration)) as IntegrationInstanceOfKind<TKind>;
  }

  return new (creator as IntegrationInstance)(integration) as IntegrationInstanceOfKind<TKind>;
};

type IntegrationInstance = new (integration: IntegrationInput) => Integration;

// factories are an array, to differentiate in js between class constructors and functions
export const integrationCreators = {
  piHole: async () => [(await import("../pi-hole/pi-hole-integration-factory")).createPiHoleIntegrationAsync],
  adGuardHome: async () => (await import("../adguard-home/adguard-home-integration")).AdGuardHomeIntegration,
  homeAssistant: async () => (await import("../homeassistant/homeassistant-integration")).HomeAssistantIntegration,
  jellyfin: async () => (await import("../jellyfin/jellyfin-integration")).JellyfinIntegration,
  plex: async () => (await import("../plex/plex-integration")).PlexIntegration,
  sonarr: async () => (await import("../media-organizer/sonarr/sonarr-integration")).SonarrIntegration,
  radarr: async () => (await import("../media-organizer/radarr/radarr-integration")).RadarrIntegration,
  sabNzbd: async () => (await import("../download-client/sabnzbd/sabnzbd-integration")).SabnzbdIntegration,
  nzbGet: async () => (await import("../download-client/nzbget/nzbget-integration")).NzbGetIntegration,
  qBittorrent: async () => (await import("../download-client/qbittorrent/qbittorrent-integration")).QBitTorrentIntegration,
  deluge: async () => (await import("../download-client/deluge/deluge-integration")).DelugeIntegration,
  transmission: async () => (await import("../download-client/transmission/transmission-integration")).TransmissionIntegration,
  aria2: async () => (await import("../download-client/aria2/aria2-integration")).Aria2Integration,
  jellyseerr: async () => (await import("../jellyseerr/jellyseerr-integration")).JellyseerrIntegration,
  overseerr: async () => (await import("../overseerr/overseerr-integration")).OverseerrIntegration,
  prowlarr: async () => (await import("../prowlarr/prowlarr-integration")).ProwlarrIntegration,
  openmediavault: async () => (await import("../openmediavault/openmediavault-integration")).OpenMediaVaultIntegration,
  lidarr: async () => (await import("../media-organizer/lidarr/lidarr-integration")).LidarrIntegration,
  readarr: async () => (await import("../media-organizer/readarr/readarr-integration")).ReadarrIntegration,
  dashDot: async () => (await import("../dashdot/dashdot-integration")).DashDotIntegration,
  tdarr: async () => (await import("../media-transcoding/tdarr-integration")).TdarrIntegration,
  proxmox: async () => (await import("../proxmox/proxmox-integration")).ProxmoxIntegration,
  emby: async () => (await import("../emby/emby-integration")).EmbyIntegration,
  nextcloud: async () => (await import("../nextcloud/nextcloud.integration")).NextcloudIntegration,
  unifiController: async () => (await import("../unifi-controller/unifi-controller-integration")).UnifiControllerIntegration,
  opnsense: async () => (await import("../opnsense/opnsense-integration")).OPNsenseIntegration,
  github: async () => (await import("../github/github-integration")).GithubIntegration,
  dockerHub: async () => (await import("../docker-hub/docker-hub-integration")).DockerHubIntegration,
  gitlab: async () => (await import("../gitlab/gitlab-integration")).GitlabIntegration,
  npm: async () => (await import("../npm/npm-integration")).NPMIntegration,
  codeberg: async () => (await import("../codeberg/codeberg-integration")).CodebergIntegration,
  linuxServerIO: async () => (await import("../linuxserverio/linuxserverio-integration")).LinuxServerIOIntegration,
  gitHubContainerRegistry: async () => (await import("../github-container-registry/github-container-registry-integration")).GitHubContainerRegistryIntegration,
  ical: async () => (await import("../ical/ical-integration")).ICalIntegration,
  quay: async () => (await import("../quay/quay-integration")).QuayIntegration,
  ntfy: async () => (await import("../ntfy/ntfy-integration")).NTFYIntegration,
  mock: async () => (await import("../mock/mock-integration")).MockIntegration,
  truenas: async () => (await import("../truenas/truenas-integration")).TrueNasIntegration,
} satisfies Record<
  IntegrationKind,
  () => Promise<IntegrationInstance | [(input: IntegrationInput) => Promise<Integration>]>
>;

type IntegrationInstanceOfKind<TKind extends keyof typeof integrationCreators> = {
  [kind in TKind]: Awaited<ReturnType<(typeof integrationCreators)[kind]>> extends [
    (input: IntegrationInput) => Promise<Integration>,
  ]
    ? Awaited<ReturnType<Awaited<ReturnType<(typeof integrationCreators)[kind]>>[0]>>
    : Awaited<ReturnType<(typeof integrationCreators)[kind]>> extends IntegrationInstance
      ? InstanceType<Awaited<ReturnType<(typeof integrationCreators)[kind]>>>
      : never;
}[TKind];

azy2k9 · 2025-12-24T12:57:12Z

Cursor Ai also recommended to change packages/integrations/src/index.ts to convert all class exports to export type instead, this ensures that other packages (like the API) can still use the types for safety, but won't accidentally trigger a full static load of the implementation. If someone wants to try benchmarking this change the code would be:

// General integrations (Types only to avoid loading every integration into memory)
export type { AdGuardHomeIntegration } from "./adguard-home/adguard-home-integration";
export type { Aria2Integration } from "./download-client/aria2/aria2-integration";
export type { DelugeIntegration } from "./download-client/deluge/deluge-integration";
export type { NzbGetIntegration } from "./download-client/nzbget/nzbget-integration";
export type { QBitTorrentIntegration } from "./download-client/qbittorrent/qbittorrent-integration";
export type { SabnzbdIntegration } from "./download-client/sabnzbd/sabnzbd-integration";
export type { TransmissionIntegration } from "./download-client/transmission/transmission-integration";
export type { HomeAssistantIntegration } from "./homeassistant/homeassistant-integration";
export type { JellyfinIntegration } from "./jellyfin/jellyfin-integration";
export type { JellyseerrIntegration } from "./jellyseerr/jellyseerr-integration";
export type { LidarrIntegration } from "./media-organizer/lidarr/lidarr-integration";
export type { RadarrIntegration } from "./media-organizer/radarr/radarr-integration";
export type { ReadarrIntegration } from "./media-organizer/readarr/readarr-integration";
export type { SonarrIntegration } from "./media-organizer/sonarr/sonarr-integration";
export type { NextcloudIntegration } from "./nextcloud/nextcloud.integration";
export type { NTFYIntegration } from "./ntfy/ntfy-integration";
export type { OpenMediaVaultIntegration } from "./openmediavault/openmediavault-integration";
export type { OverseerrIntegration } from "./overseerr/overseerr-integration";
export type { PiHoleIntegrationV5 } from "./pi-hole/v5/pi-hole-integration-v5";
export type { PiHoleIntegrationV6 } from "./pi-hole/v6/pi-hole-integration-v6";
export type { PlexIntegration } from "./plex/plex-integration";
export type { ProwlarrIntegration } from "./prowlarr/prowlarr-integration";
export type { TrueNasIntegration } from "./truenas/truenas-integration";
export type { OPNsenseIntegration } from "./opnsense/opnsense-integration";
export type { ICalIntegration } from "./ical/ical-integration";

// Types
export type { IntegrationInput } from "./base/integration";
export type { DownloadClientJobsAndStatus } from "./interfaces/downloads/download-client-data";
export type { ExtendedDownloadClientItem } from "./interfaces/downloads/download-client-items";
export type { ExtendedClientStatus } from "./interfaces/downloads/download-client-status";
export type {
  FirewallInterface,
  FirewallCpuSummary,
  FirewallInterfacesSummary,
  FirewallVersionSummary,
  FirewallMemorySummary,
} from "./interfaces/firewall-summary/firewall-summary-types";
export type { SystemHealthMonitoring } from "./interfaces/health-monitoring/health-monitoring-types";
export { UpstreamMediaRequestStatus } from "./interfaces/media-requests/media-request-types";
export type { MediaRequestList, MediaRequestStats } from "./interfaces/media-requests/media-request-types";
export type { StreamSession } from "./interfaces/media-server/media-server-types";
export type {
  TdarrQueue,
  TdarrPieSegment,
  TdarrStatistics,
  TdarrWorker,
} from "./interfaces/media-transcoding/media-transcoding-types";
export type { ReleasesRepository, ReleaseResponse } from "./interfaces/releases-providers/releases-providers-types";
export type { Notification } from "./interfaces/notifications/notification-types";

// Schemas
export { downloadClientItemSchema } from "./interfaces/downloads/download-client-items";

// Helpers
export { createIntegrationAsync } from "./base/creator";

Not sure how or why this would work but might be worth trying since the code is here and it sounds like of the devs have a bench marking mechanism setup (maybe its some nice low hanging fruit)

azy2k9 · 2025-12-24T13:06:19Z

How can i get access to submitting a PR with the suggestions Cursor has made along with some other ones its told me about? It might be easier for me to raise a PR for you guys to test? @manuel-rw @Meierschlumpf

manuel-rw · 2025-12-24T13:09:09Z

You can fork, check out this branch, make your changes and then push to your fork. Then, create a PR targeting this branch.

Note that AI suggestions can be hallucinated, please review your changes yourself and test whether they work.

azy2k9 · 2025-12-24T16:07:40Z

You can fork, check out this branch, make your changes and then push to your fork. Then, create a PR targeting this branch.

Note that AI suggestions can be hallucinated, please review your changes yourself and test whether they work.

How are you guys checking memory usage for the application?

You can fork, check out this branch, make your changes and then push to your fork. Then, create a PR targeting this branch.

Note that AI suggestions can be hallucinated, please review your changes yourself and test whether they work.

How do i run this project locally? Ive forked, cloned and:

Run pnpm install
Running docker desktop in the background
Copied the .env.example, filled in the missing variables and created a .env file
Run pnpm build
Run pnpm start but its struggling to start the web socket server

manuel-rw · 2025-12-27T23:16:44Z

Hi, you would probably be better off forking from dev because this branch does not build yet. Also, please keep the AI generated code at a minimum and verify that changes make sense and are actually needed.

manuel-rw · 2025-12-27T23:21:04Z

Dockerfile

 COPY . .

-RUN corepack enable pnpm && pnpm install --recursive --frozen-lockfile
+RUN corepack enable pnpm && pnpm install --recursive --no-frozen-lockfile


manuel-rw · 2025-12-27T23:21:35Z

Dockerfile

 ARG CI='true'
 ARG DISABLE_REDIS_LOGS='true'

+# Create database directory for build-time database access


Comment is wrong, why touch an empty file? Remove?

manuel-rw · 2025-12-27T23:21:54Z

Dockerfile

-COPY --from=builder /app/apps/tasks/tasks.cjs ./apps/tasks/tasks.cjs
-COPY --from=builder /app/apps/websocket/wssServer.cjs ./apps/websocket/wssServer.cjs
+# Tasks worker and WebSocket are now merged into Next.js server, so no separate builds needed
+# COPY --from=builder /app/apps/tasks/tasks.cjs ./apps/tasks/tasks.cjs


Remove comments

manuel-rw · 2025-12-27T23:22:15Z

apps/nextjs/next.config.ts

  experimental: {
    optimizePackageImports: ["@mantine/core", "@mantine/hooks", "@tabler/icons-react"],
    turbopackFileSystemCacheForDev: true,
+    // Reduce memory usage by limiting concurrent requests


Comments are not very useful, remove

manuel-rw · 2025-12-27T23:22:28Z

apps/nextjs/next.config.ts

      },
    ],
+    minimumCacheTTL: 60,
+    dangerouslyAllowSVG: true,


manuel-rw · 2025-12-27T23:27:31Z

packages/cron-job-api/src/client.ts


 function getBaseUrl() {
-  return `http://localhost:${CRON_JOB_API_PORT}`;
+  // Tasks API is now merged into Next.js, so use the same port


Not useful comment, remove

manuel-rw · 2025-12-27T23:28:38Z

scripts/run.sh

+# Node.js memory optimization flags:
+# --optimize-for-size: Optimizes for smaller memory footprint (trades some performance for memory)
+# --max-old-space-size: Limit heap size to prevent memory bloat (set to reasonable limit)
+# --expose-gc: Expose garbage collection API (allows manual GC if needed)


Is it a good Idea to expost this in production?

manuel-rw · 2025-12-27T23:29:15Z

workspace/MIGRATION_SUMMARY.md

@@ -0,0 +1,347 @@
+# Migration Summary: libsql-js Migration & Performance Optimizations


Remove this file entirely and move documentation if necessary to other places. It doesn't add any value and is just a changelog.

manuel-rw · 2025-12-27T23:29:57Z

workspace/Makefile

@@ -0,0 +1,69 @@
+.PHONY: help build start stop restart rebuild logs logs-homarr logs-redis status health shell clean clean-all


Why do we need such scripts?
They have nothing to do with the actual memory change.
And they are in platform agnostic languages. Why not use an independent one, such as Python for this?

manuel-rw · 2025-12-27T23:31:10Z

workspace/docker-compose.test.yml

@@ -0,0 +1,40 @@
+version: '3.8'


File not needed, delete

manuel-rw · 2025-12-28T11:38:18Z

packages/auth/configuration.ts

+            expires: expires.getTime(), // Convert Date to milliseconds timestamp
+            userId: user.id,
+          });
+          logger.info(`Session created successfully for user ${user.id}`);


Remove or reduce log level

manuel-rw · 2025-12-28T11:38:28Z

packages/auth/configuration.ts

+          // The schema defines expires as int({ mode: "timestamp_ms" })
+          await adapter.createSession({
+            sessionToken,
+            expires: expires.getTime(), // Convert Date to milliseconds timestamp


Doesn't compile, vibe coded?

manuel-rw · 2025-12-28T11:38:56Z

packages/cli/src/commands/reset-password.ts

+    if (options.password) {
+      console.log(`Password for user ${options.username} has been set to the provided value`);
+    } else {
+      console.log(`New password for user ${options.username}: ${newPassword}`);
+    }


Not relevant, revert

manuel-rw · 2025-12-28T11:48:08Z

@Meierschlumpf I think we should split this into the following changes / actions:

We keep this PR as-is but cherry-pick the relevant changes to N-PRs.
We cherry-pick the following changes:
- Cherry-pick the changes in apps/nextjs/next.config.ts and research whether they are production safe. For example, dangerouslyAllowSVG is likely not safe and doesn't have any effect on memory and needs to be reverted.
- Check whether the logging, e.g. in apps/websocket/src/main.ts has any effect at all on memory consumption and consider removing it (or conditionally logging).
- Check whether the flags in scripts/run.sh are safe and should be set.
- Check whether the flags in packages/redis/redis.conf are safe and make sense. What drawbacks to they have?
- Biggest change; migrate from better-sqlite to libsql. Keep change as little as possible.
- Migrate from the 3-app approach to the custom next.js server.
We discard all other changes in this PR, specifically the workspace/* and the password changes as they are not relevant.
We prepare a memory-observation command (CLI) to check memory consumption with the docker container and test the changes using the docker build for every iterative PR. This lets us know how much memory a certain change reduces.
We merge our changes to dev and ask users in High Memory Usage #3759 to test / confirm.

I believe we should do these changes incrementally and step by step;

it reduces the risk of breaking things
we can let users test early
we reduce our workload and go step by step with smaller changes

What do you think?

AartSchinkel · 2025-12-29T04:56:20Z

@manuel-rw, @Meierschlumpf,

With libsql I couldnt get authentication to work so you will see in my version that the better-sqlite is only used for authentication which I was still working on. Further I threw all code through Cursor to bug check and make comments on it sothat you could what was changed, thats were the workspace folder was made. Just check that libsql supports authentication on the webpage or otherwise we need to rewrite that piece to make it work with libsql instead of better-sqlite.

manuel-rw linked an issue Dec 23, 2025 that may be closed by this pull request

High Memory Usage #3759

Open

manuel-rw self-assigned this Dec 23, 2025

manuel-rw added the enhancement New feature or request label Dec 23, 2025

github-code-quality bot found potential problems Dec 23, 2025

View reviewed changes

packages/auth/configuration.ts Fixed Show fixed Hide fixed

packages/auth/configuration.ts Fixed Show fixed Hide fixed

feat: reduce memory usage

adc40bb

Co-authored-by: AartSchinkel <aartschinkel@iservecloud.com> Co-authored-by: Meier Lukas <meierschlumpf@gmail.com>

manuel-rw force-pushed the feature/reduce-memory-usage branch from 28c7159 to adc40bb Compare December 23, 2025 21:19

manuel-rw requested a review from Meierschlumpf December 23, 2025 21:20

manuel-rw mentioned this pull request Dec 23, 2025

High Memory Usage #3759

Open

manuel-rw assigned Meierschlumpf Dec 23, 2025

manuel-rw commented Dec 27, 2025

View reviewed changes

fix: remove unused imports

345ac1e

manuel-rw commented Dec 28, 2025

View reviewed changes

		@@ -0,0 +1,347 @@
		# Migration Summary: libsql-js Migration & Performance Optimizations

		@@ -0,0 +1,69 @@
		.PHONY: help build start stop restart rebuild logs logs-homarr logs-redis status health shell clean clean-all

Uh oh!

Conversation

manuel-rw commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Homarr

TODO:

Uh oh!

Uh oh!

Uh oh!

deepsource-io bot commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Analysis Summary

Uh oh!

azy2k9 commented Dec 24, 2025

Uh oh!

azy2k9 commented Dec 24, 2025

Uh oh!

azy2k9 commented Dec 24, 2025

Uh oh!

manuel-rw commented Dec 24, 2025

Uh oh!

azy2k9 commented Dec 24, 2025

Uh oh!

manuel-rw commented Dec 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manuel-rw commented Dec 28, 2025

Uh oh!

AartSchinkel commented Dec 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

manuel-rw commented Dec 23, 2025 •

edited

Loading

deepsource-io bot commented Dec 23, 2025 •

edited

Loading