RFC: Nx Custom Self-Hosted Remote Cache RFC

[vsavkin]

Due to our ongoing rewrite of Nx core from TypeScript to Rust, we are removing the ability to load custom code during task execution. We want this part to be heavily optimized with strong guarantees. To support users who need self-hosted cache solutions, we've created several free first-party packages for all major cloud providers (S3, MinIO, GCP, Azure). The packages are more secure and faster than existing self-hosted solutions. Moving from existing self-hosted cache solutions to the new packages should be straightforward.

We recognize that some organizations require more flexibility. Since we can no longer load custom code, we're offering this flexibility in a different way. Similar to other build systems, we're adding an HTTP-based API that allows you to connect your Nx workspace to a service offering self-hosted cache. This RFC provides an OpenAPI spec and some notes on how the cache will be activated.

We will be implementing with this spec for Nx 21 which will be released soon.

Open API Spec

{
  "openapi": "3.0.0",
  "info": {
    "title": "Nx custom remote cache specification.",
    "description": "Nx is a build system, optimized for monorepos, with AI-powered architectural awareness and advanced CI capabilities.",
    "version": "1.0.0"
  },
  "paths": {
    "/v1/cache/{hash}": {
      "put": {
        "description": "Upload a task output",
        "operationId": "put",
        "security": [
          {
            "bearerToken": []
          }
        ],
        "responses": {
          "202": {
            "description": "Successfully uploaded the output"
          },
          "401": {
            "description": "Missing or invalid authentication token.",
            "content": {
              "text/plain": {
                "schema": {
                  "type": "string",
                  "description": "Error message provided to the Nx CLI user"
                }
              }
            }
          },
          "403": {
            "description": "Access forbidden. Silently fails. (e.g. read-only token used to write)"
          },
          "409": {
            "description": "Cannot override an existing record. Silently fails."
          }
        },
        "parameters": [
          {
            "in": "header",
            "description": "The file size in bytes",
            "required": true,
            "schema": {
              "type": "number"
            },
            "name": "Content-Length"
          },
          {
            "name": "hash",
            "description": "The task hash corresponding to the uploaded task output",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "requestBody": {
          "content": {
            "application/octet-stream": {
              "schema": {
                "type": "string",
                "format": "binary"
              }
            }
          }
        }
      },
      "get": {
        "description": "Download a task output",
        "operationId": "get",
        "security": [
          {
            "bearerToken": []
          }
        ],
        "responses": {
          "200": {
            "description": "Successfully retrieved cache artifact",
            "content": {
              "application/octet-stream": {
                "schema": {
                  "type": "string",
                  "format": "binary",
                  "description": "An octet stream with the content."
                }
              }
            }
          },
          "403": {
            "description": "Access forbidden",
            "content": {
              "text/plain": {
                "schema": {
                  "type": "string",
                  "description": "Error message provided to the Nx CLI user"
                }
              }
            }
          },
          "404": {
            "description": "The record was not found"
          }
        },
        "parameters": [
          {
            "name": "hash",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ]
      }
    }
  },
  "components": {
    "securitySchemes": {
      "bearerToken": {
        "type": "http",
        "description": "Auth mechanism",
        "scheme": "bearer"
      }
    }
  }
}

Activating Remote Cache

Set the following environment variables:

• NX_SELF_HOSTED_REMOTE_CACHE_SERVER (e.g., https://my-custom-cache.mydomain.com/)
• NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN

Nx will include the access token as part of the payload.

[Jordan-Hall]

Thank you for or this detailed rfc.

Curiosity, your first class packages, will their now be staying put?
Will the first class packages be migrated to rust?
Is it possible to also have plugin for thr rust so have support outside of deploying custom servers? (Happy with server approach as someone that self host everything anyway )

[Tom910]

Thanks for the proposal. This will help to solve the most cases.

Also previously NX cache API had an interface like getCache(...cacheHashes) which return {[cacheHash] {get: s3 presignURL, set: s3 presignURL} which NX uses to read/write cache. This interface was good because allow to avoid a lot of traffic from API and moved it directly to S3. If it possible to implement it will be the best solution

[IKatsuba]

I completely agree with @Tom910. This would be very close to the implementation of the previous API.

[IKatsuba]

Check out the draft project for custom self-hosted server IKatsuba/nx-cache-server with S3 as storage

[RobbyRabbitman]

Hey, cool stuff! I wonder if nx can provide server stubs e.g. for node in addition to the openapi spec - of course there are many tools that can generate server stubs from an openapi spec but you still have to add the actual implementation (which is exactly wanted and nice!). I wonder if these implementations will differ or share repetitive logic. If so, you could then pass the actual cache to that http handler - a interface might look like this:

export interface NxCache {
  /** Retrieves the data for the given hash. */
  get: (hash: string) => Promise<Buffer>;

  /** Whether the cache contains data for the given hash. */
  has: (hash: string) => Promise<boolean>;

  /** Sets the data for the given hash. */
  set: (hash: string, data: Buffer) => Promise<void>;
}

with usage:

class MyCache implements NxCache {
// ...
}

const server =
  createServer(
    nxHttpCacheHandler(
      new MyCache(),
      {
        readAccessToken:
          'my-read-access-token',
        writeAccessToken:
          'my-write-access-token',
      },
    ),
  );

server.listen(
  3000,
);

i have written such a http handler that connects a custom cache solution - what do you think? :D

[IKatsuba]

I think it is cool idea!

[IKatsuba]

In the previous version token and url were declared in nx.json. It was useful because developers just cloned repo and started working. Now they will have to configure remote cache firstly? Will we able to use nx.json for it?

[eliellis]

Nx does support implicitly loading environment variables from *.env files within the workspace, which could be committed to your VCS to achieve the same effect. However, it's worth pointing out that a non-trivial portion of users will likely have these files (or some variant of these files) .gitignore'd given that oftentimes .env files will contain secrets, because folks typically would not want to fat-finger that information into their git tree.

Supporting the configuration of (at least) the cache destination / source URL in nx.json would be nice.

Getting a little creative though, one could use the new preTasksExecution hook to set these environment variables from some file (or any arbitrary source). So–if it is the case the Nx team determines that nx.json isn't a suitable place for persisting this configuration–you could conceivably cook up your own committed configuration file (e.g. nx-cache.config.json) and just read from that file and set the appropriate environment variables in a self-implemented preTasksExecution hook. This could serve as a good alternative to providing support for this configuration directly in nx.json if the Nx team wants to avoid that additional surface area.

[HaasStefan]

Will that be a remote cache solution just for CI, or local + CI?

If it is local + CI, I think it would be important to make push/pull based on the environment, because I would not want a local environment to push to the cache, only pull to avoid any possibility of cache poisoning.

[eliellis]

You could achieve this behavior by doling-out role associated (read-only or read-write) bearer tokens for usage in particular environments, and have your API respond with a 403 based on the operation (PUT or GET) and the type of token its been handed by Nx (read-only or read-write), correct? In terms of bare-minimum functionality, I think the RFC as-is should satisfy those requirements.

In terms of feedback on this topic relevant to the RFC itself:

Perhaps a response of 403 from a self-hosted remote cache could be swallowed by Nx in the general case, and would only surface in nx command output when said command was run using the existing --verbose flag to avoid unnecessary noise in nx command output in the case that a PUT operation failed with a 403 because the provided token did not have the necessary permissions to perform the operation (for example).

Reading the OpenAPI spec in the RFC, a 403 is already an expected response type for both PUT and GET HTTP methods. The question is necessarily how will Nx inform the user of this status code in the general case? Something similar to how @nx/s3-cache manages this with ("localMode" and "ciMode") might be a good way to configure the stifling of these errors. Perhaps NX_SELF_HOSTED_REMOTE_CACHE_LOCAL_MODE and NX_SELF_HOSTED_REMOTE_CACHE_CI_MODE environment variables?

[FrozenPandaz]

Perhaps a response of 403 from a self-hosted remote cache could be swallowed by Nx in the general case, and would only surface in nx command output when said command was run using the existing --verbose flag to avoid unnecessary noise in nx command output in the case that a PUT operation failed with a 403 because the provided token did not have the necessary permissions to perform the operation (for example).

Great point, we'll keep this in mind for our implementation.

[FrozenPandaz]

Reading the OpenAPI spec in the RFC, a 403 is already an expected response type for both PUT and GET HTTP methods. The question is necessarily how will Nx inform the user of this status code in the general case? Something similar to how @nx/s3-cache manages this with ("localMode" and "ciMode") might be a good way to configure the stifling of these errors. Perhaps NX_SELF_HOSTED_REMOTE_CACHE_LOCAL_MODE and NX_SELF_HOSTED_REMOTE_CACHE_CI_MODE environment variables?

Let me think more about if this is necessary and I'll get back to you about it.

[AgentEnder]

Ok, so there are two types of "auth" errors in the implementation: 401 and 403. 401 is an error that gets reported to the user, 403 is an expected response from the server that gets swallowed by Nx representing an access token thats valid but not allowed to perform an action.

[eliellis]

This is great. Thank you so much @vsavkin and team for allowing this to happen. I have two somewhat minor technical questions / callouts.

1. For self-hosted remote caches employing TLS, assuming the presented certificate was self-signed (e.g. I am my own CA), would it be possible to either provide additional CAs to the HTTP client making the requests, disable CA checking altogether, or possibly support both approaches? (Thinking something similar to the functionality offered by Node in the form of the NODE_EXTRA_CA_CERTS, and NODE_TLS_REJECT_UNAUTHORIZED environment variables.)
2. On the subject of proxy support, would it be possible to ensure that the HTTP client making these requests respect the common http_proxy, https_proxy, HTTP_PROXY, HTTPS_PROXY, no_proxy, and NO_PROXY family of environment variables?

[FrozenPandaz]

For self-hosted remote caches employing TLS, assuming the presented certificate was self-signed (e.g. I am my own CA), would it be possible to either provide additional CAs to the HTTP client making the requests, disable CA checking altogether, or possibly support both approaches? (Thinking something similar to the functionality offered by Node in the form of the NODE_EXTRA_CA_CERTS, and NODE_TLS_REJECT_UNAUTHORIZED environment variables.)

We will look into this as part of our initial implementation.

[FrozenPandaz]

...then I believe you can achieve what you want by just building a small Express, Hono, etc. API that maps these API endpoints over the filesystem. Then in CI (or wherever), just ensure your little Nx-remote-cache-to-FS-API server is running when you need it, point Nx to localhost via the proposed NX_SELF_HOSTED_REMOTE_CACHE_SERVER environment variable, and you've got exactly what you need.

• @eliellis

As mentioned in the below thread, you can implement something akin to the shared fs cache via this API as well.

[Jordan-Hall]

Apparently its for analytic data but its free

[eliellis]

I think you are confusing this RFC with the Powerpack replacement packages. Both the "new" Powerpack replacement packages, and the use of a fully custom Remote Cache API implementation are free. The Powerpack replacement packages do require a license (but it is free), or alternatively, per the docs, fully custom Remote Cache API implementations do not require a license of any kind, and logically following from that, do not track end user analytics:

"build your own cache server"

That link takes you directly to this RFC.

Which is all to say, if you want to avoid the licensing involved with the (free for everyone) Powerpack replacement packages, you should refer to this RFC for how to implement and use your own remote cache API. To me, this setup is perfectly reasonable: either build your own if you want to fully opt-out of licensing and telemetry, or provision a free license and use the Powerpack replacement packages. I assume the usage telemetry for the Powerpack replacement packages will be used by Nx internally for allocating support, etc. It appears to be fully anonymized as well. Going the fully custom cache API route, based on this RFC, also seems trivial to implement for whatever your desired goals are if you want to avoid the licensing and telemetry altogether.

[garydex]

But the RFC only covers caches via API. The file system cache still requires an activation key but has no “build your own” alternative.

[eliellis]

But the RFC only covers caches via API. The file system cache still requires an activation key but has no “build your own” alternative.

Definitely not defending the annoyance factor and additional overhead involved with this, but for the specific criteria of:

• no license requirements
• no telemetry collection
• local file-system cache

...then I believe you can achieve what you want by just building a small Express, Hono, etc. API that maps these API endpoints over the filesystem. Then in CI (or wherever), just ensure your little Nx-remote-cache-to-FS-API server is running when you need it, point Nx to localhost via the proposed NX_SELF_HOSTED_REMOTE_CACHE_SERVER environment variable, and you've got exactly what you need.

To me, the "build your own" approach is really just an agreed-upon generalized interface for Nx to delegate remote caching to when the existing packages do no suit your needs, you don't feel comfortable with the licensing model, and or you need additional functionality. Mapping from the RFC's proposed HTTP interface onto any storage system, be it a local filesystem, S3, Azure, etc. is entirely up to you.

Practically, I think, too, that the interface/transport for this being OpenAPI/HTTP makes sense given where the whole story started: Nx core is being re-written in Rust, and HTTP is a widely used and widely supported means of bridging communication across various systems. Making an HTTP request from Rust is trivial, and implementing a conforming remote cache HTTP API (per this RFC) is equally as trivial. They could have exposed this capability through other interface + transport approaches, (e.g. a custom IPC API, Protocol Buffers + gRPC, etc.) but an OpenAPI spec / HTTP ticks all the boxes, is something most people using Nx will understand how to implement, and is totally satisfactory for this use case. Again, not saying this isn't "weird" for the local filesystem use-case, but it does make sense as far as the "anything goes, build your own" implementation on the Nx-side of things.

[FrozenPandaz]

Yes, exactly. Thank you @eliellis for answering these so promptly and well. I could not have said it any better myself.

The custom self-hosted remote cache is not behind any license. It is completely open and free to use as a core feature of the Nx itself.

Only the first party provided self hosted caches (AWS, GCP, Azure, Shared FS) remain under a license.

[Hommis]

This implementation seems reasonable. Thank you for listening the community.

[kategable]

I find it unusual to have a PUT endpoint without a corresponding POST. In REST, POST is usually for creating resources, while PUT is for updating or replacing them at a known URI.

Also, I noticed there's a hash used in the PUT—is the idea to keep updating the same resource under that hash? Just trying to understand the design choice here—maybe I'm missing something.

[eliellis]

I believe PUT is perfectly legitimate for the design here. According to MDN:

The PUT HTTP method creates a new resource or replaces a representation of the target resource with the request content

In my experience the PUT method is used when you want to signal support for "upsert" behavior, which I believe is the goal with this endpoint. You (the implementor of the custom cache endpoints) are left to decide if overwriting behavior is permitted or not, and, as written in the OpenAPI spec in OP, are to return a 409 in the case an overwrite operation is unacceptable at the time of storage.

I think the design here with PUT mostly just "leaves the door open" for overwriting if that is something you want your own Remote Cache API implementation to allow for.

[kategable]

Thank you for explaining.

[FrozenPandaz]

Yes absolutely, the GET response should also be an octet stream and Nx will expect to receive it as such.

I have updated the original post. Thank you for pointing that out.

[Ambroos]

First of all, thanks for the RFC and shifting approach from the initial Nx 20/21 cache plan. This is much appreciated and the RFC here would do what we need, albeit with a bit less flexibility than what we currently do.

With our current runner I gather extra information about the task to provide to the get and put equivalents. This extra metadata helps debug cache issues and inspect cache entries, but we also print some of it when fetching data from the cache to make it easier to see where cached data came from, and when it was made.

What would be nice would be to have a JSON object with more information in a header with the PUT, if possible. Similarly, an equivalent optional header with an info string that can be sent by the server on GET (and maybe also PUT) that Nx just prints when the response comes in

Useful data:

• Task ID: Having this allows us to save cache files in separate directories by task ID, and also do some more targeted cache clearing etc. Currently we just use Task.id from the devkit and it works great for us.
• Current commit hash, suffixed with -dirty if there are changes in the working directory
• OS and arch

As an example, a log line when using a remote cached download in CI looks like this for us right now: Downloaded cached run some-package:some-target-linux-x64/318788506985000932.tar.gz uploaded by GitHub Actions (2025-04-07 17:34:44) - based on commit 26ad1ca7a424d0cb4319dee8756f5deb91ec133f

[bahizi]

@Ambroos great suggestion for extra metadata. However, I would suggest that this gets sent as individual headers (e.g. X-NX-TASK-ID, X-NX-CACHE-COMMIT-ID, X-NX-CACHE-ARCH, etc) instead. For some reason json-encoded http headers feel like a 2nd http body in the same request lol

[Ambroos]

@bahizi Fair!

[vsavkin]

Folks,

This has been released. You can see the docs here: https://nx.dev/recipes/running-tasks/self-hosted-caching#build-your-own-caching-server

Thank you so much to everyone who contributed to this!

[bahizi]

@vsavkin while testing [email protected] against our custom cache api, noticed that nx throws Misconfigured remote cache endpoint: Unexpected response status: 202 Accepted, when the custom api returns 202 for the PUT /v1/cache/{hash} endpoint's response status code. 200 status code is working fine though.
Before I get into filing a bug report, this got me thinking: would 200 or 201 be a better status code than 202 for the successful PUT /v1/cache/{hash} method requests?

[or-orchid]

Same issue here, did you modify the route to return 200?

[mikeathers]

With the nx/s3-cache package it looks like you cannot override the bucket name via environment variables like other community based packages.

Not being able to override the bucketName means we cannot use different buckets for different CI environment runs.

This to me feels like something that the nx/s3-cache package should support out of the box.

Is this feature on the roadmap?

Thanks!

[RobbyRabbitman]

Will you specify a different license than Commercial in your first class npm packages like @nx/shared-fs-cache? For companies this may be too vague regarding their (open source) license policies, especially after you wanted to put custom caching behind a paywall initially. Currently the docs say:

The official self-hosted cache packages are the easiest migration path if you've been using a community caching solution based on the old custom task runner API in the past. All of the packages are completely free but require an activation key. Getting a key is a fully automated and self-serving process that happens during the package installation.

In any case a spdx identifier would be nice.

[ardabeyazoglu]

Using self-hosted cache is documented as "free for everyone" now, with an activation key for marketing purposes.
Reference: https://nx.dev/nx-api/s3-cache

However, it does not work. nx g @nx/s3-cache:init command throws an error like this:

"No Nx keys were found for this organization. Please visit https://cloud.nx.app/powerpack to purchase or renew your key".

So, how can we create a free activation key to cache on our own s3 bucket?

[ChrisSargent]

well, you don't have to add any card details so it must be free - I assume the activation key will still work but 🤷 . An extra line in the documentation would save a lot of time here :-)

[fgkramer]

What I got when looking at my NX dashboard is that I still have to pay for a Powerpack licence, in fact it's 1 license per developer that contribute to a workspace.

So even if we are aiming to get a "free for everyone cache", we still have to pay to use it in one way or another. I find this extremely misleading if not a complete lie!

[AgentEnder]

What I got when looking at my NX dashboard is that I still have to pay for a Powerpack licence, in fact it's 1 license per developer that contribute to a workspace.

So even if we are aiming to get a "free for everyone cache", we still have to pay to use it in one way or another. I find this extremely misleading if not a complete lie!

@fgkramer power pack is separate from the remote caching plugins at this point. We do still ask that you acquire a free key through the form (or by running nx add @nx/s3-cache) but that key is entirely free. There's no payment needed to use the remote caching plugins.

Power pack as a whole still exists, but it includes things other than the caching plugins such as conformance, ownership and such

[ianzone]

How about allowing people to build their own cache plugins? #31685

[AgentEnder]

Folks are free to do their own caches, via the http api's. There is no plan to add support for an official cache plugin api outside of this, because we need the flexibility as we move more stuff to rust and don't want to expand the api surface at this time to have more things that we have to figure out at that time.

I won't say we'll never do this, that's not my place to say and I don't think anyone is certain of that, but for now this is what we are doing while things get worked out down the line.

[charlesanim]

Question to NX folks, what does the activation key do? Is this used for tracking telemetry? What data is tracked or linked ? Are repository information going to be exposed to NX if I wanted to go with the @nx/azure-cache package for example for an enterprise repo?

[joe-passenger]

Thanks for keeping self-hosted caching!

I know I'm late to be providing feedback, but I believe my PUT/GET endpoints are going to require basic auth due to reasons out of my control. Perhaps I'm missing something, but I don't see a way to handle this today.

[Kamalesh-Seervi]

https://github.com/Kamalesh-Seervi/nx-custom-cache-server/tree/main

check this out custom cache server and there will more optimized fix released soon

if any bug please create issue or contribute