Projects reserved state is moved to ProjectStateRegistry #132332

alexey-ivanov-es · 2025-08-01T14:02:14Z

This commit refactors how project reserved state is stored and accessed in Elasticsearch. Instead of storing reserved state metadata directly within each ProjectMetadata object, the commit moves it to a centralized ProjectStateRegistry.

Note: In mixed-version multiproject clusters, this may cause existing reserved state metadata for projects to temporarily disappear until all nodes have been upgraded and restarted.

It also maintains the logic for resetting reserved state metadata during snapshot restore, ensuring that the reserved state will be reread and re-applied to the cluster state.

elasticsearchmachine · 2025-08-01T21:13:15Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

ywangd

I have some initial comments.

Also, ProjectSecrets and other entities that can be created via file-based settings need to be initially created and stored in ProjectStateRegistry as well. I think it's reasonable to have them as separate PRs. If that is the case, do we have a JIRA issue for it? Thanks!

...er/src/main/java/org/elasticsearch/reservedstate/service/ReservedProjectStateUpdateTask.java

server/src/main/java/org/elasticsearch/cluster/metadata/ProjectMetadata.java

server/src/main/java/org/elasticsearch/cluster/ClusterState.java

ywangd

Still need to read through the test changes. Meanwhile I have some minor comments for the production code.

I like the changes! It would be great if someone from the core-infra team can review this as well. In addition, do you mind sending a message to the MP channel to announce this change before merging so that CP is aware any potential implication by us dropping the old-style project reserved metadata in wire communication? Thanks!

server/src/main/java/org/elasticsearch/cluster/ClusterState.java

server/src/main/java/org/elasticsearch/cluster/metadata/ProjectMetadata.java

server/src/main/java/org/elasticsearch/cluster/project/ProjectStateRegistry.java

server/src/main/java/org/elasticsearch/cluster/DiffableUtils.java

ywangd · 2025-08-11T08:30:01Z

server/src/main/java/org/elasticsearch/reservedstate/service/FileSettingsService.java

+    public void handleSnapshotRestore(
+        ClusterState clusterState,
+        ClusterState.Builder builder,
+        Metadata.Builder mdBuilder,
+        ProjectId projectId
+    ) {


The parameters are getting a bit out of hand. They are all here to just support resetting the version of file settings. We can potentially extract an interface to represent them and pass different implementation based on whether the setup is multi-project. It does not need to be this PR. Could you please add a @FixForMultiProject annotation to it and we can work on that later.

Could you please address this comment? Thanks!

ywangd

I have a question about whether we want to keep a consistent reserved_state output in cluster state for single project.

ywangd · 2025-08-13T07:25:06Z

server/src/test/java/org/elasticsearch/cluster/ClusterStateTests.java

+    public void testSerialization() throws IOException {
+        ClusterState clusterState = buildClusterState();
+        BytesStreamOutput out = new BytesStreamOutput();
+        clusterState.writeTo(out);
+
+        // check it deserializes ok
+        NamedWriteableRegistry namedWriteableRegistry = new NamedWriteableRegistry(ClusterModule.getNamedWriteables());
+        ClusterState deserialisedClusterState = ClusterState.readFrom(
+            new NamedWriteableAwareStreamInput(out.bytes().streamInput(), namedWriteableRegistry),
+            null
+        );
+
+        // check it matches the original object
+        Metadata deserializedMetadata = deserialisedClusterState.metadata();
+        assertThat(deserializedMetadata.projects(), aMapWithSize(1));
+        assertThat(deserializedMetadata.projects(), hasKey(ProjectId.DEFAULT));
+
+        assertThat(deserializedMetadata.getProject(ProjectId.DEFAULT).templates(), hasKey("template"));
+        assertThat(deserializedMetadata.getProject(ProjectId.DEFAULT).indices(), hasKey("index"));
+    }


I am ok with having the new test. But I am curious on why it is needed here? Does not seem relate to the production code change?

It would be great if you could address this comment as well. Thanks!

But I am curious on why it is needed here? Does not seem relate to the production code change?

In an earlier version I was making changes to cluster state serialization and added this test (along with another serialization test). I reverted those changes and the other test, but decided to keep this one since I think it's good to have a unit test for basic cluster state serialization/deserialization

ywangd · 2025-08-13T08:35:02Z

server/src/main/java/org/elasticsearch/cluster/metadata/Metadata.java

-        // need to combine reserved state together into a single block so we don't get duplicate keys
-        // and not include it in the project xcontent output (through the lack of multi-project params)
-        // use a tree map so the order is deterministic
-        final Map<String, ReservedStateMetadata> clusterReservedState = new TreeMap<>(reservedStateMetadata);
-        clusterReservedState.putAll(project.reservedStateMetadata());


The intention of this code is keep the output from the GetClusterState API identical in non-MP and MP (when the API is invoked via a project URL, i.e. single project context). Removing it without any replace means the responses will now differ, i.e.

// non-MP { "metadata": { "reserved_state": { ... } } }

vs

// MP when a single project is requested { "metadata": { ... }, "projects_registry": { "projects": [ { "reserved_state": { ... } } ] } }

Maybe it's OK for them to be different. But we need to make an explicit decision here.

Also, there is an existing issue that we don't filter ProjectStateRegistry for the requested project in TransportClusterStateAction so that it currently shows all projects when only a single project is requested. (EDIT: This is not an unique issue for ProjectStateRegistry. For example, blocks also have the same issue. So no need to fix it in this PR. The other point about different output formats between stateful and MP still stands).

On a second thought, the difference in ClusterState API output format between non-MP and MP setups can also be addressed in future PRs. There should not be any BWC concern to change the MP output in future. So nothing is needed in this PR other than maybe raise a JIRA issue so that we don't forget about it.

Thank you for highlighting this. I'll address the project filtering issue in a separate PR since it's straightforward, and I don't want to leave it for later.
Regarding the reserved state looking the same as before for a single project: we still have Metadata-level reserved state for cluster-level reserved state, and for all existing clusters it will still contain all the reserved state

I'll address the project filtering issue in a separate PR

👍

Regarding the reserved state looking the same as before for a single project: we still have Metadata-level reserved state for cluster-level reserved state, and for all existing clusters it will still contain all the reserved state

Yes it is not an issue for existing clusters (non-MP). My question is about MP cluster. We have a general principle that the API output for a single project should be the same regardless whether the cluster is non-MP or MP. For the GetClusterState API, non-MP already sets the convention that reserved state is printed under Metadata. So my question is whether a MP cluster should always do the same as oppose to print them under cluster level customs. Note we already intentionally hide the whole concept of ProjectMetadata in the output from a MP cluster with the intention to keep it consistent with non-MP output.

PR for the filtering issue: #133401

Yes it is not an issue for existing clusters (non-MP). My question is about MP cluster. We have a general principle that the API output for a single project should be the same regardless whether the cluster is non-MP or MP. For the GetClusterState API, non-MP already sets the convention that reserved state is printed under Metadata. So my question is whether a MP cluster should always do the same as oppose to print them under cluster level customs. Note we already intentionally hide the whole concept of ProjectMetadata in the output from a MP cluster with the intention to keep it consistent with non-MP output.

I see now, thanks for the clarification. I'm a bit concerned we're going to make a mistake in one of these many branches eventually, but I'll support the convention. However, I'd prefer to handle it at the transport action level. I think that's cleaner, and I believe we don't want to store project's reserved state metadata to disk

I'll do it in a separate PR, because turned out that I need changes from both this PR and #133401 😕

I'd prefer to handle it at the transport action level

That might be an option as well. I don't have strong opinion.

I'll do it in a separate PR

Yeah that's what I meant. No need for this PR. Since it affects only MP cluster, there is no real BWC either.

alexey-ivanov-es · 2025-08-21T18:07:21Z

Also, ProjectSecrets and other entities that can be created via file-based settings need to be initially created and stored in ProjectStateRegistry as well. I think it's reasonable to have them as separate PRs. If that is the case, do we have a JIRA issue for it? Thanks!

ProjectSecrets - yes, I don't fully understand the workflow behind why we need to move other fields. However, in this case I would prefer to have just the ProjectMetadata object for project in ProjectStateRegistry and then merge it with the ProjectMetadata in Metadata, rather than moving individual fields

ywangd · 2025-08-22T04:46:27Z

in this case I would prefer to have just the ProjectMetadata object for project in ProjectStateRegistry and then merge it with the ProjectMetadata in Metadata, rather than moving individual fields

Sorry I don't quite follow this comment. Could you please elaborate a bit? ProjectSecrets needs to be initially created inside ProjectStateRegistry since there is no ProjectMetadata at the time. Creating the project (i.e. ProjectMetadata) requires the secrets to be in place first since the process needs to read the object store.

…r' into ES-12183

ywangd

LGTM pending green CI

Thanks a lot for the work. I'd appreciate if you could address the other comments that I had. I am also putting down a couple suggestions again here to keep them in one place for easy consumption.

We need a ticket to address ProjectSecrets
It would be great to have someone from core-infra to take a look as well
It would be great to send a message in the MP channel before merging since it may destabilze existing MP clusters because new nodes will drop reserved states from old nodes. I suspect it should self-heal but a heads-up message in the channel is helpful in any case.

ywangd · 2025-08-25T01:54:15Z

server/src/main/java/org/elasticsearch/cluster/metadata/ProjectMetadata.java

                    }
                } else if (token == XContentParser.Token.START_OBJECT) {
                    switch (currentFieldName) {
+                        // Remove this


Nit: It would be better to raise a JIRA issue and link the number here to provide more context.

ywangd · 2025-08-25T01:54:22Z

server/src/main/java/org/elasticsearch/cluster/metadata/ProjectMetadata.java

                                projectBuilder.put(IndexTemplateMetadata.Builder.fromXContent(parser, parser.currentName()));
                            }
                        }
+                        // Remove this


Similar nit here.

server/src/main/java/org/elasticsearch/cluster/project/ProjectStateRegistry.java

ywangd · 2025-08-25T02:19:20Z

server/src/main/java/org/elasticsearch/reservedstate/service/FileSettingsService.java

+    public void handleSnapshotRestore(
+        ClusterState clusterState,
+        ClusterState.Builder builder,
+        Metadata.Builder mdBuilder,
+        ProjectId projectId
+    ) {


Could you please address this comment? Thanks!

ywangd · 2025-08-25T02:19:43Z

server/src/test/java/org/elasticsearch/cluster/ClusterStateTests.java

+    public void testSerialization() throws IOException {
+        ClusterState clusterState = buildClusterState();
+        BytesStreamOutput out = new BytesStreamOutput();
+        clusterState.writeTo(out);
+
+        // check it deserializes ok
+        NamedWriteableRegistry namedWriteableRegistry = new NamedWriteableRegistry(ClusterModule.getNamedWriteables());
+        ClusterState deserialisedClusterState = ClusterState.readFrom(
+            new NamedWriteableAwareStreamInput(out.bytes().streamInput(), namedWriteableRegistry),
+            null
+        );
+
+        // check it matches the original object
+        Metadata deserializedMetadata = deserialisedClusterState.metadata();
+        assertThat(deserializedMetadata.projects(), aMapWithSize(1));
+        assertThat(deserializedMetadata.projects(), hasKey(ProjectId.DEFAULT));
+
+        assertThat(deserializedMetadata.getProject(ProjectId.DEFAULT).templates(), hasKey("template"));
+        assertThat(deserializedMetadata.getProject(ProjectId.DEFAULT).indices(), hasKey("index"));
+    }


It would be great if you could address this comment as well. Thanks!

mark-vieira · 2025-08-26T17:19:11Z

...earch/action/admin/indices/template/delete/TransportDeleteComposableIndexTemplateAction.java


        validateForReservedState(
-            projectResolver.getProjectMetadata(state).reservedStateMetadata().values(),
+            ProjectStateRegistry.get(state).reservedStateMetadata(projectResolver.getProjectId()).values(),


Should we add a convenience method to get reservedStateMetadata to ProjectState as we've done for settings so this could just be projectResolver.getProjectState(state).reservedStateMetadata()?

I am not strongly oppose it, but I don't think reserved state metadata is needed in a project state - for example, Settings can be used by many components, but there are not so many things that are interested in the reserved state

alexey-ivanov-es · 2025-08-28T13:58:14Z

Ticket to move ProjectSecrets: ES-12793

) This commit refactors how project reserved state is stored and accessed in Elasticsearch. Instead of storing reserved state metadata directly within each ProjectMetadata object, the commit moves it to a centralized ProjectStateRegistry. Note: In mixed-version multiproject clusters, this may cause existing reserved state metadata for projects to temporarily disappear until all nodes have been upgraded and restarted. It also maintains the logic for resetting reserved state metadata during snapshot restore, ensuring that the reserved state will be reread and re-applied to the cluster state.

…3936) As it was noted in #132332 we have a general principle that API output for a single project should be the same regardless of whether the cluster is non-MP or MP. (However, the reserved metadata merge logic was never properly implemented to support this) As a result of changes in that PR, response formats diverged. This PR makes the single project GetClusterState API response for MP clusters match the non-MP format.

alexey-ivanov-es added 2 commits August 1, 2025 14:44

Projects reserved state is moved to ProjectStateRegistry

df3d4d6

Change the order of fields in a serialised version for consistency

d136933

alexey-ivanov-es added >non-issue :Core/Infra/Core Core issues without another label v9.2.0 labels Aug 1, 2025

alexey-ivanov-es added 2 commits August 1, 2025 15:04

Merge branch 'main' into ES-12183

97850c9

Merge branch 'main' into ES-12183

18d6a0c

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Aug 1, 2025

elasticsearchmachine and others added 6 commits August 1, 2025 14:17

[CI] Auto commit changes from spotless

6088f4a

Fix build

84b0c9e

Fix tests

a18c390

[CI] Auto commit changes from spotless

d96df29

Fix tests

9045018

Merge branch 'main' into ES-12183

7fd7234

alexey-ivanov-es marked this pull request as ready for review August 1, 2025 21:12

alexey-ivanov-es requested a review from ywangd August 1, 2025 21:13

elasticsearchmachine added the Team:Core/Infra Meta label for core/infra team label Aug 1, 2025

alexey-ivanov-es requested a review from a team August 1, 2025 21:15

elasticsearchmachine and others added 2 commits August 1, 2025 21:24

[CI] Auto commit changes from spotless

1a49726

Merge branch 'main' into ES-12183

79d943a

ywangd reviewed Aug 4, 2025

View reviewed changes

alexey-ivanov-es and others added 3 commits August 4, 2025 16:18

Address some review comments

4156e8f

[CI] Auto commit changes from spotless

e3911dd

Merge branch 'main' into ES-12183

ff8eb63

ywangd reviewed Aug 11, 2025

View reviewed changes

ywangd reviewed Aug 13, 2025

View reviewed changes

alexey-ivanov-es added 2 commits August 21, 2025 18:50

Address review comments

a869c06

Merge branch 'main' into ES-12183

1dcc91a

Merge branch 'main' into ES-12183

9198794

alexey-ivanov-es and others added 7 commits August 22, 2025 15:51

Fix projects state registry filtering in GetClusterState API

7c54ef3

Update docs/changelog/133401.yaml

28b46ce

Merge branch 'main' into fix/project_state_registry_filter

44ade8f

Delete docs/changelog/133401.yaml

f33aa53

[CI] Auto commit changes from spotless

8152173

Merge remote-tracking branch 'upstream/main' into ES-12183

e750926

Merge remote-tracking branch 'origin/fix/project_state_registry_filte…

8237c89

…r' into ES-12183

ywangd approved these changes Aug 25, 2025

View reviewed changes

mark-vieira approved these changes Aug 26, 2025

View reviewed changes

alexey-ivanov-es added 2 commits August 26, 2025 18:27

Merge branch 'main' into ES-12183

7bf5c23

Merge remote-tracking branch 'origin/ES-12183' into ES-12183

5c9b780

alexey-ivanov-es added 2 commits August 28, 2025 15:07

Address minor review comments

b473065

Merge branch 'main' into ES-12183

8ec5bdc

alexey-ivanov-es enabled auto-merge (squash) August 28, 2025 14:10

alexey-ivanov-es added 2 commits August 29, 2025 13:14

Merge branch 'main' into ES-12183

b227579

Merge remote-tracking branch 'upstream/main' into ES-12183

d181842

alexey-ivanov-es merged commit 61382f3 into elastic:main Aug 29, 2025
33 checks passed

alexey-ivanov-es deleted the ES-12183 branch August 29, 2025 13:48

alexey-ivanov-es mentioned this pull request Sep 1, 2025

Fix single project GetClusterState API response for multiproject #133936

Merged

Projects reserved state is moved to ProjectStateRegistry #132332

Projects reserved state is moved to ProjectStateRegistry #132332

Uh oh!

Conversation

alexey-ivanov-es commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 1, 2025

Uh oh!

ywangd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ywangd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ywangd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ywangd Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexey-ivanov-es commented Aug 21, 2025

Uh oh!

ywangd commented Aug 22, 2025

Uh oh!

ywangd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexey-ivanov-es commented Aug 28, 2025

Uh oh!

alexey-ivanov-es commented Aug 1, 2025 •

edited

Loading

ywangd Aug 13, 2025 •

edited

Loading