Feature: Multi-Project #123333
Closed
Feature: Multi-Project #123333
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Changes `DataStreamAutoShardingService` to take a ProjectState instead of a ClusterState.
Changes `WriteLoadForecaster` and sublcasses to take a `ProjectMetadata` builder instead of a `Metadata` builder.
This changes the `MultipleProjectsClientYamlSuiteTestCase` to verify that every project other than the active project is "empty" at the end of the test. "empty" means that it has no indices, and only has the templates, pipelines and ILM polices that are installed by default
Merge main 2024-11-29T09
Introduces `MultiProjectRestTestCase` that provides support for writing rest tests against multiple projects
Merge main 2024-11-30T10
Modifies the PUT and GET data stream APIs to be project-aware.
Merge main 2024-12-04T16
As title says. Cluster state recover can happen when master node restart or during an ungraceful fail-over.
Updates the force merge transport action to support multiple projects.
Instead of using an empty global routing table, this PR uses empty project routing table to ensure the project actual has a routing table (albeit empty) when routingTable(projectId) is called.
Merge branch main into multi-project
…ultiProject-prevalidateRemoval
Updates the service to process data streams in all projects.
On deserialization, state is split into project and cluster based on the namespace, which is hardcoded in `Metadata.Builder.MOVED_PROJECT_RESERVED_STATE`
This PR replaces elastic/elasticsearch-internal#1934 and extracts some features from elastic/elasticsearch-serverless#3422. Instead of allowing for projects to be created without a secrets file, this PR forces projects to also have a secrets file before being created. - This PR doesn't actually handle the secrets data (defines a handler for reserved project state), that will be implemented in a separate PR. - The multi-project file handling should move to the `elasticsearch-serverless` repo before being merged to main (separate PR). - This PR contains a bug fix for `ProjectMetadata` where the project reserved state wasn't always applied to project metadata. - Since the secrets file name can clash with a project file if named `project-<project_id>-secrets.json` (project id could be `-secrets`) the naming of the file has been updated to `project-<project_id>.secrets.json` (`.` is not allowed in project id) - This PR adds a separate namespace for secrets, since sharing the `file_settings` namespace with the `project-<project_id>.json` makes assumptions on the order of the increasing version number. If they share namespace, the order would have to be first increased in the project config and then in the secrets config for them to be processed (since they're separate state chunks), which is a strange assumption. One of them also wouldn't be processed if they were on the same version.
The multi-project module is actually a testing module, but was included in the modules/ tree for convenience. This commit moves it to `test/external-modules/` which is where it belongs.
This changes the `multi_project.enabled` setting to `test.multi_project.enabled`
An old node writes all tasks in the metadata custom. A new old must be able to read it and separate the cluster-scoped and project-scoped tasks and store them in the right place. This PR does that. Relates: #1945, MP-1938
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.