Skip to content

[improve][broker] Add dedicated partition configuration for system topics to reduce resource usage #23785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zjxxzjwang wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[improve][broker] Add dedicated partition configuration for system topics to reduce resource usage #23785

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
594ec35
[improve][broker]Optimize the partition size of the system theme to r…
Dec 26, 2024
d04fe96
[improve][broker]Optimize the partition size of the system theme to r…
Dec 26, 2024
586d1fc
[improve][broker]Optimize the partition size of the system theme to r…
Dec 31, 2024
5b52438
[improve][broker]Optimize the partition size of the system theme to r…
Jan 6, 2025
a1d41fa
[improve][broker]Optimize the partition size of the system theme to r…
Jan 7, 2025
3984650
fix checkstyle
dao-jun Feb 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions conf/broker.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,9 @@ allowAutoSubscriptionCreation=true
# The number of partitioned topics that is allowed to be automatically created if allowAutoTopicCreationType is partitioned.
defaultNumPartitions=1

# Default number of partitions for the system theme
systemTopicDefaultNumPartitions=1

# Enable the deletion of inactive topics. This parameter need to cooperate with the allowAutoTopicCreation parameter.
# If brokerDeleteInactiveTopicsEnabled is set to true, we should ensure that allowAutoTopicCreation is also set to true.
brokerDeleteInactiveTopicsEnabled=true
Expand Down
3 changes: 3 additions & 0 deletions conf/standalone.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1187,6 +1187,9 @@ allowAutoSubscriptionCreation=true
# The number of partitioned topics that is allowed to be automatically created if allowAutoTopicCreationType is partitioned.
defaultNumPartitions=1

# Default number of partitions for the system theme
systemTopicDefaultNumPartitions=1

### --- Transaction config variables --- ###
# Enable transaction coordinator in broker
transactionCoordinatorEnabled=false
Expand Down
6 changes: 6 additions & 0 deletions pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2167,6 +2167,12 @@ The max allowed delay for delayed delivery (in milliseconds). If the broker rece
+ " if allowAutoTopicCreationType is partitioned."
)
private int defaultNumPartitions = 1;
@FieldContext(
category = CATEGORY_STORAGE_ML,
dynamic = true,
doc = "Default number of partitions for the system theme."
)
private int systemTopicDefaultNumPartitions = 1;
@FieldContext(
category = CATEGORY_STORAGE_ML,
doc = "The class of the managed ledger storage"
Expand Down
4 changes: 3 additions & 1 deletion pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3291,12 +3291,14 @@ private CompletableFuture<PartitionedTopicMetadata> createDefaultPartitionedTopi
Optional<Policies> policies) {
final int defaultNumPartitions = pulsar.getBrokerService().getDefaultNumPartitions(topicName, policies);
final int maxPartitions = pulsar().getConfig().getMaxNumPartitionsPerPartitionedTopic();
final int systemTopicDefaultNumPartitions = pulsar.getConfiguration().getSystemTopicDefaultNumPartitions();
checkArgument(defaultNumPartitions > 0,
"Default number of partitions should be more than 0");
checkArgument(maxPartitions <= 0 || defaultNumPartitions <= maxPartitions,
"Number of partitions should be less than or equal to " + maxPartitions);

PartitionedTopicMetadata configMetadata = new PartitionedTopicMetadata(defaultNumPartitions);
PartitionedTopicMetadata configMetadata = new PartitionedTopicMetadata(
isSystemTopic(topicName) ? systemTopicDefaultNumPartitions : defaultNumPartitions);

return checkMaxTopicsPerNamespace(topicName, defaultNumPartitions)
.thenCompose(__ -> {
Expand Down
Loading