Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
e527acb
[Subtask]: Use a new configuration item to control whether master & s…
Oct 29, 2025
064f4ea
[Subtask]: Use a new configuration item to control whether master & s…
Oct 30, 2025
29338ac
[AMORO-3850] Fix openapi sdk build and refine the package name (#3847)
turboFei Oct 31, 2025
41ffdb4
[HotFix] Change the uri configuration name in terminal (#3844)
zhoujinsong Oct 31, 2025
2ea1066
[AMORO-3851] Separate spark-3.3 and spark-3.5 modules (#3848)
turboFei Oct 31, 2025
ab7c629
[AMORO-3857] Make the metadata file name conform to the Iceberg spec …
liaoyt Oct 31, 2025
7aa1fc8
[AMORO-3850][FOLLOWUP] Add GA for OpenAPI SDK (#3870)
turboFei Nov 2, 2025
703cbec
[AMORO-3864] Bump spark 3.5 version to 3.5.7 (#3860)
turboFei Nov 3, 2025
368f6bf
[AMORO-3852][BUILD] Enhance build system (#3849)
turboFei Nov 3, 2025
652973b
[AMORO-3851][FOLLOWUP] Using scala.binary.version for Spark dependenc…
turboFei Nov 3, 2025
ac7fd3f
[AMORO-3851][FOLLOWUP] Fix and simplify spark versions management (#3…
turboFei Nov 3, 2025
860df1a
[AMORO-3863] Add script to reformat code (#3862)
turboFei Nov 3, 2025
f45bc1c
[AMORO-3852] Using Maven Wrapper (#3879)
turboFei Nov 4, 2025
25bf16d
[AMORO-3880] Make amoro-openapi-sdk standalone (#3881)
turboFei Nov 4, 2025
49f47ec
[AMORO-3891] Bump netty version to 4.1.128.Final to fix CVE (#3892)
turboFei Nov 4, 2025
ea1ae84
[AMORO-3890]Bump flink versions from 1.16.2/1.17.1/1.20.0 to 1.16.3/1…
turboFei Nov 4, 2025
be28b79
[AMORO-3880][FOLLOWUP] Fix pull request paths for OpenAPI SDK CI (#3903)
turboFei Nov 6, 2025
2d2a2c2
Bump orc-core version from 1.8.3 to 1.9.7 (#3900)
xxubai Nov 6, 2025
452191f
[AMORO-3883] Bump Paimon 1.1.1 to 1.2.0 (#3884)
xuzifu666 Nov 6, 2025
6a022d6
[AMORO-3863][FOLLOWUP] Support to reformat pom (#3882)
turboFei Nov 6, 2025
f77da8c
[AMORO-3872] Support to customize basic authentication implementation…
turboFei Nov 6, 2025
9dbbdd2
[AMORO-3907] Bump kubernetes-client version to 6.13.5 (#3908)
turboFei Nov 6, 2025
6331a5a
[AMORO-3875] Support to build on Spark Scala-2.13 (#3878)
turboFei Nov 7, 2025
28b56bc
[AMORO-3875][FOLLOWUP] Fix optimizer-spark docker image build due to …
turboFei Nov 10, 2025
7a098f1
[AMORO-3863][FOLLOWUP] Enable format-mixed-format-trino profile on JD…
turboFei Nov 11, 2025
58e5b54
[AMORO-3848][FOLLOWUP] Prompt spark profile in docker build (#3904)
turboFei Nov 11, 2025
388ff78
[AMORO-3918][INFRA] Add dependencies check CI (#3917)
turboFei Nov 12, 2025
ae477d6
[AMORO-3931] Exclude curator and zookeeper deps (#3932)
turboFei Nov 14, 2025
c71ee68
[AMORO-3945][DOCS] Add docs for REST API (#3944)
turboFei Nov 15, 2025
d4f223f
[AMORO-3873] Support Bearer/JWT authentication (#3905)
turboFei Nov 17, 2025
2f79bbf
[AMORO-3938] change file_content_b64 column type to CLOB type (#3939)
simonsssu Nov 17, 2025
77c71d8
[AMORO-3933] Fix Playground demo failure due `PartitionExpressionForM…
turboFei Nov 17, 2025
7648b7b
Save the last completion time for each cleanup operation performed on…
zhangwl9 Nov 20, 2025
e5aafcf
nit: remove duplicate code (#3957)
turboFei Nov 21, 2025
9689e01
[AMORO-3873][FOLLOWUP] Rename rest auth bearer type to JWT (#3953)
turboFei Nov 21, 2025
e332d39
[AMORO-3968] Update the thrift api compile command to use amoro shade…
turboFei Nov 25, 2025
2c3ac0f
[AMORO-3961] Filter null key and value for Configurations::toMap (#3962)
turboFei Nov 25, 2025
04d418e
nit: remove unused code (#3959)
turboFei Nov 25, 2025
f59dbda
[AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 (#3975)
zhan7236 Nov 27, 2025
2d70c23
[AMORO-3966][Helm] Support custom volumes and volumeMounts (#3965)
AbhishekPathania Nov 27, 2025
9df1953
[AMORO-3970] Update README.md with correct Spark versions for Mixed f…
xykera Nov 27, 2025
5549357
[AMORO-3977] Combine amoro-site to prevent docs loose sync (#3979)
turboFei Nov 28, 2025
a24d4c7
[Improvement]: Disable verification for helm-unittest plugin installa…
tcodehuber Dec 4, 2025
7ebcd86
[AMORO_3990] Add package chart instructions to README (#3991)
tcodehuber Dec 4, 2025
1dd96dd
[Feature] Introduce a new framework that supports scheduling for Form…
LiangDai-Mars Dec 4, 2025
4192cb1
[AMORO-3981] Automatic generate amoro configuration docs (#3982)
turboFei Dec 6, 2025
887552e
[AMORO-3981][FOLLOWUP] Refine the duration to string format (#3987)
turboFei Dec 8, 2025
8c9a3f5
[AMORO-1258] Support Zookeeper kerberos authentication (#3980)
turboFei Dec 8, 2025
5bf7efd
[AMORO-3934] Manage com.fasterxml.jackson dependencies in dependencyM…
turboFei Dec 8, 2025
f222910
[AMORO-3866] Using shaded hadoop-client-api and hadoop-client-runtime…
turboFei Dec 9, 2025
00aaa87
[AMORO-3632] Refine data expiration literal calculation for date type…
davedwwang Dec 10, 2025
0a60f22
[AMORO-3885] fix missing oss.endpoint for internal mixed_iceberg cata…
liaoyt Dec 10, 2025
2d8081a
[AMORO-2635] Enhance table partition files list performance (#4003)
turboFei Dec 12, 2025
bae1605
[AMORO-3628] Add user logo wall on the home page. (#4010)
xxubai Dec 12, 2025
1076213
[AMORO-3804] Skip RUNTIME_CONTEXT_CACHE for TableMetaStore with local…
turboFei Dec 17, 2025
61d55e2
[AMORO-4011] Fix JUnit 4 tests skipped (#4017)
turboFei Dec 19, 2025
c666986
[AMORO-3775] Add support for metric-based refresh event trigger in Ta…
Jzjsnow Dec 22, 2025
02baa82
[AMORO-3998]support DATABASE HA SERVICE (#3997)
LiangDai-Mars Dec 22, 2025
6afa2e3
[Hotfix] Fix the deploy site GitHub workflow (#4020)
zhoujinsong Dec 23, 2025
4ec6437
[Hotfix] Enable manual triggering for site deployment workflow (#4021)
xxubai Dec 24, 2025
88b832d
[Hotfix]Refactor the build command in the `Publish Docker Image` GitH…
zhoujinsong Dec 29, 2025
5e6ab69
[Hotfix] Fix typos (#4026)
leosanqing Dec 29, 2025
27a60d7
[AMORO-3531] Drop support for java8 (#3899)
turboFei Dec 29, 2025
a09dc15
[AMORO-3940] Flink config load will cause ClassCastException when dir…
simonsssu Dec 29, 2025
3ecd13f
[AMORO-3943] Shade all third party classes for spark runtime (#3942)
turboFei Dec 29, 2025
303c921
[AMORO-4027] Bump shade plugin to fix compilation failure on JDK 17 (…
leosanqing Jan 5, 2026
d2714e1
[AMORO-3973] Support spark3.4 for mixed format (#4013)
zhangwl9 Jan 5, 2026
8e91a47
[Hotfix] Try to fix site-deploy permission problem (#4025)
sebbASF Jan 5, 2026
623561a
[AMORO-4022] [Improvement]: AMS Iceberg maintainer moved to the amoro…
czy006 Jan 7, 2026
afe60cf
[AMORO-3994] Support Exposing AMS High Availability (HA) Status (#3996)
davedwwang Jan 8, 2026
fbd91bc
[Feature]: Support LDAP Authentication for Dashboard Login (#4009)
davedwwang Jan 8, 2026
67d95f0
[HotFix] Remove duplicate references to scala-library and scala-compi…
zhangwl9 Jan 12, 2026
4253831
[HotFix] Shade third party classes for spark-3.4 runtime (#4032)
zhangwl9 Jan 13, 2026
f4f8fe6
[Subtask]: Use a new configuration item to control whether master & s…
Oct 29, 2025
f0918d0
[Subtask]: Use a new configuration item to control whether master & s…
Jan 15, 2026
b776399
[Subtask]: Use a new configuration item to control whether master & s…
Jan 15, 2026
1649ed5
[Subtask]: Use a new configuration item to control whether master & s…
Oct 29, 2025
71a6651
[Subtask]: Use a new configuration item to control whether master & s…
Oct 30, 2025
2799f29
[AMORO-3998]support DATABASE HA SERVICE (#3997)
LiangDai-Mars Dec 22, 2025
7079293
[AMORO-3994] Support Exposing AMS High Availability (HA) Status (#3996)
davedwwang Jan 8, 2026
ac2f468
[Subtask]: Use a new configuration item to control whether master & s…
Oct 29, 2025
0d8f33a
[Subtask]: Use a new configuration item to control whether master & s…
Jan 15, 2026
38e276a
Merge remote-tracking branch 'origin/amoro#3845' into amoro#3845
Jan 15, 2026
c381c73
Merge branch 'apache:master' into amoro#3845
wardlican Jan 15, 2026
185c030
[Subtask]: Optimize the description information for use-master-slave-…
Jan 16, 2026
d640ff6
Merge branch 'master' into amoro#3845
czy006 Jan 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,15 @@ public class AmoroManagementConf {
.defaultValue("admin")
.withDescription("The administrator password");

/** Enable master & slave mode, which supports horizontal scaling of AMS. */
public static final ConfigOption<Boolean> USE_MASTER_SLAVE_MODE =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m wondering whether we could control the single-node and multi-node modes with a configuration option, with the default being the single-node mode.

ConfigOptions.key("use-master-slave-mode")
.booleanType()
.defaultValue(false)
.withDescription(
"This setting controls whether to enable the AMS horizontal scaling feature, "
+ "which is currently under development and testing.");

public static final ConfigOption<Duration> CATALOG_META_CACHE_EXPIRATION_INTERVAL =
ConfigOptions.key("catalog-meta-cache.expiration-interval")
.durationType()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

package org.apache.amoro.server;

import static org.apache.amoro.server.AmoroManagementConf.USE_MASTER_SLAVE_MODE;

import io.javalin.Javalin;
import io.javalin.http.HttpCode;
import io.javalin.http.staticfiles.Location;
Expand Down Expand Up @@ -99,6 +101,7 @@ public class AmoroServiceContainer {
public static final Logger LOG = LoggerFactory.getLogger(AmoroServiceContainer.class);

public static final String SERVER_CONFIG_FILENAME = "config.yaml";
private static boolean IS_MASTER_SLAVE_MODE = false;

private final HighAvailabilityContainer haContainer;
private DataSource dataSource;
Expand Down Expand Up @@ -133,17 +136,24 @@ public static void main(String[] args) {
LOG.info("AMS service has been shut down");
}));
service.startRestServices();
while (true) {
try {
// Used to block AMS instances that have not acquired leadership
service.waitLeaderShip();
service.transitionToLeader();
// Used to block AMS instances that have acquired leadership
service.waitFollowerShip();
} catch (Exception e) {
LOG.error("AMS start error", e);
} finally {
service.transitionToFollower();
if (IS_MASTER_SLAVE_MODE) {
// Even if one does not become the master, it cannot block the subsequent logic.
service.registAndElect();
// Regardless of whether tp becomes the master, the service needs to be activated.
service.startOptimizingService();
} else {
while (true) {
try {
// Used to block AMS instances that have not acquired leadership
service.waitLeaderShip();
service.transitionToLeader();
// Used to block AMS instances that have acquired leadership
service.waitFollowerShip();
} catch (Exception e) {
LOG.error("AMS start error", e);
} finally {
service.transitionToFollower();
}
}
}
} catch (Throwable t) {
Expand All @@ -152,6 +162,10 @@ public static void main(String[] args) {
}
}

public void registAndElect() throws Exception {
haContainer.registAndElect();
}

public enum HAState {
INITIALIZING(0),
FOLLOWER(1),
Expand Down Expand Up @@ -306,6 +320,7 @@ public void dispose() {
private void initConfig() throws Exception {
LOG.info("initializing configurations...");
new ConfigurationHelper().init();
IS_MASTER_SLAVE_MODE = serviceConfig.getBoolean(USE_MASTER_SLAVE_MODE);
}

public Configurations getServiceConfig() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,9 @@ public void close() {
}
}

@Override
public void registAndElect() throws Exception {}

private class HeartbeatRunnable implements Runnable {
@Override
public void run() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,12 @@ public interface HighAvailabilityContainer {

/** Closes the container and releases resources. */
void close();

/**
* In master-slave mode, this is used for AMS nodes to register and participate in the master
* election process.
*
* @throws Exception If registration fails or participation in the primary election fails.
*/
void registAndElect() throws Exception;
}
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,7 @@ public void waitFollowerShip() throws InterruptedException {
public void close() {
LOG.info("Noop HA: closed");
}

@Override
public void registAndElect() throws Exception {}
}
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,11 @@ public void waitLeaderShip() throws Exception {
LOG.info("Became the leader of AMS");
}

@Override
public void registAndElect() throws Exception {
// TODO Here you can register for AMS and participate in the election.
}

@Override
public void waitFollowerShip() throws Exception {
LOG.info("Waiting to become the follower of AMS");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,14 @@ public void before() {
@AfterEach
public void after() {
LOG.info("Test finished.");
try {
// explicitly clean up possible residual table runtime records
if (catalog.tableExists(tableIdentifier)) {
catalog.dropTable(tableIdentifier, true);
}
} catch (Exception e) {
LOG.warn("Failed to drop table during cleanup", e);
}
catalog.dropDatabase(database);
}

Expand Down
1 change: 1 addition & 0 deletions docs/configuration/ams-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ table td:last-child, table th:last-child { width: 40%; word-break: break-all; }
| thrift-server.selector-thread-count | 2 | The number of selector threads for the Thrift server. |
| thrift-server.table-service.bind-port | 1260 | Port that the table service thrift server is bound to. |
| thrift-server.table-service.worker-thread-count | 20 | The number of worker threads for the Thrift server. |
| use-master-slave-mode | false | This setting controls whether to enable the AMS horizontal scaling feature, which is currently under development and testing. |


## Shade Utils Configuration
Expand Down