Very Slow

[inkeliz]

I'm testing Garage, RustFS and others... I'm now testing Ozone, but that is very slow.

I followed the bare-metal instalation (https://ozone.apache.org/docs/quick-start/installation/bare-metal), however doing:

bin/ozone sh volume create /vol1
bin/ozone sh bucket create /vol1/bucket1
echo "Hello Ozone" > /tmp/test.txt
bin/ozone sh key put /vol1/bucket1/key1 /tmp/test.txt
bin/ozone sh key list /vol1/bucket1

Never finish, for hours.
Cancelling and trying again it's stuck in the first command, without any error or message.

Why?

[errose28]

Hi @inkeliz, sounds like your client is either unable to connect to the Ozone Managers, or is able to connect but unable to identify the leader.

• Do you see anything in the client logs?
• Do you see entries for the client requests in the Ozone Manager audit logs?
• Check the output of the command ozone admin om roles to see if the Ozone Managers have elected a leader.
• Check the Ozone Manager logs for ratis related messages about leader elections

[inkeliz]

The OM seems to crash and doesn't start anymore:

2026-02-17 20:50:19,538 [main] INFO org.reflections.Reflections: Reflections took 81 ms to scan 1 urls, producing 159 keys and 477 values
2026-02-17 20:50:19,543 [main] INFO org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManager: Skipping Upgrade Action QuotaRepairUpgradeAction since it has been finalized.
2026-02-17 20:50:19,546 [main] WARN org.apache.hadoop.hdds.server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
2026-02-17 20:50:19,553 [main] INFO org.apache.hadoop.ozone.om.OzoneManager: Set default replication in OM: RATIS/3 -> RATIS/THREE
2026-02-17 20:50:19,588 [main] INFO org.apache.hadoop.hdds.scm.proxy.SCMContainerLocationFailoverProxyProvider: Created fail-over proxy for protocol StorageContainerLocationProtocolPB with 1 nodes: [nodeId=scmNodeId,nodeAddress=sybil-a/127.0.1.1:9860]
2026-02-17 20:50:19,637 [main] INFO org.apache.hadoop.hdds.scm.proxy.SCMBlockLocationFailoverProxyProvider: Created fail-over proxy for protocol ScmBlockLocationProtocolPB with 1 nodes: [nodeId=scmNodeId,nodeAddress=sybil-a/127.0.1.1:9863]
2026-02-17 20:50:19,652 [main] INFO org.apache.hadoop.hdds.scm.proxy.SCMBlockLocationFailoverProxyProvider: Created fail-over proxy for protocol ScmBlockLocationProtocolPB with 1 nodes: [nodeId=scmNodeId,nodeAddress=sybil-a/127.0.1.1:9863]
2026-02-17 20:50:19,723 [main] INFO org.apache.hadoop.ozone.om.OzoneManager: OM start with adminUsers: [ozone]
2026-02-17 20:50:19,739 [main] WARN org.apache.hadoop.hdds.server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
2026-02-17 20:50:19,855 [main] INFO org.apache.hadoop.hdds.utils.NativeLibraryLoader: Loading Library: ozone_rocksdb_tools
2026-02-17 20:50:19,855 [main] WARN org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Native Library for raw sst file reading loading failed. Cannot prune OMKeyInfo from SST files. Unable to load library ozone_rocksdb_tools from both java.library.path & resource file libozone_rocksdb_tools.so from jar.
2026-02-17 20:50:19,968 [main] INFO org.apache.hadoop.ozone.om.OzoneManager: S3 Multi-Tenancy is disabled
2026-02-17 20:50:19,974 [main] INFO org.apache.hadoop.ozone.security.acl.OzoneAuthorizerFactory: om1: Authorizer for OM is class org.apache.hadoop.ozone.security.acl.OzoneAccessAuthorizer
2026-02-17 20:50:19,979 [main] INFO org.apache.hadoop.ozone.om.OmSnapshotManager: Ozone filesystem snapshot feature is enabled.
2026-02-17 20:50:19,983 [main] WARN org.apache.hadoop.hdds.server.ServerUtils: ozone.om.snapshot.diff.db.dir is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead.
2026-02-17 20:50:20,002 [main] INFO org.apache.hadoop.hdds.utils.NativeLibraryLoader: Loading Library: ozone_rocksdb_tools
2026-02-17 20:50:20,002 [main] WARN org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Native Library for raw sst file reading loading failed. Fallback to performing a full diff instead. Unable to load library ozone_rocksdb_tools from both java.library.path & resource file libozone_rocksdb_tools.so from jar.
2026-02-17 20:50:20,004 [main] INFO org.apache.hadoop.hdds.utils.BackgroundService: Starting service SnapshotDiffCleanupService with interval 60000 milliseconds
2026-02-17 20:50:20,025 [main] WARN org.apache.hadoop.hdds.server.ServerUtils: Storage directory for Ratis is not configured. It is a good idea to map this to an SSD disk. Falling back to ozone.metadata.dirs
2026-02-17 20:50:20,026 [main] WARN org.apache.hadoop.ozone.om.ratis.utils.OzoneManagerRatisUtils: ozone.om.ratis.snapshot.dir is not configured. Falling back to ozone.metadata.dirs config
2026-02-17 20:50:20,030 [main] ERROR org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with exception
java.io.IOException: Ratis group Dir on disk tmp does not match with RaftGroupIDbf265839-605b-3f16-9796-c5ba1605619e generated from service id omServiceIdDefault. Looks like there is a change to ozone.om.service.ids value after the cluster is setup. Currently change to this value is not supported.
	at org.apache.hadoop.ozone.om.OzoneManager.initializeRatisDirs(OzoneManager.java:1659)
	at org.apache.hadoop.ozone.om.OzoneManager.<init>(OzoneManager.java:704)
	at org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:894)
	at org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.start(OzoneManagerStarter.java:188)
	at org.apache.hadoop.ozone.om.OzoneManagerStarter.startOm(OzoneManagerStarter.java:85)
	at org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:73)
	at org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:1)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2031)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2469)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2461)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2423)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2425)
	at picocli.CommandLine.execute(CommandLine.java:2174)
	at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:89)
	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:80)
	at org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:57)
2026-02-17 20:50:20,032 [shutdown-hook-0] INFO org.apache.hadoop.ozone.om.OzoneManagerStarter: SHUTDOWN_MSG: 

[inkeliz]

Now the issue is:

Ratis group Dir on disk tmp does not match with RaftGroupIDbf265839-605b-3f16-9796-c5ba1605619e generated from service id omServiceIdDefault. Looks like there is a change to ozone.om.service.ids value after the cluster is setup. Currently change to this value is not supported.
real-time non-blocking time  (microseconds, -R) unlimited

Even deleting all files, that keep happening after start it again.

---

Anyway, I will just ignore Apache Ozone. I never had good experience with anything form Apache.

[silvanias]

Which version are you on? Adjacent issue was fixed in 2.0.0:
https://issues.apache.org/jira/browse/HDDS-10909

If you are up to date, then it's likely that the OM is just misconfigured. In ozone-site.xml:

<property>
  <name>ozone.om.service.ids</name>
  <value>omService1</value>
</property>

make sure this is set before initializing your cluster. Now it appears you need a fresh start on your cluster, and then the issue will be fixed.

[errose28]

With docker, I can't repro this following the latest instructions. If you are doing a bare metal deployment, please share your configurations and which nodes are co-located.