HA Distributed Cluster

[xnotcursed]

Hello, guys. I am trying to create a helm chart for the rustfs project. And I am trying to understand how exactly rustfs creates a HA Distributed Cluster?

For example, I think this is definitely incorrect, as it would just create 3 exact replicas of the same deployment:

rustfs/docs/docker-build.md

Lines 236 to 241 in ce2ce4b

	apiVersion: apps/v1
	kind: Deployment
	metadata:
	name: rustfs
	spec:
	replicas: 3

There is no node differentiation, no sharding (they all try to access the same PVC, what if it's RWO?) and no cluster coordination.

I guess it is the same as minio, where RUSTFS_VOLUMES specifies the server pool? But it would still only run in one container (as one server)?

rustfs/rustfs/src/main.rs

Lines 195 to 196 in ce2ce4b

	let (endpoint_pools, setup_type) =
	EndpointServerPools::from_volumes(server_address.clone().as_str(), opt.volumes.clone()).map_err(Error::other)?;

But in the compose files, you mention different nodes, so I guess there is a way to manually create distributed containers/servers?

rustfs/docker-compose.yaml

Lines 15 to 19 in ce2ce4b

	services:
	node0:
	image: rustfs:v1 # 替换为你的镜像名称和标签
	container_name: node0
	hostname: node0

Would love to contribute to your project with a helm chart, if I can have some support throughout

[loverustfs]

Hi @xnotcursed ,

We are more like MinIO:

1. Distributed hosts;
2. Distributed disks;
3. We do not support three copies, we support EC mode.

[xnotcursed]

So, how exactly do I specify more than one host? Because as I said, the Kubernetes deployment in your guys README is not correct I think, it would just create 3 replicas of the same thing (pointing to the same PVC) with same variables/setup.

In minio for example, in their CRD for tenant and helm chart, I can specify both servers and volumes per server (erasure coding and io parallelization), and it would actually run as multiple pods (hosts). But then again, they have a separate k8s operator to actually create the hosts based off of tenant CRDs

Here I don't really see a way to define more than one host to achieve HA. Maybe I am looking in the wrong places?

Example of HA with EC and multiple hosts (pods)
https://github.com/minio/operator/blob/7c1f234383353cd145ab17bbb65ab19ff6a9612a/helm/tenant/values.yaml#L127-L141

[loverustfs]

So, how exactly do I specify more than one host? Because as I said, the Kubernetes deployment in your guys README is not correct I think, it would just create 3 replicas of the same thing (pointing to the same PVC) with same variables/setup.

In minio for example, in their CRD for tenant and helm chart, I can specify both servers and volumes per server (erasure coding and io parallelization), and it would actually run as multiple pods (hosts). But then again, they have a separate k8s operator to actually create the hosts based off of tenant CRDs

Here I don't really see a way to define more than one host to achieve HA. Maybe I am looking in the wrong places?

Example of HA with EC and multiple hosts (pods) https://github.com/minio/operator/blob/7c1f234383353cd145ab17bbb65ab19ff6a9612a/helm/tenant/values.yaml#L127-L141

Hi @xnotcursed ,

Yes, we haven't dealt with k8s adaptation yet.
At present, we have just finished dealing with the various parameter issues of Dockerfile.

If you finally solve the deployment of helm or operator. You are very welcome to share your helm.
Thank you!

[xnotcursed]

Hello again, @loverustfs !
Can you please tell me how do I create a distributed cluster of nodes (not in Kubernetes, since you say you haven't started its adoption yet).

Is the docker-compose provided the correct way to form a cluster? I would appreciate any guidance on whether that compose config correctly establishes inter-node communication and cluster coordination, and not just creates multiple independent instances of rustfs.

rustfs/docker-compose.yaml

Lines 15 to 83 in ce2ce4b

	services:
	node0:
	image: rustfs:v1 # 替换为你的镜像名称和标签
	container_name: node0
	hostname: node0
	environment:
	- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
	- RUSTFS_ADDRESS=0.0.0.0:9000
	- RUSTFS_CONSOLE_ENABLE=true
	- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9002
	platform: linux/amd64
	ports:
	- "9000:9000" # 映射宿主机的 9001 端口到容器的 9000 端口
	- "8000:9001" # 映射宿主机的 9001 端口到容器的 9000 端口
	volumes:
	- ./target/x86_64-unknown-linux-musl/release/rustfs:/app/rustfs
	# - ./data/node0:/data # 将当前路径挂载到容器内的 /root/data
	command: "/app/rustfs"
	node1:
	image: rustfs:v1
	container_name: node1
	hostname: node1
	environment:
	- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
	- RUSTFS_ADDRESS=0.0.0.0:9000
	- RUSTFS_CONSOLE_ENABLE=true
	- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9002
	platform: linux/amd64
	ports:
	- "9001:9000" # 映射宿主机的 9002 端口到容器的 9000 端口
	volumes:
	- ./target/x86_64-unknown-linux-musl/release/rustfs:/app/rustfs
	# - ./data/node1:/data
	command: "/app/rustfs"
	node2:
	image: rustfs:v1
	container_name: node2
	hostname: node2
	environment:
	- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
	- RUSTFS_ADDRESS=0.0.0.0:9000
	- RUSTFS_CONSOLE_ENABLE=true
	- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9002
	platform: linux/amd64
	ports:
	- "9002:9000" # 映射宿主机的 9003 端口到容器的 9000 端口
	volumes:
	- ./target/x86_64-unknown-linux-musl/release/rustfs:/app/rustfs
	# - ./data/node2:/data
	command: "/app/rustfs"
	node3:
	image: rustfs:v1
	container_name: node3
	hostname: node3
	environment:
	- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
	- RUSTFS_ADDRESS=0.0.0.0:9000
	- RUSTFS_CONSOLE_ENABLE=true
	- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9002
	platform: linux/amd64
	ports:
	- "9003:9000" # 映射宿主机的 9004 端口到容器的 9000 端口
	volumes:
	- ./target/x86_64-unknown-linux-musl/release/rustfs:/app/rustfs
	# - ./data/node3:/data
	command: "/app/rustfs"

Also regarding the k8s operator, I would be open to contributing, but in my opinion, this project should be separated from the main product and probably have its own repo. If you decide to create one in the future, that would be great!

Thank you!

[loverustfs]

Hi @xnotcursed ,

Thank you very much for your suggestion.

We have opened the Operator repository:https://github.com/rustfs/operator

You are right, but the best practice is to start with 4 nodes.

In addition, we closed port 9001 and integrated 9000 with 9001.

We look forward to your contribution. Together we can help the world make data safer.