Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 57 additions & 8 deletions docs/en/guide/cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,22 +12,50 @@ With CSI Driver, you can use either a host directory or a PVC for cache storage.
* If all worker nodes are used to run JuiceFS Mount Pods, and they host similar cache content (similar situation if you use distributed caching), Pod migration is not really a problem, and you can still use host directories for cache storage.
* When using a PVC for cache storage, different JuiceFS PVs can isolate cache data. If the Mount Pod is migrated to another node, the PVC reference remains the same. This ensures that the cache is unaffected.

## Using host directories (`hostPath`) {#cache-settings}
## Using host path (`hostPath`) {#cache-settings}

For Kubernetes nodes, a dedicated disk is often used as data and cache storage, be sure to properly configure the cache directory, or JuiceFS cache will by default be written to `/var/jfsCache`, which can easily eat up system storage space.
By default, CSI Driver uses the standard JuiceFS Client cache directory `/var/jfsCache` on the host, if you intend to use data disk as cache storage, make sure the correct path is configured, otherwise cache can drain the system disk.

After the cache directory is set, it will be accessible in the Mount Pod via `hostPath`. You might also need to configure other cache-related options (like `--cache-size`) according to [Adjust mount options](./configurations.md#mount-options).
Specify `--cache-dir` in mount options, preferably in ConfigMap, and then CSI Driver will handle the mounts accordingly:

```yaml {6} title="values-mycluster.yaml"
...
globalConfig:
enabled: true
mountPodPatch:
- mountOptions:
- cache-dir=/data/cache
- cache-size=10T
```

When Mount Pod starts, it will include the hostPath mounts:

```yaml {4,9}
...
volumeMounts:
...
- mountPath: /data/cache
name: cachedir-0
volumes:
...
- hostPath:
path: /data/cache
type: DirectoryOrCreate
name: cachedir-0
```

If you need to further customize cache related options, check out the option list in [JuiceFS Community Edition](https://juicefs.com/docs/community/command_reference/#mount) and [JuiceFS Cloud Service](https://juicefs.com/docs/cloud/reference/commands_reference/#mount).

:::note

* In CSI Driver, `cache-dir` parameter does not support wildcard character, if you need to use multiple disks as storage devices, specify multiple directories joined by the `:` character. See [JuiceFS Community Edition](https://juicefs.com/docs/community/command_reference/#mount) and [JuiceFS Cloud Service](https://juicefs.com/docs/cloud/reference/commands_reference/#mount).
* In CSI Driver, `cache-dir` parameter does not support wildcard character, if you need to use multiple disks as storage devices, specify multiple directories joined by the `:` character.
* For scenarios that involve intensive small writes, we usually recommend users to temporarily enable client write cache, but due to its inherent risks, this is advised against when using CSI Driver, because Pod lifecycle is significantly more unstable, and can cause data loss if Pod exists unexpectedly.

:::

### Using ConfigMap

Read [Custom directory](./configurations.md#custom-cachedirs).
Demostrated in the above code snippets.

### Define in PV (deprecated)

Expand Down Expand Up @@ -81,20 +109,41 @@ mountOptions:

## Use PVC as cache path

From 0.15.1 and above, JuiceFS CSI Driver supports using a PVC as cache directory. This is often used in hosted Kubernetes clusters provided by cloud services, which allows you to use a dedicated cloud disk as cache storage for CSI Driver.
If you have higher demands for cache isolation, or cannot use hostPath for cache due to other reasons, consider using PVC as cache storage.

First, create a PVC according to your cloud service provider's manual, for example:
PVC should be created in advance, and if you are using one of the following service providers, you can refer to their manual:

* [Amazon EBS CSI Driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html)
* [Use the Azure Disks CSI Driver in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/azure-disk-csi)
* [Using the Google Compute Engine persistent disk CSI Driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver)
* [DigitalOcean Volumes Block Storage](https://docs.digitalocean.com/products/kubernetes/how-to/add-volumes)

:::tip
For custom volumes, make sure `mountPath` and `hostPath` doesn't contain duplicates, to avoid conflicts.
:::

Assuming a PVC named `jfs-cache-pvc` is already created in the same namespace as the Mount Pod (which defaults to `kube-system`), use the following example to set this PVC as the cache directory for JuiceFS CSI Driver.

### Using ConfigMap

Read [Custom directory](./configurations.md#custom-cachedirs).
The minimum required version is CSI Driver v0.25.1. Upon modification, application Pods need to be re-created for changes to take effect.

When multiple cache directories are used, make sure all items have the same available capacity, and then set `--cache-size` to the sum.

```yaml
- cacheDirs:
- type: PVC
name: jfs-cache-pvc
- type: HostPath
path: /var/jfsCache
mountOptions:
- cache-size=204800
- free-space-ratio=0.01
# Optional, used when you need to customize individual PVCs
pvcSelector:
matchLabels:
need-cachedirs: "true"
```

### Define in PV (deprecated)

Expand Down
33 changes: 2 additions & 31 deletions docs/en/guide/configurations.md
Original file line number Diff line number Diff line change
Expand Up @@ -363,38 +363,9 @@ stringData:
configs: "{gc-secret: /root/.config/gcloud}"
```

### Customize cache directory {#custom-cachedirs}
### Cache {#custom-cachedirs}

This section only covers basic usage. For more details on cache warmup and cleanup, see [Cache settings](./cache.md).

With CSI Driver, you can use either a host directory or a PVC for cache storage. Note that:

- When using a PVC as a cache path, ensure that the PVC is created in advance and is in the same namespace as the Mount Pod.
- When using a custom volume, make sure its `mountPath` does not conflict with other `hostPath` directories.

#### Using ConfigMap

The minimum required version is CSI Driver v0.25.1. Upon modification, application Pods need to be re-created for changes to take effect.

```yaml
- pvcSelector:
matchLabels:
need-cachedirs: "true"
cacheDirs:
- type: PVC
name: jfs-cache-pvc
- type: HostPath
path: /var/jfsCache
```

#### Using mount options

This method only supports `hostPath` cache directories.

```yaml
mountOptions:
- cache-dir=/mnt/jfsCache1:/mnt/jfsCache2
```
Cache usage is also closely related to resource definition, data warmup, and cache cleanup, navigate to [Cache](./cache.md) to learn more.

### Other features

Expand Down
69 changes: 59 additions & 10 deletions docs/zh_cn/guide/cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,22 +11,50 @@ JuiceFS 有着强大的缓存设计,阅读[社区版文档](https://juicefs.co
* 如果你的集群中,所有节点均用于运行 Mount Pod,那么由于每个宿主机都持有大致相同的缓存(或者使用了分布式缓存),那么 Pod 漂移的问题可能也并不构成影响,完全可以使用宿主机路径作为缓存。
* 如果用 PVC 作为缓存存储,好处是不同 JuiceFS PV 可以隔离缓存数据、分别管理,并且就算 Mount Pod 随着业务被迁移到其他节点,由于 PVC 引用关系不变,所以缓存数据仍然可以访问。

## 使用宿主机目录作为缓存(`hostPath`) {#cache-settings}
## 使用宿主机路径(`hostPath`) {#cache-settings}

Kubernetes 节点往往采用单独的数据盘作为缓存盘,因此使用 JuiceFS 时,一定要注意正确设置缓存路径,否则默认使用根分区的 `/var/jfsCache` 目录来缓存数据,极易耗尽磁盘空间
默认情况下,CSI 驱动的缓存路径就是标准的 JuiceFS 客户端缓存路径 `/var/jfsCache`,考虑到 Kubernetes 节点往往采用单独的数据盘作为缓存盘,因此一定要注意正确设置缓存路径,否则使用根分区的 `/var/jfsCache` 目录来缓存数据,容易耗尽系统盘空间

设置缓存路径以后,Kubernetes 宿主机上的路径会以 `hostPath` 卷的形式挂载到 Mount Pod 中,因此还需要根据缓存盘参数,对缓存相关的[挂载参数](./configurations.md#mount-options)进行调整(如缓存大小)。
在挂载参数中指定好 `--cache-dir`,CSI 驱动就会自动将目标路径映射到 Pod 内,比方说在 ConfigMap 中这样配置:

:::tip 注意
```yaml {6} title="values-mycluster.yaml"
...
globalConfig:
enabled: true
mountPodPatch:
- mountOptions:
- cache-dir=/data/cache
- cache-size=10T
```

那么随着 Mount Pod 启动,CSI 驱动就会为其加入对应的宿主机路径挂载:

```yaml {4,9}
...
volumeMounts:
...
- mountPath: /data/cache
name: cachedir-0
volumes:
...
- hostPath:
path: /data/cache
type: DirectoryOrCreate
name: cachedir-0
```

* 在 CSI 驱动中,`cache-dir` 不支持填写通配符,如果需要用多个设备作为缓存盘,填写多个目录,以 `:` 连接。详见[社区版](https://juicefs.com/docs/zh/community/command_reference/#mount)与[云服务](https://juicefs.com/docs/zh/cloud/reference/commands_reference/#mount)文档。
如果还需要进一步调整其他缓存相关的挂载参数,请阅读对应的 JuiceFS 客户端参数列表,详见[社区版](https://juicefs.com/docs/zh/community/command_reference/#mount)与[云服务](https://juicefs.com/docs/zh/cloud/reference/commands_reference/#mount)文档。

:::tip

* 在 CSI 驱动中,`cache-dir` 不支持填写通配符 `*`,如果需要用多个设备作为缓存盘,必须直接填写多个目录,以 `:` 连接。
* 对于大量小文件写入场景,我们一般推荐临时开启客户端写缓存,但由于该模式本身带来的数据安全风险,我们尤其不推荐在 CSI 驱动中开启 `--writeback`,避免容器出现意外时,写缓存尚未完成上传,造成数据无法访问。

:::

### 使用 ConfigMap

阅读[「调整挂载参数」](./configurations.md#custom-cachedirs)
已经在上方代码块中进行示范

### 在 PV 中定义(不推荐)

Expand Down Expand Up @@ -78,22 +106,43 @@ mountOptions:
- cache-size=204800
```

## 使用 PVC 作为缓存路径
## 使用 PVC

JuiceFS CSI 驱动 0.15.1 及以上版本支持使用 PVC 作为缓存路径,该实践多用于托管 Kubernetes 集群的云服务商,让你可以使用单独的云盘来作为 CSI 驱动的缓存存储设备
如果对隔离程度有要求,或者因为其他原因而无法使用宿主机路径作为缓存目的地,那么可以使用 PVC 作为缓存

首先,按照所使用的托管 Kubernetes 集群的云服务商的说明,创建 PVC,比如
如果使用 PVC 作为缓存路径,PVC 需要提前创建,并确保和 Mount Pod 在同一个命名空间(namespace)下,如果你正在使用公有云托管服务,以下是部分常用服务商的文档

* [Amazon EBS CSI 驱动](https://docs.aws.amazon.com/zh_cn/eks/latest/userguide/ebs-csi.html)
* [在 Azure Kubernetes 服务(AKS)中使用 Azure 磁盘 CSI 驱动](https://learn.microsoft.com/zh-cn/azure/aks/azure-disk-csi)
* [使用 Google Compute Engine 永久性磁盘 CSI 驱动](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver)
* [阿里云 ACK 云盘存储卷](https://help.aliyun.com/document_detail/134767.html)

:::tip
如果自定义了 Volume,确保 `mountPath` 和 `hostPath` 没有重复项目,避免冲突。
:::

假设名为 `jfs-cache-pvc` 的 PVC 创建完毕,与 Mount Pod 在同一个命名空间下(默认 `kube-system`),参考下方示范,让 CSI 驱动使用该 PVC 作为缓存路径。

### 使用 ConfigMap

缓存相关配置均通过挂载参数进行调整,请阅读[「调整挂载参数」](./configurations.md#custom-cachedirs)。
该功能最低需要 CSI 驱动版本 v0.25.1,修改后需重建业务 Pod 生效。

如果使用了多个缓存路径,需要注意使用大小相同的存储设备,并且将 `--cache-size` 设置为可用空间之和。

```yaml
- cacheDirs:
- type: PVC
name: jfs-cache-pvc
- type: HostPath
path: /var/jfsCache
mountOptions:
- cache-size=204800
- free-space-ratio=0.01
# 可选字段,对特定 PVC 定制缓存配置时使用
pvcSelector:
matchLabels:
need-cachedirs: "true"
```

### 在 PV 中定义(不推荐)

Expand Down
33 changes: 2 additions & 31 deletions docs/zh_cn/guide/configurations.md
Original file line number Diff line number Diff line change
Expand Up @@ -361,38 +361,9 @@ stringData:
configs: "{gc-secret: /root/.config/gcloud}"
```

### 定制缓存路径 {#custom-cachedirs}
### 缓存 {#custom-cachedirs}

本小节只介绍如何设置缓存路径,如有需要,请继续阅读[缓存](./cache.md)了解如何设置多个缓存目录,以及缓存的预热和清理。

在 CSI 驱动中,可以选择使用宿主机路径或者 PVC 作为 JuiceFS 客户端缓存,对于这两种场景,需要注意:

- 如果使用 PVC 作为缓存路径,PVC 需要提前创建,并确保和 Mount Pod 在同一个 namespace。
- 如果自定义了 Volume,确保自定义 Volume 中的 `mountPath` 和 `hostPath` 没有重复,避免冲突。

#### 使用 ConfigMap

该功能最低需要 CSI 驱动版本 v0.25.1,修改后需重建业务 Pod 生效。

```yaml
- pvcSelector:
matchLabels:
need-cachedirs: "true"
cacheDirs:
- type: PVC
name: jfs-cache-pvc
- type: HostPath
path: /var/jfsCache
```

#### 使用挂载参数

该方法只支持配置 `hostPath` 形式的缓存路径。

```yaml
mountOptions:
- cache-dir=/mnt/jfsCache1:/mnt/jfsCache2
```
缓存的使用还涉及资源管理、数据预热和清理等事项,因此请移步阅读[缓存](./cache.md)来详细了解。

### 其他功能定制

Expand Down