Skip to content

Commit 9ef76dc

Browse files
hkfireshkfires
committed
Add Object Storage
1 parent 1cd275f commit 9ef76dc

File tree

9 files changed

+849
-24
lines changed

9 files changed

+849
-24
lines changed

‎.env.example‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Example environment configuration for CLIProxyAPI.
2+
# Copy this file to `.env` and uncomment the variables you need.
3+
#
4+
# NOTE: Environment variables are only required when using remote storage options.
5+
# For local file-based storage (default), no environment variables need to be set.
6+
7+
# ------------------------------------------------------------------------------
8+
# Management Web UI
9+
# ------------------------------------------------------------------------------
10+
# MANAGEMENT_PASSWORD=change-me-to-a-strong-password
11+
12+
# ------------------------------------------------------------------------------
13+
# Postgres Token Store (optional)
14+
# ------------------------------------------------------------------------------
15+
# PGSTORE_DSN=postgresql://user:pass@localhost:5432/cliproxy
16+
# PGSTORE_SCHEMA=public
17+
# PGSTORE_LOCAL_PATH=/var/lib/cliproxy
18+
19+
# ------------------------------------------------------------------------------
20+
# Git-Backed Config Store (optional)
21+
# ------------------------------------------------------------------------------
22+
# GITSTORE_GIT_URL=https://github.com/your-org/cli-proxy-config.git
23+
# GITSTORE_GIT_USERNAME=git-user
24+
# GITSTORE_GIT_TOKEN=ghp_your_personal_access_token
25+
# GITSTORE_LOCAL_PATH=/data/cliproxy/gitstore
26+
27+
# ------------------------------------------------------------------------------
28+
# Object Store Token Store (optional)
29+
# ------------------------------------------------------------------------------
30+
# OBJECTSTORE_ENDPOINT=https://s3.your-cloud.example.com
31+
# OBJECTSTORE_BUCKET=cli-proxy-config
32+
# OBJECTSTORE_ACCESS_KEY=your_access_key
33+
# OBJECTSTORE_SECRET_KEY=your_secret_key
34+
# OBJECTSTORE_LOCAL_PATH=/data/cliproxy/objectstore

‎.gitignore‎

Lines changed: 23 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,32 @@
1+
# Binaries
2+
cli-proxy-api
3+
*.exe
4+
5+
# Configuration
16
config.yaml
7+
.env
8+
9+
# Generated content
210
bin/*
3-
docs/*
411
logs/*
512
conv/*
13+
temp/*
14+
pgstore/*
15+
gitstore/*
16+
objectstore/*
17+
static/*
18+
19+
# Authentication data
620
auths/*
721
!auths/.gitkeep
8-
.vscode/*
9-
.claude/*
10-
.serena/*
22+
23+
# Documentation
24+
docs/*
1125
AGENTS.md
1226
CLAUDE.md
1327
GEMINI.md
14-
*.exe
15-
temp/*
16-
cli-proxy-api
17-
static/*
18-
.env
19-
pgstore/*
20-
gitstore/*
28+
29+
# Tooling metadata
30+
.vscode/*
31+
.claude/*
32+
.serena/*

‎README.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -456,6 +456,27 @@ You can also persist configuration and authentication data in PostgreSQL when ru
456456
3. **Bootstrapping:** If no configuration row exists, `config.example.yaml` seeds the database using the fixed identifier `config`.
457457
4. **Token Sync:** Changes flow both ways—file updates are written to PostgreSQL and database records are mirrored back to disk so watchers and management APIs continue to operate.
458458

459+
### Object Storage-backed Configuration and Token Store
460+
461+
An S3-compatible object storage service can host configuration and authentication records.
462+
463+
**Environment Variables**
464+
465+
| Variable | Required | Default | Description |
466+
|--------------------------|----------|--------------------------------|--------------------------------------------------------------------------------------------------------------------------|
467+
| `OBJECTSTORE_ENDPOINT` | Yes | | Object storage endpoint. Include `http://` or `https://` to force the protocol (omitted scheme → HTTPS). |
468+
| `OBJECTSTORE_BUCKET` | Yes | | Bucket that stores `config/config.yaml` and `auths/*.json`. |
469+
| `OBJECTSTORE_ACCESS_KEY` | Yes | | Access key ID for the object storage account. |
470+
| `OBJECTSTORE_SECRET_KEY` | Yes | | Secret key for the object storage account. |
471+
| `OBJECTSTORE_LOCAL_PATH` | No | Current working directory | Root directory for the local mirror; the server writes to `<value>/objectstore`. If unset, defaults to current CWD. |
472+
473+
**How it Works**
474+
475+
1. **Startup:** The endpoint is parsed (respecting any scheme prefix), a MinIO-compatible client is created in path-style mode, and the bucket is created when missing.
476+
2. **Local Mirror:** A writable cache at `<OBJECTSTORE_LOCAL_PATH or CWD>/objectstore` mirrors `config/config.yaml` and `auths/`.
477+
3. **Bootstrapping:** When `config/config.yaml` is absent in the bucket, the server copies `config.example.yaml`, uploads it, and uses it as the initial configuration.
478+
4. **Sync:** Changes to configuration or auth files are uploaded to the bucket, and remote updates are mirrored back to disk, keeping watchers and management APIs in sync.
479+
459480
### OpenAI Compatibility Providers
460481

461482
Configure upstream OpenAI-compatible providers (e.g., OpenRouter) via `openai-compatibility`.

‎README_CN.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -469,6 +469,27 @@ openai-compatibility:
469469
3. **引导：** 若数据库中无配置记录，会使用 `config.example.yaml` 初始化，并以固定标识 `config` 写入。
470470
4. **令牌同步：** 配置与令牌的更改会写入 PostgreSQL，同时数据库中的内容也会反向同步至本地镜像，便于文件监听与管理接口继续工作。
471471

472+
### 对象存储驱动的配置与令牌存储
473+
474+
可以选择使用 S3 兼容的对象存储来托管配置与鉴权数据。
475+
476+
**环境变量**
477+
478+
| 变量 | 是否必填 | 默认值 | 说明 |
479+
|--------------------------|----------|--------------------|--------------------------------------------------------------------------------------------------------------------------|
480+
| `OBJECTSTORE_ENDPOINT` | 是 | | 对象存储访问端点。可带 `http://` 或 `https://` 前缀指定协议（省略则默认 HTTPS）。 |
481+
| `OBJECTSTORE_BUCKET` | 是 | | 用于存放 `config/config.yaml` 与 `auths/*.json` 的 Bucket 名称。 |
482+
| `OBJECTSTORE_ACCESS_KEY` | 是 | | 对象存储账号的访问密钥 ID。 |
483+
| `OBJECTSTORE_SECRET_KEY` | 是 | | 对象存储账号的访问密钥 Secret。 |
484+
| `OBJECTSTORE_LOCAL_PATH` | 否 | 当前工作目录 (CWD) | 本地镜像根目录；服务会写入到 `<值>/objectstore`。 |
485+
486+
**工作流程**
487+
488+
1. **启动阶段：** 解析端点地址（识别协议前缀），创建 MinIO 兼容客户端并使用 Path-Style 模式，如 Bucket 不存在会自动创建。
489+
2. **本地镜像：** 在 `<OBJECTSTORE_LOCAL_PATH 或当前工作目录>/objectstore` 维护可写缓存，同步 `config/config.yaml` 与 `auths/`。
490+
3. **初始化：** 若 Bucket 中缺少配置文件，将以 `config.example.yaml` 为模板生成 `config/config.yaml` 并上传。
491+
4. **双向同步：** 本地变更会上传到对象存储，同时远端对象也会拉回到本地，保证文件监听、管理 API 与 CLI 命令行为一致。
492+
472493
### OpenAI 兼容上游提供商
473494

474495
通过 `openai-compatibility` 配置上游 OpenAI 兼容提供商（例如 OpenRouter）。

‎cmd/server/main.go‎

Lines changed: 105 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,13 @@ import (
99
"flag"
1010
"fmt"
1111
"io/fs"
12+
"net/url"
1213
"os"
1314
"path/filepath"
1415
"strings"
1516
"time"
1617

18+
"github.com/joho/godotenv"
1719
configaccess "github.com/router-for-me/CLIProxyAPI/v6/internal/access/config_access"
1820
"github.com/router-for-me/CLIProxyAPI/v6/internal/cmd"
1921
"github.com/router-for-me/CLIProxyAPI/v6/internal/config"
@@ -102,25 +104,39 @@ func main() {
102104
var cfg *config.Config
103105
var isCloudDeploy bool
104106
var (
105-
usePostgresStore bool
106-
pgStoreDSN string
107-
pgStoreSchema string
108-
pgStoreLocalPath string
109-
pgStoreInst *store.PostgresStore
110-
gitStoreLocalPath string
111-
useGitStore bool
112-
gitStoreRemoteURL string
113-
gitStoreUser string
114-
gitStorePassword string
115-
gitStoreInst *store.GitTokenStore
116-
gitStoreRoot string
107+
usePostgresStore bool
108+
pgStoreDSN string
109+
pgStoreSchema string
110+
pgStoreLocalPath string
111+
pgStoreInst *store.PostgresStore
112+
useGitStore bool
113+
gitStoreRemoteURL string
114+
gitStoreUser string
115+
gitStorePassword string
116+
gitStoreLocalPath string
117+
gitStoreInst *store.GitTokenStore
118+
gitStoreRoot string
119+
useObjectStore bool
120+
objectStoreEndpoint string
121+
objectStoreAccess string
122+
objectStoreSecret string
123+
objectStoreBucket string
124+
objectStoreLocalPath string
125+
objectStoreInst *store.ObjectTokenStore
117126
)
118127

119128
wd, err := os.Getwd()
120129
if err != nil {
121130
log.Fatalf("failed to get working directory: %v", err)
122131
}
123132

133+
// Load environment variables from .env if present.
134+
if errLoad := godotenv.Load(filepath.Join(wd, ".env")); errLoad != nil {
135+
if !errors.Is(errLoad, os.ErrNotExist) {
136+
log.WithError(errLoad).Warn("failed to load .env file")
137+
}
138+
}
139+
124140
lookupEnv := func(keys ...string) (string, bool) {
125141
for _, key := range keys {
126142
if value, ok := os.LookupEnv(key); ok {
@@ -157,6 +173,22 @@ func main() {
157173
if value, ok := lookupEnv("GITSTORE_LOCAL_PATH", "gitstore_local_path"); ok {
158174
gitStoreLocalPath = value
159175
}
176+
if value, ok := lookupEnv("OBJECTSTORE_ENDPOINT", "objectstore_endpoint"); ok {
177+
useObjectStore = true
178+
objectStoreEndpoint = value
179+
}
180+
if value, ok := lookupEnv("OBJECTSTORE_ACCESS_KEY", "objectstore_access_key"); ok {
181+
objectStoreAccess = value
182+
}
183+
if value, ok := lookupEnv("OBJECTSTORE_SECRET_KEY", "objectstore_secret_key"); ok {
184+
objectStoreSecret = value
185+
}
186+
if value, ok := lookupEnv("OBJECTSTORE_BUCKET", "objectstore_bucket"); ok {
187+
objectStoreBucket = value
188+
}
189+
if value, ok := lookupEnv("OBJECTSTORE_LOCAL_PATH", "objectstore_local_path"); ok {
190+
objectStoreLocalPath = value
191+
}
160192

161193
// Check for cloud deploy mode only on first execution
162194
// Read env var name in uppercase: DEPLOY
@@ -196,6 +228,65 @@ func main() {
196228
cfg.AuthDir = pgStoreInst.AuthDir()
197229
log.Infof("postgres-backed token store enabled, workspace path: %s", pgStoreInst.WorkDir())
198230
}
231+
} else if useObjectStore {
232+
objectStoreRoot := objectStoreLocalPath
233+
if objectStoreRoot == "" {
234+
objectStoreRoot = wd
235+
}
236+
objectStoreRoot = filepath.Join(objectStoreRoot, "objectstore")
237+
resolvedEndpoint := strings.TrimSpace(objectStoreEndpoint)
238+
useSSL := true
239+
if strings.Contains(resolvedEndpoint, "://") {
240+
parsed, errParse := url.Parse(resolvedEndpoint)
241+
if errParse != nil {
242+
log.Fatalf("failed to parse object store endpoint %q: %v", objectStoreEndpoint, errParse)
243+
}
244+
switch strings.ToLower(parsed.Scheme) {
245+
case "http":
246+
useSSL = false
247+
case "https":
248+
useSSL = true
249+
default:
250+
log.Fatalf("unsupported object store scheme %q (only http and https are allowed)", parsed.Scheme)
251+
}
252+
if parsed.Host == "" {
253+
log.Fatalf("object store endpoint %q is missing host information", objectStoreEndpoint)
254+
}
255+
resolvedEndpoint = parsed.Host
256+
if parsed.Path != "" && parsed.Path != "/" {
257+
resolvedEndpoint = strings.TrimSuffix(parsed.Host+parsed.Path, "/")
258+
}
259+
}
260+
resolvedEndpoint = strings.TrimRight(resolvedEndpoint, "/")
261+
objCfg := store.ObjectStoreConfig{
262+
Endpoint: resolvedEndpoint,
263+
Bucket: objectStoreBucket,
264+
AccessKey: objectStoreAccess,
265+
SecretKey: objectStoreSecret,
266+
LocalRoot: objectStoreRoot,
267+
UseSSL: useSSL,
268+
PathStyle: true,
269+
}
270+
objectStoreInst, err = store.NewObjectTokenStore(objCfg)
271+
if err != nil {
272+
log.Fatalf("failed to initialize object token store: %v", err)
273+
}
274+
examplePath := filepath.Join(wd, "config.example.yaml")
275+
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
276+
if errBootstrap := objectStoreInst.Bootstrap(ctx, examplePath); errBootstrap != nil {
277+
cancel()
278+
log.Fatalf("failed to bootstrap object-backed config: %v", errBootstrap)
279+
}
280+
cancel()
281+
configFilePath = objectStoreInst.ConfigPath()
282+
cfg, err = config.LoadConfigOptional(configFilePath, isCloudDeploy)
283+
if err == nil {
284+
if cfg == nil {
285+
cfg = &config.Config{}
286+
}
287+
cfg.AuthDir = objectStoreInst.AuthDir()
288+
log.Infof("object-backed token store enabled, bucket: %s", objectStoreBucket)
289+
}
199290
} else if useGitStore {
200291
if gitStoreLocalPath == "" {
201292
gitStoreLocalPath = wd
@@ -294,6 +385,8 @@ func main() {
294385
// Register the shared token store once so all components use the same persistence backend.
295386
if usePostgresStore {
296387
sdkAuth.RegisterTokenStore(pgStoreInst)
388+
} else if useObjectStore {
389+
sdkAuth.RegisterTokenStore(objectStoreInst)
297390
} else if useGitStore {
298391
sdkAuth.RegisterTokenStore(gitStoreInst)
299392
} else {

‎docker-compose.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ services:
1010
COMMIT: ${COMMIT:-none}
1111
BUILD_DATE: ${BUILD_DATE:-unknown}
1212
container_name: cli-proxy-api
13+
# env_file:
14+
# - .env
1315
environment:
1416
DEPLOY: ${DEPLOY:-}
1517
ports:

‎go.mod‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,10 @@ require (
77
github.com/gin-gonic/gin v1.10.1
88
github.com/go-git/go-git/v6 v6.0.0-20251009132922-75a182125145
99
github.com/google/uuid v1.6.0
10+
github.com/joho/godotenv v1.5.1
1011
github.com/jackc/pgx/v5 v5.7.6
11-
github.com/klauspost/compress v1.17.3
12+
github.com/klauspost/compress v1.17.4
13+
github.com/minio/minio-go/v7 v7.0.66
1214
github.com/sirupsen/logrus v1.9.3
1315
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966
1416
github.com/tidwall/gjson v1.18.0
@@ -30,6 +32,7 @@ require (
3032
github.com/cloudwego/base64x v0.1.4 // indirect
3133
github.com/cloudwego/iasm v0.2.0 // indirect
3234
github.com/cyphar/filepath-securejoin v0.4.1 // indirect
35+
github.com/dustin/go-humanize v1.0.1 // indirect
3336
github.com/emirpasic/gods v1.18.1 // indirect
3437
github.com/gabriel-vasile/mimetype v1.4.3 // indirect
3538
github.com/gin-contrib/sse v0.1.0 // indirect
@@ -48,10 +51,13 @@ require (
4851
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
4952
github.com/leodido/go-urn v1.4.0 // indirect
5053
github.com/mattn/go-isatty v0.0.20 // indirect
54+
github.com/minio/md5-simd v1.1.2 // indirect
55+
github.com/minio/sha256-simd v1.0.1 // indirect
5156
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
5257
github.com/modern-go/reflect2 v1.0.2 // indirect
5358
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
5459
github.com/pjbgf/sha1cd v0.5.0 // indirect
60+
github.com/rs/xid v1.5.0 // indirect
5561
github.com/sergi/go-diff v1.4.0 // indirect
5662
github.com/tidwall/match v1.1.1 // indirect
5763
github.com/tidwall/pretty v1.2.0 // indirect
@@ -62,4 +68,5 @@ require (
6268
golang.org/x/sys v0.37.0 // indirect
6369
golang.org/x/text v0.30.0 // indirect
6470
google.golang.org/protobuf v1.34.1 // indirect
71+
gopkg.in/ini.v1 v1.67.0 // indirect
6572
)

0 commit comments

Comments
 (0)