Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions document/docs/02-administration/deployment/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ SkillHub 通过环境变量进行配置,主要配置项如下:
| `SKILLHUB_STORAGE_S3_BUCKET` | S3 桶名 | - |
| `SKILLHUB_STORAGE_S3_ACCESS_KEY` | S3 Access Key | - |
| `SKILLHUB_STORAGE_S3_SECRET_KEY` | S3 Secret Key | - |
| `SKILLHUB_STORAGE_S3_MAX_CONNECTIONS` | S3 HTTP 连接池大小 | `100` |
| `SKILLHUB_STORAGE_S3_CONNECTION_ACQUISITION_TIMEOUT` | 等待连接池连接超时,ISO-8601 Duration | `PT2S` |
| `SKILLHUB_STORAGE_S3_API_CALL_ATTEMPT_TIMEOUT` | 单次 S3 请求尝试超时,ISO-8601 Duration | `PT10S` |
| `SKILLHUB_STORAGE_S3_API_CALL_TIMEOUT` | 整个 S3 请求总超时,ISO-8601 Duration | `PT30S` |

### OAuth 配置

Expand All @@ -64,6 +68,13 @@ SkillHub 通过环境变量进行配置,主要配置项如下:

Spring Boot 配置文件位于 `server/skillhub-app/src/main/resources/`。

## 下载与对象存储建议

- 生产环境建议显式配置 S3 连接池和超时参数,避免对象存储抖动时大量线程阻塞在连接池等待。
- 版本列表接口现在使用数据库中的下载状态元数据,不再实时探测对象存储;部署时请确保数据库迁移已经执行。
- 下载接口按用户/IP + namespace + slug + version/tag 进行更细粒度限流。如有批量分发场景,建议在入口层或专用账号策略中单独放宽,不建议直接关闭应用层限流。

## 下一步

- [认证配置](../security/authentication) - 配置身份认证
- [下载与对象存储排查](../../05-reference/download-troubleshooting) - 排查下载、限流和对象存储故障
74 changes: 74 additions & 0 deletions document/docs/05-reference/download-troubleshooting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
---
title: 下载与对象存储排查
sidebar_position: 6
description: 排查下载接口、版本列表下载可用性和对象存储连接池问题
---

# 下载与对象存储排查

本页用于排查下载接口、版本列表中的下载可用性,以及对象存储连接池耗尽等问题。

## 当前行为

- 版本列表接口不再实时访问对象存储,而是读取数据库里的 `skill_version.download_ready`
- 下载接口优先返回预签名 URL;只有需要代理流式下载时才真正打开对象流
- 当 bundle 缺失时,服务会记录告警,并回退到逐文件重新打包

## 重点指标

- `skillhub.skill.download.delivery`
- 标签:`mode=redirect|stream`
- 标签:`fallback_bundle=true|false`
- `skillhub.skill.download.bundle_missing_fallback`
- bundle 缺失并触发逐文件回退打包的次数
- `skillhub.storage.failure`
- 标签:`operation=exists|getMetadata|getObject|putObject|deleteObject|deleteObjects|generatePresignedUrl`
- `skillhub.ratelimit.exceeded`
- 标签:`category=download|search|publish|...`

## 常见症状

### 下载返回 503

接口现在会把对象存储访问失败映射为 `503 error.storage.unavailable`,不再统一返回 500。

优先检查:

- `skillhub.storage.failure` 是否持续上升
- 对象存储端点、凭证、桶名是否正确
- `SKILLHUB_STORAGE_S3_MAX_CONNECTIONS`
- `SKILLHUB_STORAGE_S3_CONNECTION_ACQUISITION_TIMEOUT`
- `SKILLHUB_STORAGE_S3_API_CALL_ATTEMPT_TIMEOUT`
- `SKILLHUB_STORAGE_S3_API_CALL_TIMEOUT`

### 日志出现 bundle missing fallback

说明数据库里该版本仍被认为可下载,但对象存储中的 `packages/{skillId}/{versionId}/bundle.zip` 已缺失。

影响:

- 下载仍可通过逐文件打包完成
- 性能会明显差于直接下载 bundle

建议:

- 优先补齐缺失 bundle
- 观察 `skillhub.skill.download.bundle_missing_fallback`
- 检查对象存储生命周期规则、人工清理脚本或历史数据不一致问题

### 用户频繁点击下载后被限流

下载限流键现在包含:

- 用户或 IP
- namespace
- slug
- version 或 tag

这样同一用户反复下载同一版本时,不会把所有下载请求混进一个粗粒度桶里。

如果确实需要更高吞吐:

- 在入口层做白名单
- 为自动化分发账号单独放宽策略
- 不建议直接移除应用层下载限流
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import com.iflytek.skillhub.dto.SkillFileResponse;
import com.iflytek.skillhub.dto.SkillVersionDetailResponse;
import com.iflytek.skillhub.dto.SkillVersionResponse;
import com.iflytek.skillhub.metrics.SkillHubMetrics;
import com.iflytek.skillhub.ratelimit.RateLimit;
import org.springframework.core.io.InputStreamResource;
import org.springframework.data.domain.Page;
Expand All @@ -37,14 +38,17 @@ public class SkillController extends BaseApiController {

private final SkillQueryService skillQueryService;
private final SkillDownloadService skillDownloadService;
private final SkillHubMetrics metrics;

public SkillController(
SkillQueryService skillQueryService,
SkillDownloadService skillDownloadService,
SkillHubMetrics metrics,
ApiResponseFactory responseFactory) {
super(responseFactory);
this.skillQueryService = skillQueryService;
this.skillDownloadService = skillDownloadService;
this.metrics = metrics;
}

@GetMapping("/{namespace}/{slug}")
Expand Down Expand Up @@ -281,7 +285,7 @@ public ApiResponse<ResolveVersionResponse> resolveVersion(
}

@GetMapping("/{namespace}/{slug}/download")
@RateLimit(category = "download", authenticated = 120, anonymous = 30)
@RateLimit(category = "download", authenticated = 30, anonymous = 10)
public ResponseEntity<InputStreamResource> downloadLatest(
@PathVariable String namespace,
@PathVariable String slug,
Expand All @@ -296,7 +300,7 @@ public ResponseEntity<InputStreamResource> downloadLatest(
}

@GetMapping("/{namespace}/{slug}/versions/{version}/download")
@RateLimit(category = "download", authenticated = 120, anonymous = 30)
@RateLimit(category = "download", authenticated = 30, anonymous = 10)
public ResponseEntity<InputStreamResource> downloadVersion(
@PathVariable String namespace,
@PathVariable String slug,
Expand All @@ -312,7 +316,7 @@ public ResponseEntity<InputStreamResource> downloadVersion(
}

@GetMapping("/{namespace}/{slug}/tags/{tagName}/download")
@RateLimit(category = "download", authenticated = 120, anonymous = 30)
@RateLimit(category = "download", authenticated = 30, anonymous = 10)
public ResponseEntity<InputStreamResource> downloadByTag(
@PathVariable String namespace,
@PathVariable String slug,
Expand All @@ -329,16 +333,24 @@ public ResponseEntity<InputStreamResource> downloadByTag(

private ResponseEntity<InputStreamResource> buildDownloadResponse(HttpServletRequest request, SkillDownloadService.DownloadResult result) {
if (shouldRedirectToPresignedUrl(request, result.presignedUrl())) {
metrics.recordDownloadDelivery("redirect", result.fallbackBundle());
if (result.fallbackBundle()) {
metrics.incrementBundleMissingFallback();
}
return ResponseEntity.status(HttpStatus.FOUND)
.header(HttpHeaders.LOCATION, result.presignedUrl())
.build();
}

metrics.recordDownloadDelivery("stream", result.fallbackBundle());
if (result.fallbackBundle()) {
metrics.incrementBundleMissingFallback();
}
return ResponseEntity.ok()
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + result.filename() + "\"")
.contentType(MediaType.parseMediaType(result.contentType()))
.contentLength(result.contentLength())
.body(new InputStreamResource(result.content()));
.body(new InputStreamResource(result.openContent()));
}

private boolean shouldRedirectToPresignedUrl(HttpServletRequest request, String presignedUrl) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@
import com.iflytek.skillhub.domain.shared.exception.DomainBadRequestException;
import com.iflytek.skillhub.domain.shared.exception.DomainForbiddenException;
import com.iflytek.skillhub.domain.shared.exception.DomainNotFoundException;
import com.iflytek.skillhub.metrics.SkillHubMetrics;
import com.iflytek.skillhub.security.SensitiveLogSanitizer;
import com.iflytek.skillhub.storage.StorageAccessException;
import jakarta.servlet.http.HttpServletRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand All @@ -27,11 +29,14 @@ public class GlobalExceptionHandler {
private static final Logger logger = LoggerFactory.getLogger(GlobalExceptionHandler.class);
private final ApiResponseFactory apiResponseFactory;
private final SensitiveLogSanitizer sensitiveLogSanitizer;
private final SkillHubMetrics metrics;

public GlobalExceptionHandler(ApiResponseFactory apiResponseFactory,
SensitiveLogSanitizer sensitiveLogSanitizer) {
SensitiveLogSanitizer sensitiveLogSanitizer,
SkillHubMetrics metrics) {
this.apiResponseFactory = apiResponseFactory;
this.sensitiveLogSanitizer = sensitiveLogSanitizer;
this.metrics = metrics;
}

@ExceptionHandler(LocalizedException.class)
Expand Down Expand Up @@ -108,6 +113,23 @@ public ResponseEntity<ApiResponse<Void>> handleAccessDenied(AccessDeniedExceptio
apiResponseFactory.error(403, "error.forbidden"));
}

@ExceptionHandler(StorageAccessException.class)
public ResponseEntity<ApiResponse<Void>> handleStorageAccess(StorageAccessException ex, HttpServletRequest request) {
metrics.incrementStorageAccessFailure(ex.getOperation());
logger.warn(
"Object storage unavailable [requestId={}, method={}, path={}, userId={}, operation={}, key={}]",
MDC.get("requestId"),
request.getMethod(),
sensitiveLogSanitizer.sanitizeRequestTarget(request),
resolveUserId(request),
ex.getOperation(),
ex.getKey(),
ex
);
return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).body(
apiResponseFactory.error(503, "error.storage.unavailable"));
}

@ExceptionHandler(Exception.class)
public ResponseEntity<ApiResponse<Void>> handleGlobalException(Exception ex, HttpServletRequest request) {
logger.error(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,30 @@ public void incrementSkillPublish(String namespace, String status) {
"status", status
).increment();
}

public void recordDownloadDelivery(String mode, boolean fallbackBundle) {
meterRegistry.counter(
"skillhub.skill.download.delivery",
"mode", mode,
"fallback_bundle", Boolean.toString(fallbackBundle)
).increment();
}

public void incrementBundleMissingFallback() {
meterRegistry.counter("skillhub.skill.download.bundle_missing_fallback").increment();
}

public void incrementRateLimitExceeded(String category) {
meterRegistry.counter(
"skillhub.ratelimit.exceeded",
"category", category
).increment();
}

public void incrementStorageAccessFailure(String operation) {
meterRegistry.counter(
"skillhub.storage.failure",
"operation", operation
).increment();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,27 +3,34 @@
import com.fasterxml.jackson.databind.ObjectMapper;
import com.iflytek.skillhub.dto.ApiResponse;
import com.iflytek.skillhub.dto.ApiResponseFactory;
import com.iflytek.skillhub.metrics.SkillHubMetrics;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerMapping;
import org.springframework.web.method.HandlerMethod;
import org.springframework.web.servlet.HandlerInterceptor;

import java.util.Map;

@Component
public class RateLimitInterceptor implements HandlerInterceptor {

private final RateLimiter rateLimiter;
private final ApiResponseFactory apiResponseFactory;
private final ObjectMapper objectMapper;
private final SkillHubMetrics metrics;

public RateLimitInterceptor(RateLimiter rateLimiter,
ApiResponseFactory apiResponseFactory,
ObjectMapper objectMapper) {
ObjectMapper objectMapper,
SkillHubMetrics metrics) {
this.rateLimiter = rateLimiter;
this.apiResponseFactory = apiResponseFactory;
this.objectMapper = objectMapper;
this.metrics = metrics;
}

@Override
Expand All @@ -48,12 +55,13 @@ public boolean preHandle(HttpServletRequest request, HttpServletResponse respons

// Build rate limit key
String identifier = isAuthenticated ? "user:" + userId : "ip:" + getClientIp(request);
String key = "ratelimit:" + rateLimit.category() + ":" + identifier;
String key = "ratelimit:" + rateLimit.category() + ":" + identifier + resolveResourceSuffix(rateLimit.category(), request);

// Check rate limit
boolean allowed = rateLimiter.tryAcquire(key, limit, rateLimit.windowSeconds());

if (!allowed) {
metrics.incrementRateLimitExceeded(rateLimit.category());
response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
response.setContentType(MediaType.APPLICATION_JSON_VALUE);
ApiResponse<Void> body = apiResponseFactory.error(429, "error.rateLimit.exceeded");
Expand All @@ -78,4 +86,32 @@ private String getClientIp(HttpServletRequest request) {
}
return ip;
}

@SuppressWarnings("unchecked")
private String resolveResourceSuffix(String category, HttpServletRequest request) {
if (!"download".equals(category)) {
return "";
}
Object attribute = request.getAttribute(HandlerMapping.URI_TEMPLATE_VARIABLES_ATTRIBUTE);
if (!(attribute instanceof Map<?, ?> templateVariables)) {
return "";
}
String namespace = stringValue(templateVariables.get("namespace"));
String slug = stringValue(templateVariables.get("slug"));
String version = stringValue(templateVariables.get("version"));
String tagName = stringValue(templateVariables.get("tagName"));
if (namespace == null || slug == null) {
return "";
}
String target = version != null ? "version:" + version : tagName != null ? "tag:" + tagName : "latest";
return ":ns:" + namespace + ":slug:" + slug + ":" + target;
}

private String stringValue(Object value) {
if (value == null) {
return null;
}
String text = value.toString();
return text.isBlank() ? null : text;
}
}
4 changes: 4 additions & 0 deletions server/skillhub-app/src/main/resources/application.yml
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,10 @@ skillhub:
force-path-style: ${SKILLHUB_STORAGE_S3_FORCE_PATH_STYLE:true}
auto-create-bucket: ${SKILLHUB_STORAGE_S3_AUTO_CREATE_BUCKET:false}
presign-expiry: ${SKILLHUB_STORAGE_S3_PRESIGN_EXPIRY:PT10M}
max-connections: ${SKILLHUB_STORAGE_S3_MAX_CONNECTIONS:100}
connection-acquisition-timeout: ${SKILLHUB_STORAGE_S3_CONNECTION_ACQUISITION_TIMEOUT:PT2S}
api-call-attempt-timeout: ${SKILLHUB_STORAGE_S3_API_CALL_ATTEMPT_TIMEOUT:PT10S}
api-call-timeout: ${SKILLHUB_STORAGE_S3_API_CALL_TIMEOUT:PT30S}
search:
engine: postgres
rebuild-on-startup: false
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
ALTER TABLE skill_version
ADD COLUMN bundle_ready BOOLEAN NOT NULL DEFAULT FALSE,
ADD COLUMN download_ready BOOLEAN NOT NULL DEFAULT FALSE;

UPDATE skill_version
SET download_ready = CASE
WHEN status = 'PUBLISHED' AND file_count > 0 THEN TRUE
ELSE FALSE
END,
bundle_ready = FALSE;
1 change: 1 addition & 0 deletions server/skillhub-app/src/main/resources/messages.properties
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ error.badRequest=Invalid request
error.forbidden=Forbidden
error.rateLimit.exceeded=Rate limit exceeded
error.internal=An unexpected error occurred
error.storage.unavailable=Object storage is temporarily unavailable
error.slug.blank=Slug cannot be blank
error.slug.length=Slug length must be between {0} and {1} characters
error.slug.pattern=Slug must contain only lowercase letters, numbers, and hyphens, and must start and end with a letter or number
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ error.badRequest=请求参数不合法
error.forbidden=没有权限执行该操作
error.rateLimit.exceeded=请求过于频繁,请稍后再试
error.internal=服务器内部错误
error.storage.unavailable=对象存储暂时不可用,请稍后再试
error.slug.blank=slug 不能为空
error.slug.length=slug 长度必须在 {0} 到 {1} 个字符之间
error.slug.pattern=slug 只能包含小写字母、数字和连字符,且必须以字母或数字开头和结尾
Expand Down
Loading
Loading