Skip to content

Commit bfe63a7

Browse files
kesonanCopilot
authored and
kevwan
committed
feat(goctl/rpc): support external proto imports with cross-package type resolution
- Add -I/--proto_path support for importing proto files from external directories - Implement transitive dependency resolution for imported protos - Add cross-package type resolution with automatic Go import generation - Support Google well-known types (Empty, Timestamp, etc.) as RPC parameters - Remove restriction on dotted type names in RPC definitions - Add new generator/typeref.go for unified type reference resolution - Add new parser/import.go for transitive import parsing - Update call.tpl with extraImports placeholder for cross-package imports - Add 10 complete examples covering all generation scenarios with bilingual docs - Add test proto scenarios for external imports and cross-package types - Rewrite rpc README.md (EN) and add README-cn.md (CN) with full command reference - Add CHANGELOG.md and CHANGELOG-cn.md documenting all changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent c12c82b commit bfe63a7

File tree

93 files changed

+4863
-262
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

93 files changed

+4863
-262
lines changed

‎tools/goctl/internal/version/version.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ import (
66
)
77

88
// BuildVersion is the version of goctl.
9-
const BuildVersion = "1.10.0"
9+
const BuildVersion = "1.11.0"
1010

1111
var tag = map[string]int{"pre-alpha": 0, "alpha": 1, "pre-beta": 2, "beta": 3, "released": 4, "": 5}
1212

‎tools/goctl/rpc/CHANGELOG-cn.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# 变更日志
2+
3+
## 未发布
4+
5+
### 新功能
6+
7+
#### 外部 Proto 导入支持（`--proto_path` / `-I`）
8+
9+
新增通过 `-I` / `--proto_path` 标志导入外部目录中的 proto 文件，支持完整的传递性依赖解析。
10+
11+
**涉及文件：**
12+
- `generator/gen.go` — `ZRpcContext` 新增 `ProtoPaths` 字段；新增 `resolveImportedProtos()` 在代码生成前填充 `ImportedProtos`。
13+
- `generator/genpb.go` — 新增 `buildProtocCmd()` 自动发现并追加传递性导入的 proto 文件到 `protoc` 命令；新增 `relativeToProtoPath()` 计算正确的相对路径。
14+
- `parser/import.go` — 新文件（主要新增）。实现 `ResolveImports()` 递归解析传递性导入，`ParseImportedProtos()` 提取导入 proto 的 `go_package` / `package` 元数据，`BuildProtoPackageMap()` 构建按 proto 包名的 O(1) 查找表。
15+
- `parser/proto.go` — `Proto` 结构体新增 `ImportedProtos []ImportedProto` 字段。
16+
- `cli/cli.go` — `RPCNew` 传递 `ProtoPaths` 到 `ZRpcContext`。
17+
- `cli/zrpc.go` — 将 `VarStringSliceProtoPath` 传递到 `ZRpcContext.ProtoPaths`。
18+
19+
**前后对比：**
20+
21+
| | 变更前 | 变更后 |
22+
|---|---|---|
23+
| 从外部目录导入 Proto | ❌ 不支持，所有类型必须在同一文件中定义 | ✅ 使用 `-I ./ext_protos` 添加搜索路径 |
24+
| 传递性导入（A → B → C） | ❌ 仅识别直接导入 | ✅ 递归解析所有传递性依赖 |
25+
| 导入 proto 的 `.pb.go` 生成 | ❌ 需手动为每个文件单独运行 protoc | ✅ 自动将导入的 proto 追加到 protoc 命令 |
26+
| Proto 搜索路径 | ❌ 仅源文件所在目录 | ✅ 支持多个 `-I` 路径，与 protoc 一致 |
27+
28+
**行为说明：**
29+
- 递归遍历 proto 文件中的所有 `import` 声明，跳过 `google/*` 知名类型。
30+
- 在每个 `-I` 目录中搜索被导入的文件，未找到的系统级 proto 静默跳过。
31+
- 将发现的 proto 文件追加到 `protoc` 命令，使其 `.pb.go` 文件与主 proto 一同生成。
32+
33+
---
34+
35+
#### 跨包类型解析
36+
37+
当导入的 proto 与主 proto 具有**不同**的 `go_package` 时，goctl 现在能够自动在 server、logic 和 client 代码中生成正确的 Go 导入路径和限定类型引用。
38+
39+
**涉及文件：**
40+
- `generator/typeref.go` — 新文件，核心类型解析引擎：
41+
- `resolveRPCTypeRef()` — 将 proto RPC 类型（简单类型、同包点号类型、跨包点号类型、Google WKT）解析为带正确导入路径的 Go 类型引用。
42+
- `resolveCallTypeRef()` — 客户端代码生成变体，支持类型别名。
43+
- `googleWKTTable` — 全部 16 种 Google 知名类型到 Go 等价类型的映射表。
44+
- `generator/genserver.go` — `genFunctions()` 调用 `resolveRPCTypeRef()` 解析请求/响应类型并收集额外导入路径。
45+
- `generator/genlogic.go` — `genLogicFunction()` 使用 `resolveRPCTypeRef()`；新增 `addLogicImports()` 按需添加主 pb 导入和跨包导入。
46+
- `generator/gencall.go` — `genFunction()` 和 `getInterfaceFuncs()` 使用 `resolveCallTypeRef()` 处理类型别名和额外导入；新增 `buildExtraImportLines()` 辅助函数。
47+
- `generator/call.tpl` — 新增 `{{.extraImports}}` 占位符用于跨包导入行。
48+
49+
**前后对比：**
50+
51+
| Proto 类型 | 变更前 | 变更后 |
52+
|---|---|---|
53+
| `GetReq`（同文件） | `pb.GetReq` | `pb.GetReq`（无变化） |
54+
| `ext.ExtReq`（相同 `go_package`） | ❌ 报错："request type must defined in" | ✅ `pb.ExtReq` — 合并到主包 |
55+
| `common.TypesReq`（不同 `go_package`） | ❌ 报错："request type must defined in" | ✅ `common.TypesReq` + 自动生成 `import "example.com/demo/pb/common"` |
56+
| `google.protobuf.Empty` | ❌ 报错："request type must defined in" | ✅ `emptypb.Empty` + 自动生成导入 |
57+
58+
**行为说明：**
59+
- 简单类型（如 `GetReq`）解析为 `pb.GetReq`，无额外导入。
60+
- 同包点号类型（如 `ext.ExtReq`，其中 `ext` 与主 proto 有相同的 `go_package`）解析为 `pb.ExtReq`。
61+
- 跨包点号类型（如 `common.TypesReq`，其中 `common` 有不同的 `go_package`）解析为 `common.TypesReq`，并自动添加正确的 Go 导入路径。
62+
63+
---
64+
65+
#### Google 知名类型作为 RPC 参数
66+
67+
Google protobuf 知名类型现在可以直接用作 RPC 的请求/响应类型（而不仅仅是消息字段）。
68+
69+
**涉及文件：**
70+
- `generator/typeref.go` — `resolveGoogleWKT()` + `googleWKTTable` 处理所有标准类型。
71+
72+
**前后对比：**
73+
74+
| Proto 类型 | 变更前（作为 RPC 参数） | 变更后（作为 RPC 参数） |
75+
|---|---|---|
76+
| `google.protobuf.Empty` | ❌ 报错 | ✅ `emptypb.Empty` |
77+
| `google.protobuf.Timestamp` | ❌ 报错 | ✅ `timestamppb.Timestamp` |
78+
| `google.protobuf.Duration` | ❌ 报错 | ✅ `durationpb.Duration` |
79+
| `google.protobuf.Any` | ❌ 报错 | ✅ `anypb.Any` |
80+
| `google.protobuf.Struct` | ❌ 报错 | ✅ `structpb.Struct` |
81+
| `google.protobuf.FieldMask` | ❌ 报错 | ✅ `fieldmaskpb.FieldMask` |
82+
| `google.protobuf.*Value` | ❌ 报错 | ✅ `wrapperspb.*Value` |
83+
84+
> 注：这些类型此前已可用作**消息字段**。本次变更使其可直接用作 **RPC 请求/响应类型**。
85+
86+
**完整类型映射表：**
87+
88+
| Proto 类型 | Go 类型 |
89+
|---|---|
90+
| `google.protobuf.Empty` | `emptypb.Empty` |
91+
| `google.protobuf.Timestamp` | `timestamppb.Timestamp` |
92+
| `google.protobuf.Duration` | `durationpb.Duration` |
93+
| `google.protobuf.Any` | `anypb.Any` |
94+
| `google.protobuf.Struct` | `structpb.Struct` |
95+
| `google.protobuf.Value` | `structpb.Value` |
96+
| `google.protobuf.ListValue` | `structpb.ListValue` |
97+
| `google.protobuf.FieldMask` | `fieldmaskpb.FieldMask` |
98+
| `google.protobuf.*Value`（包装类型） | `wrapperspb.*Value` |
99+
100+
---
101+
102+
### 不兼容变更
103+
104+
#### RPC 定义中允许使用点号类型名
105+
106+
此前 goctl 会拒绝 RPC 请求/响应类型中包含点号的情况（如 `base.Req`），要求所有类型必须定义在同一个 proto 文件中。此限制已移除。
107+
108+
**前后对比：**
109+
110+
| Proto 定义 | 变更前 | 变更后 |
111+
|---|---|---|
112+
| `rpc Fetch(base.Req) returns (base.Reply)` | ❌ 解析错误："request type must defined in xxx.proto" | ✅ 解析成功，`base.Req` 通过导入的 proto 解析 |
113+
| `rpc Ping(google.protobuf.Empty) returns (Reply)` | ❌ 解析错误："request type must defined in xxx.proto" | ✅ 解析成功，解析为 `emptypb.Empty` |
114+
115+
**涉及文件：**
116+
- `parser/service.go` — 移除了拒绝点号类型名的验证循环（原错误信息为 `"request type must defined in"` / `"returns type must defined in"`）。
117+
- `parser/parser_test.go` — `TestDefaultProtoParseCaseInvalidRequestType` 和 `TestDefaultProtoParseCaseInvalidResponseType` 重命名并更新，验证点号类型现在可以正常解析。

‎tools/goctl/rpc/CHANGELOG.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# Changelog
2+
3+
## Unreleased
4+
5+
### New Features
6+
7+
#### External Proto Import Support (`--proto_path` / `-I`)
8+
9+
Added support for importing proto files from external directories via `-I` / `--proto_path` flags, with full transitive dependency resolution.
10+
11+
**Affected files:**
12+
- `generator/gen.go` — Added `ProtoPaths` field to `ZRpcContext`; added `resolveImportedProtos()` to populate `ImportedProtos` before code generation.
13+
- `generator/genpb.go` — Added `buildProtocCmd()` to automatically discover and append transitively imported proto files to the `protoc` command; added `relativeToProtoPath()` to compute correct relative paths for protoc output.
14+
- `parser/import.go` — New file (major addition). Implements `ResolveImports()` for recursive transitive import resolution, `ParseImportedProtos()` for extracting `go_package` / `package` metadata from imported protos, and `BuildProtoPackageMap()` for O(1) lookup by proto package name.
15+
- `parser/proto.go` — Added `ImportedProtos []ImportedProto` field to the `Proto` struct.
16+
- `cli/cli.go` — Passes `ProtoPaths` from `RPCNew` to `ZRpcContext`.
17+
- `cli/zrpc.go` — Passes `VarStringSliceProtoPath` to `ZRpcContext.ProtoPaths`.
18+
19+
**Before vs After:**
20+
21+
| | Before | After |
22+
|---|---|---|
23+
| Proto imports from external dirs | ❌ Not supported, all types must be in the same file | ✅ Use `-I ./ext_protos` to add search paths |
24+
| Transitive imports (A → B → C) | ❌ Only direct imports recognized | ✅ Recursively resolves all transitive dependencies |
25+
| Imported proto `.pb.go` generation | ❌ Manual, must run protoc separately for each file | ✅ Automatic, imported protos appended to protoc command |
26+
| Proto search paths | ❌ Only source file directory | ✅ Multiple `-I` paths, same as protoc |
27+
28+
**Behavior:**
29+
- Transitively walks all `import` declarations in proto files, skipping `google/*` well-known types.
30+
- Searches each `-I` directory for imported files, silently skipping system-level protos not found in user paths.
31+
- Appends discovered proto files to the `protoc` command so their `.pb.go` files are generated alongside the main proto.
32+
33+
---
34+
35+
#### Cross-Package Type Resolution
36+
37+
When an imported proto has a **different** `go_package` from the main proto, goctl now automatically generates the correct Go import paths and qualified type references in server, logic, and client code.
38+
39+
**Affected files:**
40+
- `generator/typeref.go` — New file. Core type resolution engine:
41+
- `resolveRPCTypeRef()` — Resolves proto RPC types (simple, same-package dotted, cross-package dotted, Google WKT) to Go type references with correct import paths.
42+
- `resolveCallTypeRef()` — Variant for client code generation with type alias support.
43+
- `googleWKTTable` — Mapping table for all 16 Google well-known types to their Go equivalents.
44+
- `generator/genserver.go` — `genFunctions()` now calls `resolveRPCTypeRef()` for request/response types and collects extra import paths.
45+
- `generator/genlogic.go` — `genLogicFunction()` uses `resolveRPCTypeRef()`; added `addLogicImports()` to conditionally include main pb import and cross-package imports.
46+
- `generator/gencall.go` — `genFunction()` and `getInterfaceFuncs()` use `resolveCallTypeRef()` for type aliases and extra imports; added `buildExtraImportLines()` helper.
47+
- `generator/call.tpl` — Added `{{.extraImports}}` placeholder for cross-package import lines.
48+
49+
**Before vs After:**
50+
51+
| Proto type | Before | After |
52+
|---|---|---|
53+
| `GetReq` (same file) | `pb.GetReq` | `pb.GetReq` (unchanged) |
54+
| `ext.ExtReq` (same `go_package`) | ❌ Error: "request type must defined in" | ✅ `pb.ExtReq` — merged into main package |
55+
| `common.TypesReq` (different `go_package`) | ❌ Error: "request type must defined in" | ✅ `common.TypesReq` + auto-generated `import "example.com/demo/pb/common"` |
56+
| `google.protobuf.Empty` | ❌ Error: "request type must defined in" | ✅ `emptypb.Empty` + auto-generated import |
57+
58+
**Behavior:**
59+
- Simple types (e.g., `GetReq`) resolve to `pb.GetReq` with no extra import.
60+
- Same-package dotted types (e.g., `ext.ExtReq` where `ext` has the same `go_package`) resolve to `pb.ExtReq`.
61+
- Cross-package dotted types (e.g., `common.TypesReq` where `common` has a different `go_package`) resolve to `common.TypesReq` with the correct Go import path added automatically.
62+
63+
---
64+
65+
#### Google Well-Known Types as RPC Parameters
66+
67+
Google protobuf well-known types can now be used directly as RPC request/response types (not just as message fields).
68+
69+
**Affected files:**
70+
- `generator/typeref.go` — `resolveGoogleWKT()` + `googleWKTTable` handles all standard types.
71+
72+
**Before vs After:**
73+
74+
| Proto Type | Before (as RPC param) | After (as RPC param) |
75+
|---|---|---|
76+
| `google.protobuf.Empty` | ❌ Error | ✅ `emptypb.Empty` |
77+
| `google.protobuf.Timestamp` | ❌ Error | ✅ `timestamppb.Timestamp` |
78+
| `google.protobuf.Duration` | ❌ Error | ✅ `durationpb.Duration` |
79+
| `google.protobuf.Any` | ❌ Error | ✅ `anypb.Any` |
80+
| `google.protobuf.Struct` | ❌ Error | ✅ `structpb.Struct` |
81+
| `google.protobuf.FieldMask` | ❌ Error | ✅ `fieldmaskpb.FieldMask` |
82+
| `google.protobuf.*Value` | ❌ Error | ✅ `wrapperspb.*Value` |
83+
84+
> Note: These types were already usable as **message fields** before. This change allows them as **RPC request/response types** directly.
85+
86+
**Supported types:**
87+
88+
| Proto Type | Go Type |
89+
|---|---|
90+
| `google.protobuf.Empty` | `emptypb.Empty` |
91+
| `google.protobuf.Timestamp` | `timestamppb.Timestamp` |
92+
| `google.protobuf.Duration` | `durationpb.Duration` |
93+
| `google.protobuf.Any` | `anypb.Any` |
94+
| `google.protobuf.Struct` | `structpb.Struct` |
95+
| `google.protobuf.Value` | `structpb.Value` |
96+
| `google.protobuf.ListValue` | `structpb.ListValue` |
97+
| `google.protobuf.FieldMask` | `fieldmaskpb.FieldMask` |
98+
| `google.protobuf.*Value` (wrappers) | `wrapperspb.*Value` |
99+
100+
---
101+
102+
### Breaking Changes
103+
104+
#### Dotted Type Names Now Allowed in RPC Definitions
105+
106+
Previously, goctl rejected any RPC request/response type containing a dot (e.g., `base.Req`), requiring all types to be defined in the same proto file. This restriction has been removed.
107+
108+
**Before vs After:**
109+
110+
| Proto Definition | Before | After |
111+
|---|---|---|
112+
| `rpc Fetch(base.Req) returns (base.Reply)` | ❌ Parse error: "request type must defined in xxx.proto" | ✅ Parsed successfully, `base.Req` resolved via imported proto |
113+
| `rpc Ping(google.protobuf.Empty) returns (Reply)` | ❌ Parse error: "request type must defined in xxx.proto" | ✅ Parsed successfully, resolved to `emptypb.Empty` |
114+
115+
**Affected files:**
116+
- `parser/service.go` — Removed the validation loop that rejected dotted type names with `"request type must defined in"` / `"returns type must defined in"` errors.
117+
- `parser/parser_test.go` — `TestDefaultProtoParseCaseInvalidRequestType` and `TestDefaultProtoParseCaseInvalidResponseType` renamed and updated to verify that dotted types now parse successfully.

0 commit comments

Comments
 (0)