Skip to content

Commit 15613db

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #7 from wk8/wk8/server_scaffolding
Adding the skeleton for versioned APIs, both servers and clients
2 parents 7f17ad2 + b9eac70 commit 15613db

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+3092
-0
lines changed

‎client/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Client go module
2+
3+
This separate go module is intended to be imported by clients that want to use the CSI-proxy.
4+
5+
It should strive to keep as few dependencies as possible, to make it easy to import in other repositories.

‎client/api/README.md‎

Lines changed: 251 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,251 @@
1+
# CSI-proxy's API
2+
3+
## Overview
4+
5+
CSI-proxy's API is a GRPC, versioned API.
6+
7+
The server exposes a number of API groups, all independent of each other. Additionally, each API group has one or more versions. Each version in each group listens for GRPC messages on a Windows named pipe of the form `\\.\\pipe\\csi-proxy-<api_group_name>-<version>` (e.g. `\\.\\pipe\\csi-proxy-filesystem-v2alpha1` or `\\.\\pipe\\csi-proxy-iscsi-v1`).
8+
9+
APIs are defined by protobuf files; each API group should live in its own directory under `client/api/<api_group_name>` in this repo's root (e.g. `client/api/iscsi`), and then define each of its version in `client/api/<api_group_name>/<version>/api.proto` files (e.g. `client/api/iscsi/v1/api.proto`). Each `proto` file should define exactly one RPC service.
10+
11+
Internally, there is only one server `struct` per API group, that handles all the versions for that API group. That server is defined in this repo's `internal/server/<api_group_name>` (e.g. `internal/server/iscsi`) go package. This go package should follow the following pattern:
12+
13+
<a name="serverPkgTree"></a>
14+
```
15+
internal/server/<api_group_name>
16+
├── internal
17+
│   └── types.go
18+
└── server.go
19+
```
20+
where `types.go` should contain the internal types corresponding to the various protobuf types for that API group - these internal structures must be able to represent all the different versions of the API. For example, given a `dummy` API group with two versions defined by the following `proto` files:
21+
22+
`client/api/dummy/v1alpha1/api.proto`
23+
```proto
24+
syntax = "proto3";
25+
26+
package v1alpha1;
27+
28+
service Dummy {
29+
// ComputeDouble computes the double of the input. Real smart stuff!
30+
rpc ComputeDouble(ComputeDoubleRequest) returns (ComputeDoubleResponse) {}
31+
}
32+
33+
message ComputeDoubleRequest{
34+
int32 input32 = 1;
35+
}
36+
37+
message ComputeDoubleResponse{
38+
int32 response32 = 1;
39+
}
40+
```
41+
42+
and
43+
44+
`client/api/dummy/v1/api.proto`
45+
```proto
46+
syntax = "proto3";
47+
48+
package v1;
49+
50+
service Dummy {
51+
// ComputeDouble computes the double of the input. Real smart stuff!
52+
rpc ComputeDouble(ComputeDoubleRequest) returns (ComputeDoubleResponse) {}
53+
}
54+
55+
message ComputeDoubleRequest{
56+
// we changed in favor of an int64 field here
57+
int64 input = 2;
58+
}
59+
60+
message ComputeDoubleResponse{
61+
int64 response = 2;
62+
63+
// set to true if the result overflowed
64+
bool overflow = 3;
65+
}
66+
```
67+
68+
then `internal/server/dummy/internal/types.go` could look something like:
69+
```go
70+
type ComputeDoubleRequest struct {
71+
Input int64
72+
}
73+
74+
type ComputeDoubleResponse struct {
75+
Response int64
76+
Overflow bool
77+
}
78+
```
79+
and then the API group's server (`internal/server/dummy/server.go`) needs to define the callbacks to handle requests for all API versions, e.g.:
80+
```go
81+
type Server struct{}
82+
83+
func (s *Server) ComputeDouble(ctx context.Context, request *internal.ComputeDoubleRequest, version apiversion.Version) (*internal.ComputeDoubleResponse, error) {
84+
in := request.Input64
85+
out := 2 * in
86+
87+
response := &internal.ComputeDoubleResponse{}
88+
89+
if sign(in) != sign(out) {
90+
// overflow
91+
response.Overflow = true
92+
} else {
93+
response.Response = out
94+
}
95+
96+
return response, nil
97+
}
98+
99+
func sign(x int64) int {
100+
switch {
101+
case x > 0:
102+
return 1
103+
case x < 0:
104+
return -1
105+
default:
106+
return 0
107+
}
108+
}
109+
```
110+
All the boilerplate code to:
111+
* add a named pipe to the server for each version of the API group, listening for each version's requests, and replying with each version's responses
112+
* convert versioned requests to internal representations
113+
* convert internal responses back to versioned responses
114+
* create clients to talk to the API group and its versions
115+
is generated automatically using [gengo](https://github.com/kubernetes/gengo).
116+
117+
The only caveat is that when conversions cannot be made trivially (e.g. when fields from internal and versioned `struct`s have different types), API devs need to define conversion functions. They can do that by creating an (otherwise optional) `internal/server/<api_group_name>/internal/<version>/conversion.go` file, containing functions of the form `func convert_pb_<Type>_To_internal_<Type>(in *pb.<Type>, out *internal.<Type>) error` or `func convert_internal_<Type>_To_pb_<Type>(in *internal.<Type>, out *pb.<Type>) error`; for example, in our `dummy` example above, we need to define a conversion function to account for the different fields in requests and responses from `v1alpha1` to `v1`; so `internal/server/dummy/internal/v1alpha1/conversion.go` could look like:
118+
```go
119+
func convert_pb_ComputeDoubleRequest_To_internal_ComputeDoubleRequest(in *pb.ComputeDoubleRequest, out *internal.ComputeDoubleRequest) error {
120+
out.Input64 = int64(in.Input32)
121+
return nil
122+
}
123+
124+
func convert_internal_ComputeDoubleResponse_To_pb_ComputeDoubleResponse(in *internal.ComputeDoubleResponse, out *pb.ComputeDoubleResponse) error {
125+
i := in.Response
126+
if i > math.MaxInt32 || i < math.MinInt32 {
127+
return fmt.Errorf("int32 overflow for %d", i)
128+
}
129+
out.Response32 = int32(i)
130+
return nil
131+
}
132+
```
133+
134+
## How to change the API
135+
136+
Existing API versions are immutable.
137+
138+
### How to add a new API group
139+
140+
Simply create a new `client/api/<api_group_name>/<version>/api.proto` file, defining your new service; then generate the Go protobuf code, and run the CSI-proxy generator to generate all the boilerplate code.
141+
142+
FIXME: add more details on which commands to run, and which files to edit when done generating.
143+
144+
### How to add a new version to an existing API group
145+
146+
Any changes to the API of an existing API group requires creating a new API version.
147+
148+
Steps to add a new API version:
149+
1. define it its own new `api.proto` file
150+
2. generate the Go protobuf code
151+
3. update the API group's internal representations (in its `types.go` file) to be able to represent all of the group's version (the new and the old ones)
152+
4. add any needed conversion functions for all existing versions of this API group to account for the changes made at the previous step
153+
5. re-generate all of the Go boilerplate code
154+
6. now you can change the API group's server to add your new feature!
155+
156+
### How to deprecate, and eventually remove...
157+
158+
From the CSI [proxy KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-windows/20190714-windows-csi-support.md#csi-proxy-grpc-api-graduation-and-deprecation-policy):
159+
160+
> In accordance with standard Kubernetes conventions, the above API will be introduced as v1alpha1 and graduate to v1beta1 and v1 as the feature graduates. Beyond a vN release in the future, new RPCs and enhancements to parameters will be introduced through vN+1alpha1 and graduate to vN+1beta1 and vN+1 stable versions as the new APIs mature.
161+
>
162+
> Members of CSIProxyService API may be deprecated and then removed from csi-proxy.exe in a manner similar to Kubernetes deprecation [policy](https://kubernetes.io/docs/reference/using-api/deprecation-policy/) although maintainers will make an effort to ensure such deprecation is as rare as possible. After their announced deprecation, a member of CSIProxyService API must be supported:
163+
>
164+
> 1. 12 months or 3 releases (whichever is longer) if the API member is part of a Stable/vN version.
165+
> 2. 9 months or 3 releases (whichever is longer) if the API member is part of a Beta/vNbeta1 version.
166+
> 3. 0 releases if the API member is part of an Alpha/vNalpha1 version.
167+
168+
With that in mind, each subsection below details how to deprecate, then remove:
169+
170+
#### A field in an API object
171+
172+
Mark it as deprecated in the `proto` file, e.g.:
173+
```proto
174+
message ComputeDoubleRequest{
175+
int32 input32 = 1 [deprecated=true];
176+
}
177+
```
178+
then regenerate the protobuf code.
179+
180+
For removal, simply remove it in the first API version that doesn't support that field any more.
181+
182+
#### An API procedure
183+
184+
Similarly, mark the procedure as deprecated in the protobuf definition, e.g.:
185+
```proto
186+
service Dummy {
187+
// ComputeDouble computes the double of the input. Real smart stuff!
188+
rpc ComputeDouble(ComputeDoubleRequest) returns (ComputeDoubleResponse) {
189+
option deprecated = true;
190+
}
191+
}
192+
```
193+
then regenerate the protobuf code, and remove it from the first API version that doesn't support that procedure any more.
194+
195+
#### An API version
196+
197+
Again, mark the version as deprecated in its protobuf definition, e.g.:
198+
```proto
199+
// Deprecated: Do not use.
200+
// v1alpha1 is no longer maintained, and will be removed soon.
201+
package v1alpha1;
202+
203+
service Dummy {
204+
...
205+
}
206+
```
207+
then regenerate the protobuf code, and run `csi-proxy-gen`: it will also mark the version's server (``) and client (``) packages as deprecated.
208+
209+
For removal, remove the whole `client/api/<api_group_name>/<version>` directory, and run `csi-proxy-gen`, it will remove all references to the removed version.
210+
211+
#### An API group
212+
213+
Deprecate and remove all its versions as explained in the previous version; then remove the entire `client/api/<api_group_name>` directory, and run `csi-proxy-gen`, it will remove all references to the removed API group.
214+
215+
## Detailed breakdown of generated files
216+
217+
This section details how `csi-proxy-gen` works, and what files it generates; `csi-proxy-gen` is built on top of [gengo](https://github.com/kubernetes/gengo), and re-uses part of [k8s' code-generator](https://github.com/kubernetes/code-generator), notably to generate conversion functions.
218+
219+
First, it looks for all API group definitions, which are either subdirectories of `client/api/`, or any go package that contains a `doc.go` file containing a `// +csi-proxy-gen` comment.
220+
221+
Then for each API group it finds:
222+
1. it iterates through each version subpackage, and in each looks for the `<ApiGroupName>Server` interface, and compiles the list of callbacks that the group's `Server` needs to implement as well as the list of top-level `struct`s (`*Request`s and `*Response`s)
223+
2. it looks for an existing `internal/server/<api_group_name>/internal/types.go` file:
224+
* if it exists, it checks that it contains all the expected top-level `struct`s from the previous step
225+
* if it doesn't exist, _and_ the API group only defines one version, it auto-generates one that simply copies the protobuf `struct`s (from the previous step) - this is meant to make it easy to bootstrap a new API group
226+
3. it generates the `internal/server/<api_group_name>/internal/types_generated.go` file, using the list of callbacks from the first step above
227+
4. if `internal/server/<api_group_name>/server.go` doesn't exist, it generates a skeleton for it - this, too, is meant to make it easy to bootstrap new API groups
228+
5. then for each version of the API:
229+
1. it looks for an existing `internal/server/<api_group_name>/internal/<version>/conversion.go`, generates an empty one if it doesn't exist; then looks for existing conversion functions
230+
2. it generates missing conversion functions to `internal/server/<api_group_name>/internal/<version>/conversion_generated.go`
231+
3. it generates `internal/server/<api_group_name>/internal/<version>/server_generated.go`
232+
6. it generates `internal/server/<api_group_name>/internal/api_group_generated.go` to list all the versioned servers it's just created
233+
7. and finally, it generates `client/groups/<api_group_name>/<version>/client_generated.go`
234+
235+
When `csi-proxy-gen` has successfully run to completion, [our example API group's go package from earlier](#serverPkgTree) will look something like:
236+
```
237+
internal/server/<api_group_name>
238+
├── api_group_generated.go
239+
├── internal
240+
│   ├── types.go
241+
│   ├── types_generated.go
242+
│   ├── v1
243+
│   │   ├── conversion.go
244+
│   │   ├── conversion_generated.go
245+
│   │   └── server_generated.go
246+
│   └── v1alpha1
247+
│      ├── conversion.go
248+
│      ├── conversion_generated.go
249+
│   └── server_generated.go
250+
└── server.go
251+
```

‎client/api/errors.pb.go‎

Lines changed: 98 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎client/api/errors.proto‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
syntax = "proto3";
2+
3+
package api;
4+
5+
// CommandError details errors yielded by cmdlet calls.
6+
message CmdletError {
7+
// Name of the cmdlet that errored out.
8+
string cmdlet_name = 1;
9+
10+
// Error code that got returned.
11+
uint32 code = 2;
12+
13+
// Human-readable error message - can be empty.
14+
string message = 3;
15+
}

0 commit comments

Comments
 (0)