Skip to content

Commit 5fcb3a0

Browse files
yaxifesimon-mo
yaxife
and
simon-mo
authored
[serve] Java HTTP Ingress based on JAX-RS (#15)
* Create 2022-10-11-serve-java-http-ingress.md Signed-off-by: xiaofeng <[email protected]> * Update 2022-10-11-serve-java-http-ingress.md Signed-off-by: xiaofeng <[email protected]> * Update 2022-10-11-serve-java-http-ingress.md Signed-off-by: xiaofeng <[email protected]> * fix review issues Signed-off-by: xiaofeng <[email protected]> * Update 2022-10-11-serve-java-http-ingress.md Signed-off-by: xiaofeng <[email protected]> * Update reps/2022-10-11-serve-java-http-ingress.md Signed-off-by: Simon Mo <[email protected]> * edit pass Signed-off-by: simon-mo <[email protected]> Signed-off-by: xiaofeng <[email protected]> Signed-off-by: Simon Mo <[email protected]> Signed-off-by: simon-mo <[email protected]> Co-authored-by: Simon Mo <[email protected]>
1 parent 5e9ea92 commit 5fcb3a0

File tree

1 file changed

+197
-0
lines changed

1 file changed

+197
-0
lines changed

‎reps/2022-10-11-serve-java-http-ingress.md

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
## Summary
2+
This enhancement proposal recommends adding an ingress feature to facilitate HTTP handling to the Ray Serve package in Java.
3+
4+
### General Motivation
5+
We want to call various methods within a Ray Serve Java deployment through HTTP proxy, in a way that's similar to the [FastAPI ingress](https://docs.ray.io/en/latest/serve/http-guide.html#fastapi-http-deployments) in Python version.
6+
7+
### Should this change be within `ray` or outside?
8+
The main `ray` project. Changes are made to the Ray Serve module in the Java codebase.
9+
10+
## Stewardship
11+
12+
### Required Reviewers
13+
@simon-mo @sihanwang41
14+
15+
### Shepherd of the Proposal (should be a senior committer)
16+
@simon-mo
17+
18+
## Design and Architecture
19+
20+
### User Workflow
21+
22+
1. Define the model with [JAX-RS API](https://docs.oracle.com/javaee/6/tutorial/doc/gilik.html)
23+
24+
```java
25+
@Path("user")
26+
public class UserRestService {
27+
@GET
28+
@Path("helloWorld")
29+
@Consumes({"application/json"})
30+
@Produces({"application/json"})
31+
public String helloWorld(String name) {
32+
return "hello world, " + name;
33+
}
34+
35+
@GET
36+
@Path("paramPathTest/{name}")
37+
@Consumes({"application/json"})
38+
@Produces({"application/json"})
39+
public String paramPathTest(@PathParam("name")String name) {
40+
return "paramPathTest, " + name;
41+
}
42+
}
43+
```
44+
2. Create deployment
45+
```java
46+
Deployment deployment =
47+
Serve.deployment()
48+
.setName("deploymentName")
49+
.setDeploymentDef(UserRestService.class.getName())
50+
.ingress("jax-rs")
51+
.setNumReplicas(1)
52+
.create();
53+
deployment.deploy();
54+
```
55+
56+
3. Calling deployments via HTTP
57+
58+
The URI is determined by the deployment name and JAX-RS @PATH annotations.
59+
```java
60+
curl http://127.0.0.1:8000/deploymentName/user/helloWorld?name=test
61+
curl http://127.0.0.1:8000/deploymentName/user/paramPathTest/test
62+
```
63+
64+
### HTTP Ingress Implementation
65+
#### Ingress API Annotation: JAX-RS
66+
67+
In Java application development, restful API is generally implemented through two sets of annotations, spring web or JAX-RS.
68+
69+
JAX-RS is a specification for implementing REST web services in Java, currently defined by the JSR-370. JAX-RS is just an API specification and has been implemented by many components: jersey, resteasy, etc.
70+
71+
Spring-web contains a lot of features that we don't need to use: IOC (inversion of control), DI (dependency injection), etc. The parsing of spring-web annotations depends on the spring framework.
72+
73+
In order not to import too many unnecessary dependencies to the user's `Callable`. We choose JAX-RS to support HTTP ingress.
74+
75+
For more information about JAX-RS and spring web, please click the following links.
76+
77+
JAX-RS: https://projects.eclipse.org/projects/ee4j.rest
78+
79+
Spring-web: https://docs.spring.io/spring-framework/docs/current/reference/html/web.html#mvc-ann-requestmapping
80+
81+
#### Annotation parser: Jersey
82+
Jersey RESTful Web Services 2.x framework is open source, production quality, a framework for developing RESTful Web Services in Java that provides support for JAX-RS APIs and serves as a JAX-RS (JSR 311 & JSR 339 & JSR 370) Reference Implementation.
83+
84+
Jersey framework is more than the JAX-RS Reference Implementation. Jersey provides its own API that extends the JAX-RS toolkit with additional features and utilities to further simplify RESTful service and client development. Jersey also exposes numerous extension SPIs so that developers may extend Jersey to best suit their needs.
85+
86+
The most important thing is that the Jersey does not depend on a servlet container to run. And many other open-source frameworks need to integrate servlet containers.
87+
88+
For more information about Jersey, please click this link: https://eclipse-ee4j.github.io/jersey/
89+
90+
##### Maven dependency
91+
92+
```xml
93+
<dependency>
94+
<groupId>org.glassfish.jersey.core</groupId>
95+
<artifactId>jersey-server</artifactId>
96+
<version>2.30.1</version>
97+
</dependency>
98+
<dependency>
99+
<groupId>org.glassfish.jersey.inject</groupId>
100+
<artifactId>jersey-hk2</artifactId>
101+
<version>2.30.1</version>
102+
</dependency>
103+
```
104+
##### Jersey Callable Integration
105+
In order to integrate Jersey in the the current `ray.serve` package, we will need to take the following step:
106+
107+
- convert `RequestWrapper` to jersey `ContainerRequest`
108+
- call `ApplicationHandler.apply` with `ContainerRequest`
109+
- return `ContainerResponse.getEntity` to the HTTP proxy
110+
111+
```java
112+
public class JaxrsIngressCallable {
113+
private ApplicationHandler app;
114+
115+
public JaxrsIngressCallable(Class clazz) {
116+
ResourceConfig resourceConfig = new ResourceConfig(clazz);
117+
this.app = new ApplicationHandler(resourceConfig);
118+
}
119+
120+
public Object call(RequestWrapper httpProxyRequest) {
121+
ContainerRequest jerseyRequest = convertRequestWrap2ContainerRequest(httpProxyRequest)
122+
ContainerResponse response = app.apply(jerseyRequest).get();
123+
Object rspBody = response.getEntity();
124+
return rspBody;
125+
}
126+
}
127+
```
128+
129+
### Extension of `callable`
130+
Normally, we use the class instance as a `ray.serve.Callable`. But in HTTP ingress, the `ray.serve.Callable` we use is wrapped with the Jersey application handler. We have two different implementations of `ray.serve.Callable`.
131+
132+
In python, when we add the `@ingress` annotation to an object, a new object will be generated, that is, a new `ray.serve.Callable` instance will be generated.
133+
134+
In java, we add annotations to the class, but it can not enhance the features of `ray.serve.Callable`. So we need a mechanism to generate different `ray.serve.Callable` instances according to different ingress types.
135+
136+
Here we use java SPI to implement this feature. We add a `ServeCallableProvider` SPI.
137+
138+
```java
139+
public interface ServeCallableProvider {
140+
/**
141+
* Get Callable type
142+
* @return Callable type
143+
*/
144+
String type();
145+
146+
/**
147+
* Generate a Callable instance
148+
* @param deploymentWrapper deployment info and config
149+
* @return Callable instance
150+
*/
151+
Object buildCallable(DeploymentWrapper deploymentWrapper);
152+
153+
/**
154+
* Get the signature of callable
155+
* @param deploymentWrapper deployment info and config
156+
* @param callable Callable instance
157+
* @return
158+
*/
159+
Map<String, Pair<Method, Object>> getSignatures(DeploymentWrapper deploymentWrapper, Object callable);
160+
}
161+
```
162+
163+
The current version of `Callable` is implemented as the default interface implementation. An additional implementation of the `ServeCallableProvider` is added for each additional ingress type.
164+
165+
### Method Signature Cache
166+
Typically, our `Callable` are user-provided class instances. In JAX-RS, the `Callable` is the jersey application handler when the replica is called using an HTTP proxy. When using the serve handle to call the replica, we cannot call any methods in the class. We are not compatible with the serve handle.
167+
168+
On the other hand, every time we call replica, we need to use reflection to get the method that needs to be executed. This will reduce the performance and throughput of the replica
169+
170+
In order to solve the above problems, We need to hold the cache of method signatures.
171+
172+
![image](https://user-images.githubusercontent.com/11265783/195356752-95cf595b-f235-477c-b041-47b3760ad0f5.png)
173+
174+
Ray core uses the signature to decide which method to call. we want to be consistent with it.
175+
176+
When init java replica, we will parse signatures from the `Callable` class. Generate the following map. the key is the method signature, value is the pair of the method instance and the `Callable` instance.
177+
178+
```java
179+
Map<String, Pair<Method, Object>> signatures;
180+
```
181+
182+
If the user configures JAX-rs ingress, we will add one data to the signature cache. The key is always set to `__call__`, the left of the pair in the value is a `Method` instance of the `JaxrsIngressCallable.call` and the right of the pair is an instance of `JaxrsIngressCallable`.
183+
184+
For requests from HTTP proxy, the method signature will be fixed to `__call__`. The request from the serve handle will set the method signature on the client side and hit the signature cache on the server side.
185+
186+
## Compatibility, Deprecation, and Migration Plan
187+
New features are incremental and do not affect any existing features. And we use SPI to make the modification of the java HTTP ingress meet the open-closed principle.
188+
189+
## Test Plan and Acceptance Criteria
190+
- Unit and integration test for core components
191+
- Benchmarks on java HTTP ingress performance
192+
193+
## (Optional) Follow-on Work
194+
- `callable` support SPI
195+
- method signature cache
196+
- http ingress with jersey application handler
197+
- benchmark test

0 commit comments

Comments
 (0)