Skip to content

Commit 3ef8cc7

Browse files
feat: Migrate from Restana to Express, add large data support and misc changes (#381)
* Add initial implementation, tested already for HTTP not for GRPC yet Signed-off-by: Xavier Geerinck <[email protected]> * Add gRPC test for max payload exceeding Signed-off-by: Xavier Geerinck <[email protected]> * Resolve linter issues Signed-off-by: Xavier Geerinck <[email protected]> * WIP on tests Signed-off-by: Xavier Geerinck <[email protected]> * Test passes Signed-off-by: Xavier Geerinck <[email protected]> * Fix unused Signed-off-by: Xavier Geerinck <[email protected]> * Rename bodySizeMb to maxBodySizeMb Signed-off-by: Xavier Geerinck <[email protected]> * Pretty fix Signed-off-by: Xavier Geerinck <[email protected]> * Improve serializer tests Signed-off-by: Xavier Geerinck <[email protected]> * Add resolution Signed-off-by: Xavier Geerinck <[email protected]> * Pretty fix Signed-off-by: Xavier Geerinck <[email protected]> * Work on HTTP Test to make it pass Signed-off-by: Xavier Geerinck <[email protected]> * Remove unused var Signed-off-by: Xavier Geerinck <[email protected]> * Prettier Signed-off-by: Xavier Geerinck <[email protected]> * Add runtime 1.8.4 to check if tests run now as they finish locally Signed-off-by: Xavier Geerinck <[email protected]> * Add max session to gRPC client Signed-off-by: Shubham Sharma <[email protected]> * Add documentation Signed-off-by: Xavier Geerinck <[email protected]> * Remove old code Signed-off-by: Xavier Geerinck <[email protected]> * Style fix Signed-off-by: Xavier Geerinck <[email protected]> * Resolve binding test Signed-off-by: Xavier Geerinck <[email protected]> * Pretty fix Signed-off-by: Xavier Geerinck <[email protected]> * Remove unused functions Signed-off-by: Xavier Geerinck <[email protected]> * Improve serializer Signed-off-by: Xavier Geerinck <[email protected]> * Remove pnpm lock Signed-off-by: Xavier Geerinck <[email protected]> * PRetty fix Signed-off-by: Xavier Geerinck <[email protected]> * Changes for removing Restana Signed-off-by: Xavier Geerinck <[email protected]> * Remove uuid package Signed-off-by: Xavier Geerinck <[email protected]> * Satisfy the linter Signed-off-by: Xavier Geerinck <[email protected]> * Ignore types Signed-off-by: Xavier Geerinck <[email protected]> * Ignore types Signed-off-by: Xavier Geerinck <[email protected]> * Ignore types Signed-off-by: Xavier Geerinck <[email protected]> * Ignore types Signed-off-by: Xavier Geerinck <[email protected]> * Add an example Signed-off-by: Xavier Geerinck <[email protected]> * Add typings for actors Signed-off-by: Xavier Geerinck <[email protected]> * Use Dapr 1.9.5 Signed-off-by: Xavier Geerinck <[email protected]> * Remove as we don't need it anymore since we close it automatically Signed-off-by: Xavier Geerinck <[email protected]> * Stabilize test more Signed-off-by: Xavier Geerinck <[email protected]> * Correct gRPC tests Signed-off-by: Xavier Geerinck <[email protected]> * Forgot this one Signed-off-by: Xavier Geerinck <[email protected]> * In gRPC strings are serialized as Buffer Signed-off-by: Xavier Geerinck <[email protected]> * Fix hanging tests Signed-off-by: Xavier Geerinck <[email protected]> * Update CHANGELOG.md Co-authored-by: Shubham Sharma <[email protected]> Signed-off-by: Xavier Geerinck <[email protected]> * Update daprdocs/content/en/js-sdk-docs/js-client/_index.md Co-authored-by: Shubham Sharma <[email protected]> Signed-off-by: Xavier Geerinck <[email protected]> * Mention the default body size Signed-off-by: Xavier Geerinck <[email protected]> * generateChannelOptions is private Signed-off-by: Xavier Geerinck <[email protected]> * Improve serializer Signed-off-by: Xavier Geerinck <[email protected]> * Improve serializer Signed-off-by: Xavier Geerinck <[email protected]> * Pretty fix Signed-off-by: Xavier Geerinck <[email protected]> * Add possibility to add own express server Signed-off-by: Xavier Geerinck <[email protected]> * Pretty fix Signed-off-by: Xavier Geerinck <[email protected]> * Add HTTP Serializer tests Signed-off-by: Xavier Geerinck <[email protected]> * Add extra tests for body inflation prevention and fix serialization of octet-stream data Signed-off-by: Xavier Geerinck <[email protected]> * Remove wrong change Signed-off-by: Xavier Geerinck <[email protected]> * Remove debug logging Signed-off-by: Xavier Geerinck <[email protected]> * Remove debug logging Signed-off-by: Xavier Geerinck <[email protected]> * Reword docs Signed-off-by: Xavier Geerinck <[email protected]> * Add changes from review Signed-off-by: Xavier Geerinck <[email protected]> * Add boolean and number type check Signed-off-by: Xavier Geerinck <[email protected]> * Import randomUUID directly Signed-off-by: Xavier Geerinck <[email protected]> * Fix missed vars Signed-off-by: Xavier Geerinck <[email protected]> * Added body inflation test for gRPC Signed-off-by: Xavier Geerinck <[email protected]> * Fix deprecation warning Signed-off-by: Xavier Geerinck <[email protected]> * Trying to resolve open handlers in actors Signed-off-by: Xavier Geerinck <[email protected]> * Pretty fix Signed-off-by: Xavier Geerinck <[email protected]> * Actor check Signed-off-by: Xavier Geerinck <[email protected]> * Actor reminder updates Signed-off-by: Xavier Geerinck <[email protected]> Signed-off-by: Xavier Geerinck <[email protected]> Signed-off-by: Shubham Sharma <[email protected]> Co-authored-by: Shubham Sharma <[email protected]>
1 parent 276bb0e commit 3ef8cc7

34 files changed

+650
-207
lines changed

.github/workflows/test-e2e.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ jobs:
3636
env:
3737
GOVER: 1.19
3838
DAPR_CLI_VER: 1.9.1
39-
DAPR_RUNTIME_VER: 1.9.0
39+
DAPR_RUNTIME_VER: 1.9.5
4040
DAPR_INSTALL_URL: https://raw.githubusercontent.com/dapr/cli/master/install/install.sh
4141
DAPR_CLI_REF: ""
4242
DAPR_REF: ""

CHANGELOG.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
# Changelog
22

3+
## 3.x release
4+
5+
### v3.0.0
6+
7+
#### Breaking Changes
8+
9+
##### GENERAL: Serialization changed and data is correctly being serialized now
10+
11+
Previously when sending data, it could happen that it was incorrectly serialized. This has been resolved and a new serializer is in place that will automatically detect the Content-Type for the data when using the HTTP Protocol and serialize accordingly. Objects will be send as `application/json`, Cloud-Events as `applications/cloudevents+json`, Strings as `text/plain`, and others as `application/octet-stream`.
12+
13+
As an example, when performing `client.invoke` with `"hello world"` as data instead of an object, the `JSON.serialize` method would be called and we would receive `'"hello world"'` (notice the `'`). Now due to the new serializer we will correctly return the string type.
14+
315
## 2.x release
416

517
### v2.0.1

daprdocs/content/en/js-sdk-docs/js-client/_index.md

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -77,11 +77,25 @@ dapr run --app-id example-sdk --app-protocol grpc -- npm run start
7777
npm run start:dapr-grpc
7878
```
7979

80-
## Proxying Requests
80+
## General
81+
82+
### Increasing Body Size
83+
84+
You can increase the body size that is used by the application to communicate with the sidecar by using a`DaprClient`'s option.
85+
86+
```typescript
87+
import { DaprClient, CommunicationProtocol } from "@dapr/dapr";
88+
89+
// Allow a body size of 10Mb to be used
90+
// The default is 4Mb
91+
const client = new DaprClient(daprHost, daprPort, CommunicationProtocol.HTTP, { maxBodySizeMb: 10 });
92+
```
93+
94+
### Proxying Requests
8195

8296
By proxying requests, we can utilize the unique capabilities that Dapr brings with its sidecar architecture such as service discovery, logging, etc., enabling us to instantly "upgrade" our gRPC services. This feature of gRPC proxying was demonstrated in [community call 41](https://www.youtube.com/watch?v=B_vkXqptpXY&t=71s).
8397

84-
### Creating a Proxy
98+
#### Creating a Proxy
8599

86100
To perform gRPC proxying, simply create a proxy by calling the `client.proxy.create()` method:
87101

@@ -96,7 +110,7 @@ const clientProxy = await clientSidecar.proxy.create<GreeterClient>(GreeterClien
96110

97111
We can now call the methods as defined in our `GreeterClient` interface (which in this case is from the [Hello World example](https://github.com/grpc/grpc-go/blob/master/examples/helloworld/helloworld/helloworld.proto))
98112

99-
### Behind the Scenes (Technical Working)
113+
#### Behind the Scenes (Technical Working)
100114

101115
![Architecture](assets/architecture.png)
102116

daprdocs/content/en/js-sdk-docs/js-server/_index.md

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ const server = new DaprServer(serverHost, serverPort, daprHost, daprPort, Commun
4545

4646
To run the examples, you can use two different protocols to interact with the Dapr sidecar: HTTP (default) or gRPC.
4747

48-
### Using HTTP (default)
48+
### Using HTTP (built-in express webserver)
4949

5050
```typescript
5151
import { DaprServer } from "@dapr/dapr";
@@ -66,6 +66,48 @@ npm run start:dapr-http
6666

6767
> ℹ️ **Note:** The `app-port` is required here, as this is where our server will need to bind to. Dapr will check for the application to bind to this port, before finishing start-up.
6868
69+
### Using HTTP (bring your own express webserver)
70+
71+
Instead of using the built-in web server for Dapr sidecar to application communication, you can also bring your own instance. This is helpful in scenarios like when you are building a REST API back-end and want to integrate Dapr directly in it.
72+
73+
Note, this is currently available for [`express`](https://www.npmjs.com/package/express) only.
74+
75+
> 💡 Note: when using a custom web-server, the SDK will configure server properties like max body size, and add new routes to it. The routes are unique on their own to avoid any collisions with your application, but it's not guaranteed to not collide.
76+
77+
```typescript
78+
import { DaprServer, CommunicationProtocolEnum } from "@dapr/dapr";
79+
import express from "express";
80+
81+
const myApp = express();
82+
83+
myApp.get("/my-custom-endpoint", (req, res) => {
84+
res.send({ msg: "My own express app!" });
85+
});
86+
87+
const daprServer = new DaprServer(
88+
"127.0.0.1", // App Host
89+
"50002", // App Port
90+
daprHost,
91+
daprPort,
92+
CommunicationProtocolEnum.HTTP,
93+
{},
94+
{
95+
serverHttp: myApp,
96+
},
97+
);
98+
99+
// Initialize subscriptions before the server starts, the Dapr sidecar uses it.
100+
// This will also initialize the app server itself (removing the need for `app.listen` to be called).
101+
await daprServer.start();
102+
```
103+
104+
After configuring the above, you can call your custom endpoint as you normally would:
105+
106+
```typescript
107+
const res = await fetch(`http://127.0.0.1:50002/my-custom-endpoint`);
108+
const json = await res.json();
109+
```
110+
69111
### Using gRPC
70112

71113
Since HTTP is the default, you will have to adapt the communication protocol to use gRPC. You can do this by passing an extra argument to the client or server constructor.

package.json

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,10 @@
1111
"test:e2e:all": "npm run test:e2e:http; npm run test:e2e:grpc",
1212
"test:e2e:grpc": "npm run test:e2e:grpc:client && npm run test:e2e:grpc:server",
1313
"test:e2e:grpc:client": "npm run prebuild && TEST_SECRET_1=secret_val_1 TEST_SECRET_2=secret_val_2 dapr run --app-id test-suite --app-protocol grpc --app-port 50001 --dapr-grpc-port 50000 --components-path ./test/components -- jest --runInBand --detectOpenHandles --testMatch [ '**/test/e2e/grpc/(client).test.ts' ]",
14-
"test:e2e:grpc:server": "npm run prebuild && TEST_SECRET_1=secret_val_1 TEST_SECRET_2=secret_val_2 dapr run --app-id test-suite --app-protocol grpc --app-port 50001 --dapr-grpc-port 50000 --components-path ./test/components -- jest --runInBand --detectOpenHandles --testMatch [ '**/test/e2e/grpc/(server).test.ts' ]",
14+
"test:e2e:grpc:server": "npm run prebuild && TEST_SECRET_1=secret_val_1 TEST_SECRET_2=secret_val_2 dapr run --app-id test-suite --app-protocol grpc --app-port 50001 --dapr-grpc-port 50000 --dapr-http-max-request-size 10 --components-path ./test/components -- jest --runInBand --detectOpenHandles --testMatch [ '**/test/e2e/grpc/(server).test.ts' ]",
1515
"test:e2e:http": "npm run test:e2e:http:client && npm run test:e2e:http:server && npm run test:e2e:http:actors",
1616
"test:e2e:http:client": "npm run prebuild && TEST_SECRET_1=secret_val_1 TEST_SECRET_2=secret_val_2 dapr run --app-id test-suite --app-protocol http --app-port 50001 --dapr-http-port 50000 --components-path ./test/components -- jest --runInBand --detectOpenHandles --testMatch [ '**/test/e2e/http/(client).test.ts' ]",
17-
"test:e2e:http:server": "npm run prebuild && TEST_SECRET_1=secret_val_1 TEST_SECRET_2=secret_val_2 dapr run --app-id test-suite --app-protocol http --app-port 50001 --dapr-http-port 50000 --components-path ./test/components -- jest --runInBand --detectOpenHandles --testMatch [ '**/test/e2e/http/(server).test.ts' ]",
17+
"test:e2e:http:server": "npm run prebuild && TEST_SECRET_1=secret_val_1 TEST_SECRET_2=secret_val_2 dapr run --app-id test-suite --app-protocol http --app-port 50001 --dapr-http-port 50000 --dapr-http-max-request-size 10 --components-path ./test/components -- jest --runInBand --detectOpenHandles --testMatch [ '**/test/e2e/http/(server).test.ts' ]",
1818
"test:e2e:http:actors": "npm run prebuild && TEST_SECRET_1=secret_val_1 TEST_SECRET_2=secret_val_2 dapr run --app-id test-suite --app-protocol http --app-port 50001 --dapr-http-port 50000 --components-path ./test/components -- jest --runInBand --detectOpenHandles --testMatch [ '**/test/e2e/http/actors.test.ts' ]",
1919
"test:unit": "jest --runInBand --detectOpenHandles",
2020
"test:unit:all": "npm run test:unit:main && npm run test:unit:actors && npm run test:unit:logger && npm run test:unit:utils",
@@ -38,14 +38,14 @@
3838
"@types/google-protobuf": "^3.15.5",
3939
"@types/node-fetch": "^2.6.2",
4040
"body-parser": "^1.19.0",
41+
"express": "^4.18.2",
4142
"google-protobuf": "^3.18.0",
4243
"http-terminator": "^3.0.4",
43-
"node-fetch": "^2.6.7",
44-
"restana": "^4.9.1",
45-
"uuid": "^8.3.2"
44+
"node-fetch": "^2.6.7"
4645
},
4746
"devDependencies": {
4847
"@types/body-parser": "^1.19.1",
48+
"@types/express": "^4.17.15",
4949
"@types/jest": "^27.0.1",
5050
"@types/node": "^16.9.1",
5151
"@types/uuid": "^8.3.1",

src/actors/ActorId.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ See the License for the specific language governing permissions and
1111
limitations under the License.
1212
*/
1313

14-
import { v4 as uuidv4 } from "uuid";
14+
import { randomUUID } from "crypto";
1515

1616
export default class ActorId {
1717
private readonly id: string;
@@ -21,7 +21,7 @@ export default class ActorId {
2121
}
2222

2323
static createRandomId(): ActorId {
24-
return new ActorId(uuidv4());
24+
return new ActorId(randomUUID());
2525
}
2626

2727
getId() {

src/actors/runtime/AbstractActor.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -86,9 +86,9 @@ export default abstract class AbstractActor {
8686
*
8787
* @param reminderName name of the reminder
8888
* @param state the state to be send along with the reminder trigger
89-
* @param dueTime due time for the first trigger
89+
* @param dueTime Specifies the time after which the reminder is invoked
90+
* @param period Specifies the period between different invocations
9091
* @param ttl time to duration after which the reminder will be expired and deleted
91-
* @param period frequency for the triggers
9292
* @param <Type> Type of the state object
9393
* @return Async void response
9494
*/

src/implementation/Client/GRPCClient/GRPCClient.ts

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -69,10 +69,28 @@ export default class GRPCClient implements IClient {
6969
return this.clientCredentials;
7070
}
7171

72+
private generateChannelOptions(): Record<string, string | number> {
73+
const options: Record<string, string | number> = {};
74+
75+
// See: GRPC_ARG_MAX_SEND_MESSAGE_LENGTH, it is in bytes
76+
// https://grpc.github.io/grpc/core/group__grpc__arg__keys.html#ga813f94f9ac3174571dd712c96cdbbdc1
77+
// Default is 4Mb
78+
options["grpc.max_send_message_length"] = (this.options.maxBodySizeMb ?? 4) * 1024 * 1024;
79+
80+
// There was an issue that there was no default set in grpc-node, so we set it here
81+
// https://github.com/grpc/grpc-node/issues/1158#issuecomment-1137023216
82+
options["grpc-node.max_session_memory"] = Number.MAX_SAFE_INTEGER;
83+
84+
// Add user agent
85+
options["grpc.primary_user_agent"] = "dapr-sdk-js/v" + SDK_VERSION;
86+
87+
return options;
88+
}
89+
7290
private generateClient(host: string, port: string, credentials: grpc.ChannelCredentials): GrpcDaprClient {
73-
const client = new GrpcDaprClient(`${host}:${port}`, credentials, {
74-
"grpc.primary_user_agent": "dapr-sdk-js/v" + SDK_VERSION,
75-
});
91+
const options = this.generateChannelOptions();
92+
const client = new GrpcDaprClient(`${host}:${port}`, credentials, options);
93+
7694
return client;
7795
}
7896

src/implementation/Client/GRPCClient/invoker.ts

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,11 @@ export default class GRPCClientInvoker implements IClientInvoker {
6666
return reject(err);
6767
}
6868

69-
// const res = await fetch(`${this.daprUrl}/invoke/${appId}/method/${methodName}`, fetchOptions);
70-
// return ResponseUtil.handleResponse(res);
71-
const resData = Buffer.from((res.getData() as Any).getValue()).toString();
69+
let resData = "";
70+
71+
if (res.getData()) {
72+
resData = Buffer.from((res.getData() as Any).getValue()).toString();
73+
}
7274

7375
try {
7476
const parsedResData = JSON.parse(resData);

src/implementation/Client/HTTPClient/HTTPClient.ts

Lines changed: 25 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ See the License for the specific language governing permissions and
1111
limitations under the License.
1212
*/
1313

14-
import fetch from "node-fetch";
14+
import fetch, { RequestInit } from "node-fetch";
1515
import { CommunicationProtocolEnum, DaprClient } from "../../..";
1616
import IClient from "../../../interfaces/Client/IClient";
1717
import http from "http";
@@ -22,6 +22,7 @@ import { THTTPExecuteParams } from "../../../types/http/THTTPExecuteParams.type"
2222
import { Logger } from "../../../logger/Logger";
2323
import HTTPClientSidecar from "./sidecar";
2424
import { SDK_VERSION } from "../../../version";
25+
import * as SerializerUtil from "../../../utils/Serializer.util";
2526

2627
export default class HTTPClient implements IClient {
2728
private isInitialized: boolean;
@@ -144,50 +145,44 @@ export default class HTTPClient implements IClient {
144145
params?: THTTPExecuteParams | undefined | null,
145146
requiresInitialization = true,
146147
): Promise<object | string> {
147-
if (!params || typeof params !== "object") {
148-
params = {
149-
method: "GET",
150-
};
151-
}
148+
const clientOptions: RequestInit = {};
152149

153-
if (!params?.headers) {
154-
params.headers = {};
155-
}
150+
// Set Method
151+
clientOptions.method = params?.method.toLocaleUpperCase() || (params?.body ? "POST" : "GET");
152+
153+
// Set Headers
154+
clientOptions.headers = params?.headers ?? {};
156155

157156
if (this.options.daprApiToken) {
158-
params.headers["dapr-api-token"] = this.options.daprApiToken;
157+
clientOptions.headers["dapr-api-token"] = this.options.daprApiToken;
159158
}
160159

161-
params.headers["user-agent"] = `dapr-sdk-js/v${SDK_VERSION} http/1`;
160+
clientOptions.headers["user-agent"] = `dapr-sdk-js/v${SDK_VERSION} http/1`;
162161

163-
if (!params?.method) {
164-
params.method = "GET";
165-
}
162+
// Set Body and Content-Type Header
163+
if (params?.body) {
164+
const { serializedData, contentType } = SerializerUtil.serializeHttp(params?.body);
166165

167-
if (params?.body && !params?.headers["Content-Type"]) {
168-
switch (typeof params?.body) {
169-
case "object":
170-
params.headers["Content-Type"] = "application/json";
171-
params.body = JSON.stringify(params?.body);
172-
break;
173-
case "string":
174-
params.headers["Content-Type"] = "text/plain";
175-
break;
176-
default:
177-
this.logger.warn(`Unknown body type: ${typeof params?.body}, defaulting to "text/plain"`);
178-
params.headers["Content-Type"] = "text/plain";
179-
break;
166+
// Don't overwrite it
167+
if (!params?.headers?.["Content-Type"]) {
168+
clientOptions.headers["Content-Type"] = contentType;
180169
}
170+
171+
clientOptions.body = serializedData;
181172
}
182173

183174
const urlFull = url.startsWith("http") ? url : `${this.clientUrl}${url}`;
184175
const agent = urlFull.startsWith("https") ? HTTPClient.httpsAgent : HTTPClient.httpAgent;
185-
params.agent = agent;
176+
clientOptions.agent = agent;
186177

187-
this.logger.debug(`Fetching ${params.method} ${urlFull} with body: (${params.body})`);
178+
this.logger.debug(
179+
`Fetching ${clientOptions.method} ${urlFull} with (headers: ${JSON.stringify(
180+
clientOptions.headers,
181+
)}, body size: ${(clientOptions.body?.toString()?.length ?? 0) / 1024 / 1024} Mb)`,
182+
);
188183

189184
const client = await this.getClient(requiresInitialization);
190-
const res = await client(urlFull, params);
185+
const res = await client(urlFull, clientOptions);
191186

192187
// Parse body
193188
const txt = await res.text();

0 commit comments

Comments
 (0)