Skip to content

Commit 164ddd4

Browse files
igorbernstein2igorbernstein2
authored
docs: packaging & docs (#9695)
stack-info: PR: #9695, branch: igorbernstein2/stack/6
1 parent 00b8db6 commit 164ddd4

File tree

7 files changed

+199
-4
lines changed

7 files changed

+199
-4
lines changed

‎bigtable/bigtable-proxy/README.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Bigtable proxy
2+
3+
## Overview
4+
5+
A simple server meant to be used as a sidecar to maintain a persistent connection to Bigtable and
6+
collect metrics. The primary purpose is to support applications that can't maintain a longlived
7+
gRPC connection (ie. php in apache).
8+
9+
The proxy is intended to be used as a local sidecar process. The proxy is intended to be shared by
10+
all processes on the VM that it is running on. It's listening address is hardcoded to `localhost`.
11+
The proxy will use [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials)
12+
for all outbound RPCs.
13+
14+
The proxy will accept local unencrypted connections from Bigtable clients, and:
15+
- attach credentials
16+
- export metrics
17+
- send the RPC over an encrypted channel pool to Bigtable service
18+
19+
## Features
20+
21+
* Metrics - The proxy will track RPC metrics and export them to Google Cloud Monitoring
22+
* Multi tenant - The proxy can be used to connect to many different Bigtable instances
23+
* Credential handling - The proxy has its own set of credentials. It will ignore any inbound
24+
credentials from the client
25+
* Channel pooling - The proxy will maintain and autosize the outbound channel pool to properly
26+
load balance RPCs.
27+
28+
## Metrics
29+
30+
The proxy is instrumented with Opentelemtry and will export those metrics to Google Cloud Monitoring
31+
in a project your choosing. The metrics will be published under the namespace
32+
`workload.googleapis.com`. Available metrics:
33+
34+
* `bigtableproxy.server.call.started` The total number of RPCs started, including those that have
35+
not completed.
36+
* `bigtableproxy.client.call.credential.duration` Latency of getting credentials
37+
* `bigtableproxy.client.call.queue.duration` Duration of how long the outbound side of the proxy had
38+
the RPC queued
39+
* `bigtableproxy.client.call.sent_total_message_size` Total bytes sent per call to Bigtable service
40+
(excluding metadata, grpc and transport framing bytes
41+
* `bigtableproxy.client.call.rcvd_total_message_size` Total bytes received per call from Bigtable
42+
service (excluding metadata, grpc and transport framing bytes)
43+
* `bigtableproxy.client.gfe.duration` Latency as measured by Google load balancer from the time it
44+
received the first byte of the request until it received the first byte of the response from the
45+
Cloud Bigtable service.
46+
* `bigtableproxy.client.gfe.duration_missing.count` Count of calls missing gfe response headers
47+
* `bigtableproxy.client.call.duration` Total duration of how long the outbound call took
48+
* `bigtableproxy.client.channel.count` Number of open channels
49+
* `bigtableproxy.client.call.max_outstanding_count` Maximum number of concurrent RPCs in a single
50+
minute window
51+
52+
## Requirements
53+
54+
* JVM >= 11
55+
* Ensure that the service account includes the IAM roles:
56+
* `Monitoring Metric Writer`
57+
* `Bigtable User`
58+
* Ensure that the metrics project has `Stackdriver Monitoring API` enabled
59+
60+
## Expected usage
61+
62+
```sh
63+
# Build the binary
64+
mvn package
65+
66+
# use the binary
67+
unzip target/bigtable-proxy-0.0.1-SNAPSHOT-bin.zip
68+
cd bigtable-proxy-0.0.1-SNAPSHOT
69+
./bigtable-proxy.sh \
70+
--listen-port=1234 \
71+
--metrics-project-id=SOME_GCP_PROJECT
72+
73+
export BIGTABLE_EMULATOR_HOST=1234
74+
path/to/application/with/bigtable/client
75+
```
76+
77+
## Configuration
78+
79+
Required options:
80+
* `--listen-port=<port>` The local port to listen for Bigtable client connections. This needs to
81+
match port in the `BIGTABLE_EMULATOR_HOST="localhost:<port>` environment variable passed to your
82+
application.
83+
* `--metrics-project-id=<projectid>` The Google Cloud project that should be used to collect metrics
84+
emitted from the proxy.
85+
86+
Optional configuration:
87+
* The environment variable `GOOGLE_APPLICATION_CREDENTIALS` can be used to use a non-default service
88+
account. More details can be found here: https://cloud.google.com/docs/authentication/application-default-credentials

‎bigtable/bigtable-proxy/pom.xml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,45 @@
217217
<artifactId>maven-surefire-plugin</artifactId>
218218
<version>3.5.2</version>
219219
</plugin>
220+
221+
<plugin>
222+
<groupId>org.apache.maven.plugins</groupId>
223+
<artifactId>maven-jar-plugin</artifactId>
224+
<version>3.4.2</version>
225+
<configuration>
226+
<archive>
227+
<manifest>
228+
<addClasspath>true</addClasspath>
229+
230+
<classpathPrefix>lib/</classpathPrefix>
231+
<mainClass>com.google.cloud.bigtable.examples.proxy.Main</mainClass>
232+
</manifest>
233+
</archive>
234+
</configuration>
235+
</plugin>
236+
237+
<plugin>
238+
<groupId>org.apache.maven.plugins</groupId>
239+
<artifactId>maven-assembly-plugin</artifactId>
240+
<version>3.7.1</version>
241+
242+
<configuration>
243+
<!-- Configures the used assembly descriptor -->
244+
<descriptors>
245+
<descriptor>src/main/assembly/assembly.xml</descriptor>
246+
</descriptors>
247+
</configuration>
248+
249+
<executions>
250+
<execution>
251+
<id>assemble</id>
252+
<goals>
253+
<goal>single</goal>
254+
</goals>
255+
<phase>package</phase>
256+
</execution>
257+
</executions>
258+
</plugin>
220259
</plugins>
221260
</build>
222261
</project>

‎bigtable/bigtable-proxy/src/main/assembly/assembly.xml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.2.0"
2+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
3+
xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.2.0 http://maven.apache.org/xsd/assembly-2.2.0.xsd">
4+
<id>bin</id>
5+
6+
<formats>
7+
<format>zip</format>
8+
</formats>
9+
10+
11+
<dependencySets>
12+
<!-- Add deps -->
13+
<dependencySet>
14+
<useProjectArtifact>false</useProjectArtifact>
15+
<outputDirectory>lib</outputDirectory>
16+
<unpack>false</unpack>
17+
</dependencySet>
18+
</dependencySets>
19+
20+
<fileSets>
21+
<!-- docs -->
22+
<fileSet>
23+
<directory>${project.basedir}</directory>
24+
<outputDirectory></outputDirectory>
25+
<includes>
26+
<include>README*</include>
27+
<include>LICENSE*</include>
28+
<include>NOTICE*</include>
29+
</includes>
30+
</fileSet>
31+
32+
<!-- Add scripts -->
33+
<fileSet>
34+
<directory>${project.build.scriptSourceDirectory}</directory>
35+
<outputDirectory></outputDirectory>
36+
<includes>
37+
<include>*.sh</include>
38+
</includes>
39+
<filtered>true</filtered>
40+
</fileSet>
41+
42+
43+
<!-- Add project jar to root -->
44+
<fileSet>
45+
<directory>${project.build.directory}</directory>
46+
<outputDirectory></outputDirectory>
47+
<includes>
48+
<include>*.jar</include>
49+
</includes>
50+
</fileSet>
51+
</fileSets>
52+
</assembly>

‎bigtable/bigtable-proxy/src/main/java/com/google/cloud/bigtable/examples/proxy/Main.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
* Main entry point for proxy commands under {@link
2828
* com.google.cloud.bigtable.examples.proxy.commands}.
2929
*/
30-
@Command(subcommands = {Serve.class})
30+
@Command(subcommands = {Serve.class}, name = "bigtable-proxy")
3131
public final class Main {
3232
private static final Logger LOGGER = LoggerFactory.getLogger(Main.class);
3333

‎bigtable/bigtable-proxy/src/main/java/com/google/cloud/bigtable/examples/proxy/commands/Serve.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@
5252
import picocli.CommandLine.Help.Visibility;
5353
import picocli.CommandLine.Option;
5454

55-
@Command(name = "serve", mixinStandardHelpOptions = true, description = "Start the proxy server")
55+
@Command(name = "serve", description = "Start the proxy server")
5656
public class Serve implements Callable<Void> {
5757
private static final Logger LOGGER = LoggerFactory.getLogger(Serve.class);
5858

‎bigtable/bigtable-proxy/src/main/java/com/google/cloud/bigtable/examples/proxy/metrics/MetricsImpl.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ public MetricsImpl(Credentials credentials, String projectId) throws IOException
7878

7979
clientCredLatencies =
8080
meter
81-
.histogramBuilder(METRIC_PREFIX + "client.call.credential.refresh.duration")
81+
.histogramBuilder(METRIC_PREFIX + "client.call.credential.duration")
8282
.setDescription("Latency of getting credentials")
8383
.setUnit("ms")
8484
.build();
@@ -144,7 +144,7 @@ public MetricsImpl(Credentials credentials, String projectId) throws IOException
144144

145145
meter
146146
.gaugeBuilder(METRIC_PREFIX + "client.call.max_outstanding_count")
147-
.setDescription("Number of concurrent")
147+
.setDescription("Maximum number of concurrent RPCs in a single minute window")
148148
.setUnit("{call}")
149149
.ofLongs()
150150
.buildWithCallback(o -> o.record(maxSeen.getAndSet(0)));

‎bigtable/bigtable-proxy/src/main/scripts/bigtable-proxy.sh‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/sh
2+
# Copyright 2024 Google LLC
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
java -jar ${project.build.finalName}.jar serve "$@"

0 commit comments

Comments
 (0)