Skip to content

Commit 99af615

Browse files
0xgouda0xgouda
authored
[+] add rpc folder (#1)
* add `.github` folder * add .gitignore * copy contents of `destrex271/pgwatch3_rpc_server` repo to `gRPC_sinks` folder * add README for `gRPC_sinks` folder * update `gRPC_sinks/assets` image * rename `gRPC_sinks` dir to `rpc` * update README
1 parent 7c70865 commit 99af615

File tree

102 files changed

+9412
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

102 files changed

+9412
-0
lines changed

‎.github/ISSUE_TEMPLATE/bug_report.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
name: Bug report
3+
about: Create a report to help us improve
4+
title: ''
5+
labels: ''
6+
assignees: ''
7+
8+
---
9+
10+
**Describe the bug**
11+
A clear and concise description of what the bug is.
12+
13+
**To Reproduce**
14+
Steps to reproduce the behavior:
15+
1. Go to '...'
16+
2. Click on '....'
17+
3. Scroll down to '....'
18+
4. See error
19+
20+
**Expected behavior**
21+
A clear and concise description of what you expected to happen.
22+
23+
**Screenshots**
24+
If applicable, add screenshots to help explain your problem.
25+
26+
**Desktop (please complete the following information):**
27+
- OS: [e.g. iOS]
28+
29+
**Additional context**
30+
Add any other context about the problem here.

‎.github/ISSUE_TEMPLATE/enhancement.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
name: Enhancement
3+
about: Short Description
4+
title: ''
5+
labels: ''
6+
assignees: ''
7+
8+
---
9+
10+
**What does this enhancement request propose?**
11+
12+
13+
**How does it improve current state of the product?**
14+
15+
**Additional Context**

‎.github/ISSUE_TEMPLATE/feature_request.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
name: Feature request
3+
about: Suggest an idea for this project
4+
title: ''
5+
labels: ''
6+
assignees: ''
7+
8+
---
9+
10+
**Is your feature request related to a problem? Please describe.**
11+
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
12+
13+
**Describe the solution you'd like**
14+
A clear and concise description of what you want to happen.
15+
16+
**Describe alternatives you've considered**
17+
A clear and concise description of any alternative solutions or features you've considered.
18+
19+
**Additional context**
20+
Add any other context or screenshots about the feature request here.

‎.github/actions/setup-protobuf/action.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
name: 'Setup Protobuf'
2+
description: 'Install protoc and Go plugins with caching'
3+
runs:
4+
using: 'composite'
5+
steps:
6+
- name: Cache protobuf tools
7+
uses: actions/cache@v4
8+
with:
9+
path: |
10+
~/go/bin/protoc-gen-go
11+
~/go/bin/protoc-gen-go-grpc
12+
key: ${{ runner.os }}-protoc-tools-${{ hashFiles('go.sum') }}
13+
restore-keys: |
14+
${{ runner.os }}-protoc-tools-
15+
16+
- name: Install protoc
17+
uses: arduino/setup-protoc@v3
18+
with:
19+
version: '25.x'
20+
repo-token: ${{ github.token }}
21+
22+
- name: Install protobuf tools
23+
shell: bash
24+
run: |
25+
if [ ! -f ~/go/bin/protoc-gen-go ] || [ ! -f ~/go/bin/protoc-gen-go-grpc ]; then
26+
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
27+
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
28+
fi

‎.github/workflows/test.yml‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
name: Test
2+
3+
on:
4+
push:
5+
branches: [ "main" ]
6+
pull_request:
7+
branches: [ "main" ]
8+
workflow_dispatch:
9+
10+
concurrency:
11+
group: ${{ github.workflow }}-${{ github.ref }}
12+
cancel-in-progress: true
13+
14+
jobs:
15+
Lint:
16+
runs-on: ubuntu-latest
17+
steps:
18+
19+
- name: Check out code
20+
uses: actions/checkout@v4
21+
22+
- name: Set up Golang
23+
uses: actions/setup-go@v5
24+
25+
- name: Setup Protobuf
26+
uses: ./.github/actions/setup-protobuf
27+
28+
- name: Generate protobuf files
29+
run: go generate ./sinks/pb/
30+
31+
- name: GolangCI-Lint
32+
uses: golangci/golangci-lint-action@v8
33+
with:
34+
version: latest
35+
36+
Unit-Test:
37+
runs-on: ubuntu-latest
38+
steps:
39+
- uses: actions/checkout@v4
40+
41+
- name: Set up Go
42+
uses: actions/setup-go@v5
43+
with:
44+
go-version: '1.23'
45+
cache-dependency-path: 'go.sum'
46+
47+
- name: Setup Protobuf
48+
uses: ./.github/actions/setup-protobuf
49+
50+
- name: Set up Python
51+
uses: actions/setup-python@v5
52+
with:
53+
python-version: '3.12'
54+
55+
- name: Install dependencies
56+
run: |
57+
python3 -m pip install --upgrade pip
58+
pip install pytest
59+
pip install -r cmd/pyiceberg_receiver/requirements.txt
60+
61+
- name: gRPC Golang Tests
62+
run: |
63+
cd gRPC_sinks
64+
go generate ./sinks/pb
65+
go test -timeout 20m -failfast -v -coverprofile=profile.cov ./...
66+
67+
- name: gRPC Python Tests
68+
run: |
69+
cd gRPC_sinks
70+
python3 -m grpc_tools.protoc -I sinks/pb --python_out=cmd/pyiceberg_receiver --grpc_python_out=cmd/pyiceberg_receiver sinks/pb/pgwatch.proto
71+
pytest
72+
73+
- name: Coveralls
74+
uses: coverallsapp/github-action@v2
75+
with:
76+
file: profile.cov

‎.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
./*/*/*.txt
2+
.vscode/
3+
*pb.go
4+
*pb2*.py
5+
__pycache__/
6+
.pytest_cache/

‎rpc/README.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Pgwatch RPC Sinks
2+
3+
![RPC sinks architecture diagram](assets/gRPC_servers_architecture.jpg)
4+
5+
## Intro
6+
7+
A **community-maintained** collection of gRPC sink implementations for pgwatch.
8+
9+
This provides sinks for common data solutions but makes no guarantees about their suitability for real production use.
10+
11+
Our main purpose is to provide examples and building blocks that users can extend to integrate with pgwatch and develop their own production-ready gRPC servers.
12+
13+
Check out <a href="https://github.com/cybertec-postgresql/pgwatch">PgWatch</a> to get started with this project.
14+
15+
## Running Sinks
16+
17+
If you are using pgwatch's gRPC client sink with
18+
authentication credentials or TLS configured, you'll
19+
need to set the following environment variables
20+
to ensure the server works properly.
21+
22+
```
23+
# if empty, password is ignored during authentication
24+
export PGWATCH_RPC_SERVER_USERNAME="username"
25+
26+
# if empty, username is ignored during authentication
27+
export PGWATCH_RPC_SERVER_PASSWORD="password"
28+
29+
# if not set, TLS is not used
30+
export PGWATCH_RPC_SERVER_CERT="/path/to/server.crt"
31+
32+
# if not set, TLS is not used
33+
export PGWATCH_RPC_SERVER_KEY="/path/to/server.key"
34+
```
35+
36+
To start any of the provided receivers, you can use:
37+
```bash
38+
# generate golang code from protobuf
39+
go generate ./sinks/pb
40+
go run ./cmd/[receiver_dir] [OPTIONS]
41+
42+
# OR
43+
44+
# generate python gRPC code from protobuf
45+
python3 -m grpc_tools.protoc -I sinks/pb --python_out=cmd/pyiceberg_receiver --grpc_python_out=cmd/pyiceberg_receiver sinks/pb/pgwatch.proto
46+
python3 ./cmd/[receiver_dir] [OPTIONS]
47+
```
48+
By default, all sinks will listen at `0.0.0.0` with the specified port number.
49+
50+
Now once your receiver is up, run pgwatch with the argument:
51+
`--sink=grpc://<sink_ip/hostname>:<sink-port> [OPTIONS]`
52+
53+
Voila! You have seamless integration between pgwatch and your custom sink.
54+
55+
## Developing Custom Sinks
56+
57+
To get started with developing your own custom sinks, refer to this mini [tutorial](TUTORIAL.md).
58+
59+
You can also look at our [example sinks](./cmd/) to help with your implementation or extend them for your own use cases.
60+
61+
## Credits
62+
63+
[Akshat Jaimini](https://github.com/destrex271)
64+
[Ahmed Gouda](https://github.com/0xgouda)

0 commit comments

Comments
 (0)