Skip to content

Commit 4ead36f

Browse files
dhirupandeydhirupandey
committed
Initial commit
1 parent 71eb425 commit 4ead36f

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

69 files changed

+13188
-17
lines changed

‎.coveragerc

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[run]
2+
omit =
3+
coherence/messages_pb2.py
4+
coherence/messages_pb2_grpc.py
5+
coherence/services_pb2.py
6+
coherence/services_pb2_grpc.py

‎.github/workflows/validate.yml

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Copyright 2022 Oracle Corporation and/or its affiliates.
2+
# Licensed under the Universal Permissive License v 1.0 as shown at
3+
# https://oss.oracle.com/licenses/upl.
4+
5+
name: CI Build
6+
on:
7+
workflow_dispatch:
8+
schedule:
9+
- cron: "0 5 * * *"
10+
push:
11+
branches: [ main ]
12+
pull_request:
13+
branches: [ main ]
14+
jobs:
15+
ci:
16+
strategy:
17+
fail-fast: false
18+
matrix:
19+
python-version: ["3.10"]
20+
poetry-version: ["1.1.15"]
21+
os: [ubuntu-latest]
22+
runs-on: ${{ matrix.os }}
23+
steps:
24+
- uses: actions/checkout@v3
25+
- uses: actions/setup-python@v4
26+
with:
27+
python-version: ${{ matrix.python-version }}
28+
- name: Run image
29+
uses: abatilo/actions-poetry@v2
30+
with:
31+
poetry-version: ${{ matrix.poetry-version }}
32+
- name: Install Dependencies
33+
run: python -m poetry install
34+
- name: Configure Validation
35+
run: python -m poetry run make validate-setup
36+
- name: Validate Sources
37+
run: python -m poetry run make validate
38+
- name: Start Container
39+
run: python -m poetry run bin/docker-utils.sh -u
40+
- name: Wait for Container Start
41+
run: sleep 30
42+
- name: Run Tests
43+
run: python -m poetry run make test
44+
- name: Stop Container
45+
run: python -m poetry run bin/docker-utils.sh -d

‎.gitignore

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Copyright (c) 2022, Oracle and/or its affiliates.
2+
# Licensed under the Universal Permissive License v 1.0 as shown at
3+
# http://oss.oracle.com/licenses/upl.
4+
5+
.idea
6+
.python-version
7+
.coverage
8+
.pytest_cache
9+
*.iml
10+
etc
11+
build
12+
poetry.lock
13+
__pycache__
14+
tests/java/coherence-python-test/target
15+
tests/utils/certs
16+
tests/utils/.env

‎.pre-commit-config.yaml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Copyright (c) 2022, Oracle and/or its affiliates.
2+
# Licensed under the Universal Permissive License v 1.0 as shown at
3+
# https://oss.oracle.com/licenses/upl.
4+
5+
# See https://pre-commit.com for more information
6+
# See https://pre-commit.com/hooks.html for more hooks
7+
exclude: \w*(_pb2)\w*
8+
9+
repos:
10+
- repo: https://github.com/pre-commit/pre-commit-hooks
11+
rev: v4.4.0
12+
hooks:
13+
- id: trailing-whitespace
14+
- id: end-of-file-fixer
15+
- id: check-yaml
16+
- id: check-added-large-files
17+
18+
- repo: https://github.com/PyCQA/flake8
19+
rev: 6.0.0
20+
hooks:
21+
- id: flake8
22+
23+
- repo: https://github.com/psf/black
24+
rev: 22.10.0
25+
hooks:
26+
- id: black
27+
28+
- repo: https://github.com/pre-commit/mirrors-mypy
29+
rev: v0.991
30+
hooks:
31+
- id: mypy
32+
33+
- repo: https://github.com/PyCQA/isort
34+
rev: 5.10.1
35+
hooks:
36+
- id: isort

‎DEVELOPING.md

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
<!--
2+
Copyright (c) 2022, Oracle and/or its affiliates.
3+
4+
Licensed under the Universal Permissive License v 1.0 as shown at
5+
http://oss.oracle.com/licenses/upl.
6+
-->
7+
8+
# Developing Coherence Python Client
9+
10+
### Pre-Requisites
11+
* Use `bash` shell
12+
* [pyenv](https://github.com/pyenv/pyenv) version 2.2.4 or later
13+
* [poetry](https://github.com/python-poetry/poetry) version 1.1.13 or later
14+
15+
### Project Structure
16+
* `bin` - various shell scripts
17+
* `etc` - contains the ProtoBuff .protoc files and other various files
18+
* `coherence` - Python source files and related resources
19+
* `tests` - contains the library test cases in plain Python
20+
21+
### Setup working environment
22+
1. Install Python version 3.10.1
23+
24+
```pyenv install 3.10.1```
25+
26+
2. Checkout the source from GitHub and change directory to the project root dir
27+
28+
3. Set pyenv local version for this project to 3.10.1
29+
30+
```pyenv local 3.10.1```
31+
4. Set poetry to use python 3.10.1
32+
33+
```poetry env use 3.10.1```
34+
5. Install all the required dependencies using poetry
35+
36+
```poetry install```
37+
6. For setting up IntelliJ IDE, install the Python and PyEnv plugins
38+
7. Get the full path of the poetry/python virtualenv
39+
40+
```poetry env list --full-path```
41+
8. Use the path above to setup the project settings in IntelliJ IDE. See [this](https://www.jetbrains.com/help/idea/creating-virtual-environment.html)
42+
43+
Use the *Existing Environment* option to use the path above
44+
45+
**Note:** Incase one needs to re-generate the sources from .proto files, make sure steps 1-5 are done.
46+
Run the Makefile in the virtualenv shell:
47+
```
48+
poetry shell
49+
make generate-proto
50+
```

‎Makefile

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
# ----------------------------------------------------------------------------------------------------------------------
2+
# Copyright (c) 2022, 2023, Oracle and/or its affiliates.
3+
# Licensed under the Universal Permissive License v 1.0 as shown at
4+
# https://oss.oracle.com/licenses/upl.
5+
#
6+
# ----------------------------------------------------------------------------------------------------------------------
7+
# This is the Makefile to build the Coherence Python Client
8+
# ----------------------------------------------------------------------------------------------------------------------
9+
10+
VERSION ?=0.9.0
11+
CURRDIR := $(shell pwd)
12+
USER_ID := $(shell echo "`id -u`:`id -g`")
13+
14+
override BUILD_BIN := $(CURRDIR)/bin
15+
override PROTO_DIR := $(CURRDIR)/etc/proto
16+
17+
# ----------------------------------------------------------------------------------------------------------------------
18+
# Set the location of various build tools
19+
# ----------------------------------------------------------------------------------------------------------------------
20+
override BUILD_OUTPUT := $(CURRDIR)/build/_output
21+
override BUILD_BIN := $(CURRDIR)/bin
22+
override PROTO_OUT := $(CURRDIR)/proto
23+
override BUILD_TARGETS := $(BUILD_OUTPUT)/targets
24+
override TEST_LOGS_DIR := $(BUILD_OUTPUT)/test-logs
25+
override COVERAGE_DIR := $(BUILD_OUTPUT)/coverage
26+
override COPYRIGHT_JAR := glassfish-copyright-maven-plugin-2.4.jar
27+
override BUILD_CERTS := $(CURRDIR)/tests/utils/certs
28+
override ENV_FILE := tests/utils/.env
29+
30+
# Maven version is always 1.0.0 as it is only for testing
31+
MVN_VERSION ?= 1.0.0
32+
33+
# Coherence CE version to run base tests against
34+
COHERENCE_VERSION ?= 22.06.2
35+
COHERENCE_GROUP_ID ?= com.oracle.coherence.ce
36+
COHERENCE_WKA1 ?= server1
37+
COHERENCE_WKA2 ?= server1
38+
CLUSTER_PORT ?= 7574
39+
# Profiles to include for building
40+
PROFILES ?=
41+
COHERENCE_BASE_IMAGE ?= gcr.io/distroless/java:11
42+
43+
# ----------------------------------------------------------------------------------------------------------------------
44+
# Set the location of various build tools
45+
# ----------------------------------------------------------------------------------------------------------------------
46+
TOOLS_DIRECTORY = $(CURRDIR)/build/tools
47+
TOOLS_BIN = $(TOOLS_DIRECTORY)/bin
48+
49+
# ----------------------------------------------------------------------------------------------------------------------
50+
# The test application images used in integration tests
51+
# ----------------------------------------------------------------------------------------------------------------------
52+
RELEASE_IMAGE_PREFIX ?= ghcr.io/oracle/
53+
TEST_APPLICATION_IMAGE_1 := $(RELEASE_IMAGE_PREFIX)coherence-python-test-1:1.0.0
54+
TEST_APPLICATION_IMAGE_2 := $(RELEASE_IMAGE_PREFIX)coherence-python-test-2:1.0.0
55+
GO_TEST_FLAGS ?= -timeout 20m
56+
57+
# ----------------------------------------------------------------------------------------------------------------------
58+
# Options to append to the Maven command
59+
# ----------------------------------------------------------------------------------------------------------------------
60+
MAVEN_OPTIONS ?= -Dmaven.wagon.httpconnectionManager.ttlSeconds=25 -Dmaven.wagon.http.retryHandler.count=3
61+
MAVEN_BUILD_OPTS :=$(USE_MAVEN_SETTINGS) -Drevision=$(MVN_VERSION) -Dcoherence.version=$(COHERENCE_VERSION) -Dcoherence.group.id=$(COHERENCE_GROUP_ID) $(MAVEN_OPTIONS)
62+
63+
CURRDIR := $(shell pwd)
64+
65+
# ----------------------------------------------------------------------------------------------------------------------
66+
# Clean-up all of the build artifacts
67+
# ----------------------------------------------------------------------------------------------------------------------
68+
.PHONY: clean
69+
clean: ## Cleans the build
70+
@echo "Cleaning Project"
71+
-rm -rf $(CURRDIR)/build
72+
# -rm -rf $(PROTO_DIR)
73+
-rm -rf $(CURRDIR)/htmlcov
74+
-rm -rf $(CURRDIR)/.pytest_cache
75+
-rm -rf $(BUILD_CERTS)
76+
@mkdir -p $(BUILD_CERTS)
77+
mvn -B -f tests/java/coherence-python-test $(MAVEN_BUILD_OPTS) clean
78+
79+
.PHONY: certs
80+
certs: ## Generates certificates for TLS tests
81+
@echo "Generating certs"
82+
./tests/scripts/keys.sh $(BUILD_CERTS)
83+
84+
# ----------------------------------------------------------------------------------------------------------------------
85+
# Configure the build properties
86+
# ----------------------------------------------------------------------------------------------------------------------
87+
$(BUILD_PROPS):
88+
@echo "Creating build directories"
89+
@mkdir -p $(BUILD_OUTPUT)
90+
@mkdir -p $(BUILD_BIN)
91+
92+
# ----------------------------------------------------------------------------------------------------------------------
93+
# Build the Coherence Go Client Test Image
94+
# ----------------------------------------------------------------------------------------------------------------------
95+
.PHONY: build-test-images
96+
build-test-images: ## Build the Test images
97+
@echo "${MAVEN_BUILD_OPTS}"
98+
mvn -B -f tests/java/coherence-python-test clean package jib:dockerBuild -DskipTests -P member1$(PROFILES) -Djib.to.image=$(TEST_APPLICATION_IMAGE_1) -Dcoherence.test.base.image=$(COHERENCE_BASE_IMAGE) $(MAVEN_BUILD_OPTS)
99+
mvn -B -f tests/java/coherence-python-test clean package jib:dockerBuild -DskipTests -P member2$(PROFILES) -Djib.to.image=$(TEST_APPLICATION_IMAGE_2) -Dcoherence.test.base.image=$(COHERENCE_BASE_IMAGE) $(MAVEN_BUILD_OPTS)
100+
echo "CURRENT_UID=$(USER_ID)" >> $(ENV_FILE)
101+
102+
103+
# ----------------------------------------------------------------------------------------------------------------------
104+
# Download and build proto files
105+
# ----------------------------------------------------------------------------------------------------------------------
106+
.PHONY: generate-proto
107+
generate-proto: ## Generate Proto Files
108+
mkdir -p $(PROTO_DIR) || true
109+
curl -o $(PROTO_DIR)/services.proto https://raw.githubusercontent.com/oracle/coherence/v21.12/prj/coherence-grpc/src/main/proto/services.proto
110+
curl -o $(PROTO_DIR)/messages.proto https://raw.githubusercontent.com/oracle/coherence/v21.12/prj/coherence-grpc/src/main/proto/messages.proto
111+
python -m grpc_tools.protoc --proto_path=$(CURRDIR)/etc/proto --python_out=$(CURRDIR)/coherence --grpc_python_out=$(CURRDIR)/coherence $(CURRDIR)/etc/proto/messages.proto $(CURRDIR)/etc/proto/services.proto
112+
113+
# ----------------------------------------------------------------------------------------------------------------------
114+
# Run tests with code coverage
115+
# ----------------------------------------------------------------------------------------------------------------------
116+
.PHONY: test
117+
test: ##
118+
pytest -W error --cov coherence --cov-report=term --cov-report=html
119+
120+
# ----------------------------------------------------------------------------------------------------------------------
121+
# Run standards validation across project
122+
# ----------------------------------------------------------------------------------------------------------------------
123+
.PHONY: validate-setup
124+
validate-setup: ##
125+
poetry update
126+
pre-commit autoupdate
127+
128+
# ----------------------------------------------------------------------------------------------------------------------
129+
# Run standards validation across project
130+
# ----------------------------------------------------------------------------------------------------------------------
131+
.PHONY: validate
132+
validate: ##
133+
pre-commit run --all-files
134+
135+
# ----------------------------------------------------------------------------------------------------------------------
136+
# Obtain the protoc binary
137+
# ----------------------------------------------------------------------------------------------------------------------
138+
$(TOOLS_BIN)/protoc:
139+
@mkdir -p $(TOOLS_BIN)
140+
curl -Lo $(TOOLS_DIRECTORY)/protoc-3.19.4-osx-x86_64.zip https://github.com/protocolbuffers/protobuf/releases/download/v3.19.4/protoc-3.19.4-osx-x86_64.zip
141+
cd $(TOOLS_DIRECTORY)
142+
unzip -d $(TOOLS_DIRECTORY) $(TOOLS_DIRECTORY)/protoc-3.19.4-osx-x86_64.zip
143+
144+
#-----------------------------------------------------------------------------------------------------------------------
145+
# Generate HTML documentation
146+
# Run this target only in poetry shell
147+
# The generated html pages are in $(CURRDIR)/docs/_build
148+
# ----------------------------------------------------------------------------------------------------------------------
149+
.PHONY: docs
150+
docs: ## Generate doc
151+
cd $(CURRDIR)/docs; \
152+
poetry run sphinx-build -b html . _build
153+
154+
# ----------------------------------------------------------------------------------------------------------------------
155+
# Startup cluster members via docker compose
156+
# ----------------------------------------------------------------------------------------------------------------------
157+
.PHONY: test-cluster-startup
158+
test-cluster-startup: $(BUILD_PROPS) ## Startup any test cluster members using docker-compose
159+
cd tests/utils && docker-compose -f docker-compose-2-members.yaml up -d
160+
161+
# ----------------------------------------------------------------------------------------------------------------------
162+
# Shutdown any cluster members via docker compose
163+
# ----------------------------------------------------------------------------------------------------------------------
164+
.PHONY: test-cluster-shutdown
165+
test-cluster-shutdown: ## Shutdown any test cluster members using docker-compose
166+
cd tests/utils && docker-compose -f docker-compose-2-members.yaml down || true
167+
168+
169+
# ----------------------------------------------------------------------------------------------------------------------
170+
# Startup standalone coherence via java -jar
171+
# ----------------------------------------------------------------------------------------------------------------------
172+
.PHONY: test-coherence-startup
173+
test-coherence-startup: ## Startup standalone cluster
174+
scripts/startup-clusters.sh $(TEST_LOGS_DIR) $(CLUSTER_PORT) $(COHERENCE_GROUP_ID) ${COHERENCE_VERSION}
175+
@echo "Clusters started up"
176+
177+
# ----------------------------------------------------------------------------------------------------------------------
178+
# Shutdown coherence via java -jar
179+
# ----------------------------------------------------------------------------------------------------------------------
180+
.PHONY: test-coherence-shutdown
181+
test-coherence-shutdown: ## shutdown standalone cluster
182+
@ps -ef | grep shutMeDownPlease | grep -v grep | awk '{print $$2}' | xargs kill -9 || true
183+
@echo "Clusters shutdown"
184+
185+
# ----------------------------------------------------------------------------------------------------------------------
186+
# wait for 30 seconds
187+
# ----------------------------------------------------------------------------------------------------------------------
188+
.PHONY: just-wait
189+
just-wait: ## sleep for 30 seconds
190+
@echo "Sleep for 30 seconds"
191+
sleep 30
192+
193+
# ----------------------------------------------------------------------------------------------------------------------
194+
# Remove docker images
195+
# ----------------------------------------------------------------------------------------------------------------------
196+
.PHONY: remove-app-images
197+
remove-app-images: ## Remove docker images
198+
@echo "Remove docker images"
199+
docker image rmi $(TEST_APPLICATION_IMAGE_1) $(TEST_APPLICATION_IMAGE_2) || true

0 commit comments

Comments
 (0)