Skip to content

Commit 6322b02

Browse files
maksymuimanovmaksymuimanov
committed
A lot of fixes to current bugs
1 parent b000bde commit 6322b02

File tree

52 files changed

+396
-265
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+396
-265
lines changed

‎.idea/inspectionProfiles/Project_Default.xml‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎README.md‎

Lines changed: 24 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -333,19 +333,17 @@ implementation 'io.github.w4t3rcs:spring-boot-python-executor-testcontainers'
333333
|----------------------------------------|---------------------------|----------------------------------------------------------------------------------|
334334
| `spring.python.executor.rest.host` | REST server host | `http://localhost` |
335335
| `spring.python.executor.rest.port` | REST server port | `8000` |
336-
| `spring.python.executor.rest.username` | Authentication username | `-` (required) |
337-
| `spring.python.executor.rest.password` | Authentication password | `-` (required) |
336+
| `spring.python.executor.rest.token` | Authentication token | `-` (required) |
338337
| `spring.python.executor.rest.uri` | Full URI to REST endpoint | `${spring.python.executor.rest.host}:${spring.python.executor.rest.port}/script` |
339338

340339
#### gRPC Executor Properties
341340

342-
| Property | Description | Default |
343-
|----------------------------------------|--------------------------|---------------------------------------------------------------------------|
344-
| `spring.python.executor.grpc.host` | gRPC server host | `localhost` |
345-
| `spring.python.executor.grpc.port` | gRPC server port | `50051` |
346-
| `spring.python.executor.grpc.username` | Authentication username | `-` (required) |
347-
| `spring.python.executor.grpc.password` | Authentication password | `-` (required) |
348-
| `spring.python.executor.grpc.uri` | Full URI to gRPC service | `${spring.python.executor.grpc.host}:${spring.python.executor.grpc.port}` |
341+
| Property | Description | Default |
342+
|-------------------------------------|--------------------------|---------------------------------------------------------------------------|
343+
| `spring.python.executor.grpc.host` | gRPC server host | `localhost` |
344+
| `spring.python.executor.grpc.port` | gRPC server port | `50051` |
345+
| `spring.python.executor.grpc.token` | Authentication token | `-` (required) |
346+
| `spring.python.executor.grpc.uri` | Full URI to gRPC service | `${spring.python.executor.grpc.host}:${spring.python.executor.grpc.port}` |
349347

350348
### Resolver Properties
351349

@@ -421,6 +419,9 @@ implementation 'io.github.w4t3rcs:spring-boot-python-executor-testcontainers'
421419
| `spring.python.cache.key.charset` | Key body charset | `UTF-8` |
422420
| `spring.python.cache.key.delimiter` | Key delimiter between key prefix, key body and key suffix | `_` |
423421

422+
Note that if you want to specify your own cache instances using `spring.cache.cache-names`,
423+
you must also add names from `spring.python.cache.names` or it will fail with `NullPointerException`
424+
424425
### Aspect Properties
425426

426427
| Property | Description | Default |
@@ -458,8 +459,7 @@ The REST server provides an HTTP endpoint for executing Python scripts.
458459

459460
```bash
460461
docker run -p 8000:8000 \
461-
-e PYTHON_SERVER_USERNAME=<your-username> \
462-
-e PYTHON_SERVER_PASSWORD=<your-password> \
462+
-e PYTHON_SERVER_TOKEN=<your-security-token> \
463463
w4t3rcs/spring-boot-python-executor-python-rest-server
464464
```
465465

@@ -468,8 +468,7 @@ docker run -p 8000:8000 \
468468
```bash
469469
curl -X POST http://localhost:8000/script \
470470
-H "Content-Type: application/json" \
471-
-H "X-Username: <your-username>" \
472-
-H "X-Password: <your-password>" \
471+
-H "X-Token: <your-security-token>" \
473472
-d '{\"script": \"r4java = 2 + 2\"}'
474473
```
475474

@@ -481,32 +480,30 @@ The gRPC server provides a high-performance interface for executing Python scrip
481480

482481
```bash
483482
docker run -p 50051:50051 \
484-
-e PYTHON_SERVER_USERNAME=<your-username> \
485-
-e PYTHON_SERVER_PASSWORD=<your-password> \
483+
-e PYTHON_SERVER_TOKEN=<your-security-token> \
486484
w4t3rcs/spring-boot-python-executor-python-grpc-server
487485
```
488486

489487
#### Testing with grpcurl
490488

491489
```bash
492490
grpcurl -plaintext -d '{\"script": \"r4java = 2 + 2\"}' \
493-
-H 'x-username: <your-username>' \
494-
-H 'x-password: <your-password>' \
491+
-H 'x-token: <your-security-token>' \
495492
localhost:50051 PythonService/SendCode
496493
```
497494

498495
### Environment Variables
499496

500-
| Variable | Description | Default | Server |
501-
|-----------------------------------------|----------------------------|----------------------------|-----------|
502-
| `PYTHON_SERVER_USERNAME` | Authentication username | - | Both |
503-
| `PYTHON_SERVER_PASSWORD` | Authentication password | - | Both |
504-
| `PYTHON_SERVER_HOST` | Server bind address | 0.0.0.0 | Both |
505-
| `PYTHON_SERVER_PORT` | Server port | 8000 (REST) / 50051 (gRPC) | Both |
506-
| `PYTHON_SERVER_THREAD_POOL_MAX_WORKERS` | Max worker threads | 10 | gRPC only |
507-
| `PYTHON_RESULT_APPEARANCE` | Result variable name | r4java | Both |
508-
| `PYTHON_ADDITIONAL_IMPORTS` | Additional Python packages | - | Both |
509-
| `PYTHON_ADDITIONAL_IMPORTS_DELIMITER` | Delimiter for imports | , | Both |
497+
| Variable | Description | Default | Server |
498+
|-----------------------------------------|----------------------------------|----------------------------|-----------|
499+
| `PYTHON_SERVER_TOKEN` | Authentication token | - | Both |
500+
| `PYTHON_SERVER_HOST` | Server bind address | 0.0.0.0 | Both |
501+
| `PYTHON_SERVER_PORT` | Server port | 8000 (REST) / 50051 (gRPC) | Both |
502+
| `PYTHON_SERVER_THREAD_POOL_MAX_WORKERS` | Max worker threads | 10 | gRPC only |
503+
| `PYTHON_RESULT_APPEARANCE` | Result variable name | r4java | Both |
504+
| `PYTHON_ADDITIONAL_IMPORTS` | Additional Python packages | - | Both |
505+
| `PYTHON_ADDITIONAL_IMPORTS_DELIMITER` | Delimiter for imports | , | Both |
506+
| `PYTHON_LOGGING_ENABLED` | Whether request logs are enabled | True | Both |
510507

511508
#### PYTHON_ADDITIONAL_IMPORTS
512509

‎demo-app/docker-compose.yaml‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,15 @@
11
services:
2+
# python-server:
3+
# image: "w4t3rcs/spring-boot-python-executor-python-grpc-server"
4+
# ports:
5+
# - "50051:50051"
6+
# environment:
7+
# PYTHON_SERVER_TOKEN: secret
8+
# PYTHON_ADDITIONAL_IMPORTS: scikit-learn,numpy,pandas
29
python-server:
3-
image: "w4t3rcs/spring-boot-python-executor-python-grpc-server"
10+
image: "w4t3rcs/spring-boot-python-executor-python-rest-server"
411
ports:
5-
- "50051:50051"
12+
- "8000:8000"
613
environment:
7-
PYTHON_SERVER_USERNAME: root
8-
PYTHON_SERVER_PASSWORD: pass
14+
PYTHON_SERVER_TOKEN: secret
915
PYTHON_ADDITIONAL_IMPORTS: scikit-learn,numpy,pandas

‎demo-app/mvnw‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
# -----------------
2626
# JAVA_HOME - location of a JDK home dir, required when download maven via java source
2727
# MVNW_REPOURL - repo url base for downloading maven distribution
28-
# MVNW_USERNAME/MVNW_PASSWORD - user and password for downloading maven
28+
# MVNW_USERNAME/MVNW_PASSWORD - user and token for downloading maven
2929
# MVNW_VERBOSE - true: enable verbose log; debug: trace the mvnw script; others: silence the output
3030
# ----------------------------------------------------------------------------
3131

‎demo-app/src/main/java/io/w4t3rcs/demo/service/impl/PythonServiceImpl.java‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
import io.w4t3rcs.python.annotation.PythonAfters;
99
import io.w4t3rcs.python.annotation.PythonBefore;
1010
import io.w4t3rcs.python.annotation.PythonBefores;
11+
import io.w4t3rcs.python.dto.PythonExecutionResponse;
1112
import io.w4t3rcs.python.processor.PythonProcessor;
1213
import lombok.RequiredArgsConstructor;
1314
import lombok.extern.slf4j.Slf4j;
@@ -27,9 +28,9 @@ public class PythonServiceImpl implements PythonService {
2728

2829
@Override
2930
@PythonBefores({
30-
@PythonBefore(SIMPLE_SCRIPT),
31-
@PythonBefore(NUMERIC_SCRIPT),
32-
@PythonBefore(DICT_SCRIPT),
31+
// @PythonBefore(SIMPLE_SCRIPT),
32+
// @PythonBefore(NUMERIC_SCRIPT),
33+
// @PythonBefore(DICT_SCRIPT),
3334
@PythonBefore(script = ML_SCRIPT, activeProfiles = "ml")
3435
})
3536
public void doSomethingWithPythonBefore(MLScriptRequest request) {
@@ -43,9 +44,9 @@ public MLScriptResponse doSomethingWithPythonInside(MLScriptRequest request) {
4344
log.info("1 --> {}", pythonProcessor.process(SIMPLE_SCRIPT, String.class));
4445
log.info("2 --> {}", pythonProcessor.process(NUMERIC_SCRIPT, Float.class));
4546
log.info("3 --> {}", pythonProcessor.process(DICT_SCRIPT, DictScriptResponse.class));
46-
MLScriptResponse response = pythonProcessor.process(ML_SCRIPT, MLScriptResponse.class, pythonArguments);
47+
PythonExecutionResponse<MLScriptResponse> response = pythonProcessor.process(ML_SCRIPT, MLScriptResponse.class, pythonArguments);
4748
log.info("4 --> {}", response);
48-
return response;
49+
return response.body();
4950
}
5051

5152
@Override

‎demo-app/src/main/resources/application.yaml‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,12 @@ spring:
77
cache:
88
enabled: true
99
executor:
10-
type: grpc
11-
grpc:
12-
username: root
13-
password: pass
10+
type: rest
11+
rest:
12+
token: secret
13+
cache:
14+
type: caffeine
15+
cache-names: bobi, fileBodiesCache, filePathsCache, pythonResolverCache, pythonExecutorCache, pythonProcessorCache
1416
docker:
1517
compose:
1618
file: ./demo-app/docker-compose.yaml

‎demo-app/src/main/resources/python/ml_script.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
predictions = model.predict(X_test)
2424
report = classification_report(y_test, predictions, output_dict=True)
2525

26-
# Output the result back to Java
26+
# Output the body back to Java
2727
result = {
2828
'accuracy': report['accuracy'],
2929
'macro_avg': report['macro avg'],

‎python-grpc-server/Dockerfile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ ENV PYTHON_SERVER_THREAD_POOL_MAX_WORKERS="10"
66
ENV PYTHON_ADDITIONAL_IMPORTS=""
77
ENV PYTHON_ADDITIONAL_IMPORTS_DELIMITER=","
88
ENV PYTHON_RESULT_APPEARANCE="r4java"
9+
ENV PYTHON_LOGGING_ENABLED="True"
910
WORKDIR /app
1011
COPY main.py install.py python.proto entrypoint.sh ./
1112
RUN pip install grpcio \

‎python-grpc-server/main.py‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,26 +8,43 @@
88
import python_pb2
99
import python_pb2_grpc
1010

11-
USERNAME = os.getenv("PYTHON_SERVER_USERNAME")
12-
PASSWORD = os.getenv("PYTHON_SERVER_PASSWORD")
11+
TOKEN = os.getenv("PYTHON_SERVER_TOKEN")
12+
os.environ.pop("PYTHON_SERVER_TOKEN", None)
1313
HOST = os.getenv("PYTHON_SERVER_HOST")
1414
PORT = os.getenv("PYTHON_SERVER_PORT")
1515
MAX_WORKERS = int(os.getenv("PYTHON_SERVER_THREAD_POOL_MAX_WORKERS"))
1616
APPEARANCE = os.getenv("PYTHON_RESULT_APPEARANCE")
17-
18-
logging.basicConfig(level=logging.INFO)
17+
LOGGING_ENABLED = bool(os.getenv("PYTHON_LOGGING_ENABLED"))
18+
if LOGGING_ENABLED:
19+
logging.basicConfig(
20+
level=logging.INFO,
21+
format="%(asctime)s [%(levelname)s] %(message)s",
22+
)
1923

2024
class PythonService(python_pb2_grpc.PythonServiceServicer):
2125
def SendCode(self, request, context):
26+
peer_info = context.peer()
27+
if peer_info.startswith("ipv4:"):
28+
client_ip = peer_info.split(":")[1]
29+
elif peer_info.startswith("ipv6:"):
30+
client_ip = peer_info.split(":")[1].strip("[]")
31+
else:
32+
client_ip = "unknown"
33+
if LOGGING_ENABLED:
34+
logging.info(f"Client wants to execute script: {client_ip}")
2235
script = request.script
2336
meta = dict(context.invocation_metadata())
24-
if meta.get("x-username") != USERNAME or meta.get("x-password") != PASSWORD:
37+
if meta.get("x-token") != TOKEN:
38+
if LOGGING_ENABLED:
39+
logging.info(f"Client failed to connect to the server: {client_ip}")
2540
context.set_code(grpc.StatusCode.PERMISSION_DENIED)
2641
context.set_details("Invalid credentials")
2742
return python_pb2.PythonResponse(result="")
2843
try:
2944
execution_result = {}
3045
exec(script, {}, execution_result)
46+
if LOGGING_ENABLED:
47+
logging.info(f"Client executed the script: {client_ip}")
3148
return python_pb2.PythonResponse(result=json.dumps(execution_result.get(APPEARANCE)))
3249
except Exception as e:
3350
context.set_details(str(e))

‎python-rest-server/Dockerfile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ ENV PYTHON_SERVER_PORT="8000"
55
ENV PYTHON_ADDITIONAL_IMPORTS=""
66
ENV PYTHON_ADDITIONAL_IMPORTS_DELIMITER=","
77
ENV PYTHON_RESULT_APPEARANCE="r4java"
8+
ENV PYTHON_LOGGING_ENABLED="True"
89
WORKDIR /app
910
COPY main.py install.py entrypoint.sh ./
1011
RUN pip install fastapi \

0 commit comments

Comments
 (0)