Merge pull request #423 from vprashar2929/add-marklint

feat(ci): Add markdownlint workflow

Author: sunya-ch

.github/pull_request_template.md

@@ -1,11 +1,7 @@
-<!--
-Pull Request Template
--->
+# Checklist for PR Author
 
 ---
 
-### Checklist for PR Author
-
 In addition to approval, the author must confirm the following check list:
 
 - [ ] Run the following command to format your code:

.github/workflows/lint.yml

@@ -0,0 +1,30 @@
+name: Run linters and formatters
+
+on:
+  pull_request:
+
+jobs:
+  markdown-lint:
+    runs-on: ubuntu-latest
+    steps:
+      # checkout soruce code
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      # setup Python environment
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.10"
+
+      # install hatch
+      - name: Install hatch
+        run: |
+          python -m pip install --upgrade pip
+          pip install hatch
+
+      # scan for markdown linting errors
+      - name: Run pymarkdownlnt on markdown files
+        shell: bash
+        run: |
+          make lint

Makefile

@@ -15,6 +15,10 @@ PYTHON  := python3.10
 DOCKERFILES_PATH := ./dockerfiles
 MODEL_PATH := ${PWD}/tests/models
 
+.PHONY: lint
+lint:
+	@hatch run pymarkdownlnt scan -r .
+
 .PHONY: build
 build:
 	$(CTR_CMD) build -t $(IMAGE) -f $(DOCKERFILES_PATH)/Dockerfile .

README.md

@@ -1,11 +1,14 @@
 # Kepler Power Model
+
 [Get started with Kepler Model Server.](https://sustainable-computing.io/kepler_model_server/get_started/)
 
 This repository contains source code related to Kepler power model. The modules in this repository connects to [core Kepler project](https://github.com/sustainable-computing-io/kepler) and [kepler-model-db](https://github.com/sustainable-computing-io/kepler-model-db) as below.
-![](./fig/comm_diagram.png)
+
+![Diagram](./fig/comm_diagram.png)
+
 For more details, check [the component diagram](./fig/model-server-components-simplified.png).
 
-## Model server and estimator deployment 
+## Model server and estimator deployment
 
 ### Using Kepler Operator
 
@@ -28,25 +31,32 @@ spec:
         initUrl: <static model URL>
 ```
 
-### Using manifests with setup script:
+### Using manifests with setup script
+
 Deploy with estimator sidecar
-```sh
+
+```bash
 OPTS="ESTIMATOR" make deploy
 ```
 
-Deploy with estimator sidecar and model server 
-```sh
+Deploy with estimator sidecar and model server
+
+```bash
 OPTS="ESTIMATOR SERVER" make deploy
 ```
 
 ## Model Training
+
 - [Use Tekton pipeline](./model_training/tekton/README.md)
 - [Use Bash script with CPE operator](./model_training/cpe_script_instruction.md)
 
 ## Local test
-### via docker
-1. Build image for testing, run 
-    ```sh
+
+### Via docker
+
+1. Build image for testing, run
+
+    ```bash
     make build-test
     ```
 
@@ -61,27 +71,29 @@ OPTS="ESTIMATOR SERVER" make deploy
 
     For more test information, check [here](./tests/).
 
-### with native python environment
+### With native python environment
+
 Compatible version: `python 3.10`
 
 1. Install [`hatch`](https://hatch.pypa.io/latest/install/)
 2. Prepare environment
 
     ```bash
-		hatch shell
+    hatch shell
     ```
 
 3. Run the test
 
-    |Test case|Command|
-    |---|---|
-    |[Training pipeline](./tests/README.md#pipeline)|python -u ./tests/pipeline_test.py|
-    |[Model server](./tests/README.md#estimator-model-request-to-model-server)|Terminal 1: export MODEL_PATH=$(pwd)/tests/models;python src/server/model_server.py <br>Terminal 2: python -u tests/estimator_model_request_test.py|
-    |[Estimator](./tests/README.md#estimator-power-request-from-collector)|Terminal 1: python src/estimate/estimator.py<br>Terminal 2: python -u tests/estimator_power_request_test.py|
-    |Estimator with Model Server|Terminal 1: export MODEL_PATH=$(pwd)/tests/models;python src/server/model_server.py <br>Terminal 2: export MODEL_SERVER_URL=http://localhost:8100;export MODEL_SERVER_ENABLE=true;python -u src/estimate/estimator.py<br>Terminal 3: python -u tests/estimator_power_request_test.py
-    |[Offline Trainer](./tests/README.md#offline-trainer)|Terminal 1: python src/train/offline_trainer.py<br>Terminal 2: python -u tests/offline_trainer_test.py|
+| Test case                   | Command                                                                                                                                                                                                                                                                        |
+|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Training pipeline](./tests/README.md#pipeline)| python -u ./tests/pipeline_test.py                                                                                                                                                                                                                                             |
+| [Model server](./tests/README.md#estimator-model-request-to-model-server)| Terminal 1: export MODEL_PATH=$(pwd)/tests/models;python src/server/model_server.py Terminal 2: python -u tests/estimator_model_request_test.py                                                                                                                                |
+| [Estimator](./tests/README.md#estimator-power-request-from-collector) | Terminal 1: python src/estimate/estimator.py Terminal 2: python -u tests/estimator_power_request_test.py                                                                                                                                                                       |
+| Estimator with Model Server | Terminal 1: export MODEL_PATH=$(pwd)/tests/models;python src/server/model_server.py Terminal 2: export MODEL_SERVER_URL=<http://localhost:8100>;export MODEL_SERVER_ENABLE=true;python -u src/estimate/estimator.py Terminal 3: python -u tests/estimator_power_request_test.py |
+| [Offline Trainer](./tests/README.md#offline-trainer) | Terminal 1: python src/train/offline_trainer.py Terminal 2: python -u tests/offline_trainer_test.py                                                                                                                                                                            |
 
-    For more test information, check [here](./tests/).
+  For more test information, check [here](./tests/).
 
 ### Contributing
+
 Please check the roadmap and guidelines to join us [here](./contributing.md).

contributing.md

@@ -1,58 +1,71 @@
 # Contributing
+
 [Get started with Kepler Model Server.](https://sustainable-computing.io/kepler_model_server/get_started/)
 
 - The main source codes are in [src directory](./src/).
 
 ## PR Hands-on
 
-- Create related [issue](https://github.com/sustainable-computing-io/kepler-model-server/issues) with your name assigned first (if not exist). 
+- Create related [issue](https://github.com/sustainable-computing-io/kepler-model-server/issues) with your name assigned first (if not exist).
 
 - Set required secret and environment for local repository test if needed. Check below table.
 
-Objective|Required Secret|Required Environment
----|---|---
-Push to private repo|BOT_NAME, BOT_TOKEN|IMAGE_REPO
-Change on base image|BOT_NAME, BOT_TOKEN|IMAGE_REPO
-Save data/models to AWS COS|AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_REGION|
+| Objective | Required Secret | Required Environment |
+| --------- | --------------- |----------------------|
+| Push to private repo |BOT_NAME, BOT_TOKEN | IMAGE_REPO |
+| Change on base image | BOT_NAME, BOT_TOKEN | IMAGE_REPO |
+| Save data/models to AWS COS | AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_REGION | |
 
 ## Improve components in training pipelines
+
 Learn more details about [Training Pipeline](https://sustainable-computing.io/kepler_model_server/pipeline/)
 
 ### Introduce new feature group
+
 - Define new feature group name `FeatureGroup` and update metric list map `FeatureGroups` in [train types](./src/util/train_types.py)
 
 ### Introduce new energy sources
+
 - Define new energy source map `PowerSourceMap` in [train types](./src/util/train_types.py)
 
 ### Improve preprocessing method
+
 - [extractor](./src/train/extractor/): convert from numerically aggregated metrics to per-second value
 - [isolator](./src/train/isolator/): isolate background (idle) power from the collected power
 
 ### Introduce new learning method
+
 - [trainer](./src/train/trainer/): apply learning method to build a model using extracted data and isolated data
 
 ## Model training
+
 Learn more details about [model training](./model_training/)
 
 ### Introduce new benchmarks
+
 The new benchmark must be supported by [CPE operator](https://github.com/IBM/cpe-operator) for automation.
 Find [examples](https://github.com/IBM/cpe-operator/tree/main/examples).
 
 ### CPE-based (deprecated)
+
 `Benchmark` CR has a dependency on `BenchmarkOperator`. Default `BechmarkOperator` is to support [batch/v1/Job API](https://github.com/IBM/cpe-operator/blob/main/examples/none/cpe_v1_none_operator.yaml).
 
 ### Tekton
+
 Create workload `Task` and provide example `Pipeline` to run.
 
 ### Add new trained models
+
 TBD
 
 ## Source improvement
+
 Any improvement in `src` and `cmd`.
 
 ## Test and CI improvement
+
 Any improvement in `tests`, `dockerfiles`, `manifests` and `.github/workflows`
 
 ## Documentation
 
-Detailed documentation should be posted to [kepler-doc](https://github.com/sustainable-computing-io/kepler-doc) repository.
+Detailed documentation should be posted to [kepler-doc](https://github.com/sustainable-computing-io/kepler-doc) repository.

docs/developer/README.md

@@ -1,16 +1,19 @@
-### 0. Temporarily add `__init__.py` to all directories
+# Developer Guide
 
-```
+- Temporarily add `__init__.py` to all directories
+
+```bash
 find ./src -type d -exec touch {}/__init__.py \;
 ```
 
-### 1. Generate `classes.plantuml` and `packages.plantuml` using the following commands
-```
+- Generate `classes.plantuml` and `packages.plantuml` using the following commands
+
+```bash
 pyreverse --colorized --output plantuml --module-names y --show-stdlib --show-associated 2  --show-ancestors 1 --verbose -d umls/server/ --source-roots ./src/ ./src/server/
 pyreverse --colorized --output plantuml --module-names y --show-stdlib --show-associated 2  --show-ancestors 1 --verbose -d umls/estimate/ --source-roots ./src/ ./src/estimate/
 pyreverse --colorized --output plantuml --module-names y --show-stdlib --show-associated 2  --show-ancestors 1 --verbose -d umls/train/ --source-roots ./src/ ./src/train/
 pyreverse --colorized --output plantuml --module-names y --show-stdlib --show-associated 2  --show-ancestors 1 --verbose -d umls/train/trainer/ --source-roots ./src/ ./src/train/trainer/
 ```
 
-### 2. Use [plantuml](https://plantuml.com/download) to convert planuml files  to `svg` files
+- Use [plantuml](https://plantuml.com/download) to convert planuml files  to `svg` files
 NeoVim plugin `neovim-soil` was used to generate svg files from plantuml files

model_training/README.md

@@ -1,6 +1,7 @@
 # Contribute to power profiling and model training
 
 <!--toc:start-->
+
 - [Contribute to power profiling and model training](#contribute-to-power-profiling-and-model-training)
   - [Requirements](#requirements)
   - [Pre-step](#pre-step)
@@ -10,8 +11,8 @@
     - [For managed cluster](#for-managed-cluster)
     - [Run benchmark and collect metrics](#run-benchmark-and-collect-metrics)
     - [With manual execution](#with-manual-execution)
-    - [[Manual Metric Collection and Training with Entrypoint](./cmd_instruction.md)](#manual-metric-collection-and-training-with-entrypointcmdinstructionmd)
   - [Clean up](#clean-up)
+
 <!--toc:end-->
 
 ## Requirements
@@ -68,7 +69,7 @@ There are two options to run the benchmark and collect the metrics, [CPE-operato
 
 - [CPE Operator Instruction](./cpe_script_instruction.md)
 
-With manual execution
+### With manual execution
 
 In addition to the above two automation approach, you can manually run your own benchmarks, then collect, train, and export the models by the entrypoint `cmd/main.py`
 

model_training/cmd_instruction.md

@@ -1,6 +1,7 @@
 # Manual Metric Collection and Training with Entrypoint
 
 ## 1. Collect metrics
+
 Without benchmark/pipeline automation, kepler metrics can be collected by `query` function by setting `BENCHMARK`, `PROM_URL`, `COLLECT_ID` and either one of the following time options.
 
 > It is recommend to set BENCHMARK name as a part of the pod name such as `stressng` to filter the validated results. BENCHMARK name will be also used by the TrainerIsolator to filter the target pods. If the BENCHMARK cannot be used to filter the target pods, the validated results will show result from all pods.
@@ -32,8 +33,10 @@ INTERVAL= # in second
 DATAPATH=/path/to/workspace python cmd/main.py query --benchmark $BENCHMARK --server $PROM_URL --output kepler_query --interval $INTERVAL --id $COLLECT_ID
 ```
 
-### Output:
+### Output
+
 There will three files created in the `/path/to/workspace`, those are:
+
 - `kepler_query.json`: raw prometheus query response
 - `<COLLECT_ID>.json`: machine system features (spec)
 - `<BENCHMARK>.json`: an item contains startTimeUTC and endTimeUTC
@@ -50,6 +53,7 @@ DATAPATH=/path/to/workspace MODEL_PATH=/path/to/workspace python cmd/main.py tra
 ```
 
 ## 3. Export models
+
 Export function is to archive the model that has an error less than threshold from the trained pipeline and make a report in the format that is ready to push to kepler-model-db. To use export function, need to set `EXPORTER_PATH` and `PUBLISHER`, and collect date option.
 
 ```bash
@@ -81,4 +85,3 @@ COLLECT_DATE= # collect date
 # require PIPELINE_NAME from train step
 DATAPATH=/path/to/workspace MODEL_PATH=/path/to/workspace python cmd/main.py export --pipeline-name $PIPELINE_NAME -o $EXPORT_PATH --publisher $PUBLISHER --zip=true --collect-date $COLLECT_DATE
 ```
-