Skip to content

Commit 139c1b7

Browse files
metric: enhance mo-tool and optimize file service metrics (#23253)
### **User description** ## What type of PR is this? - [ ] API-change - [ ] BUG - [x] Improvement - [ ] Documentation - [ ] Feature - [ ] Test and CI - [ ] Code Refactoring ## Which issue(s) this PR fixes: issue #23246 ## What this PR does / why we need it: ```shell ./mo-tool dashboard create --mode local --port 3001 ./mo-tool dashboard list --mode cloud --port 3000 ./mo-tool dashboard delete --mode local --port 3001 # Delete all dashboards in folder ./mo-tool dashboard delete-dashboard --uid <uid> # Delete single dashboard by UID ./mo-tool dashboard delete-folder --mode local --port 3001 # Delete entire folder and all dashboards ``` <img width="1028" height="314" alt="image" src="https://github.com/user-attachments/assets/f7468043-ee48-45e3-b653-01dc09864b27" /> <img width="1465" height="283" alt="image" src="https://github.com/user-attachments/assets/a64e23e0-06d5-438d-a51f-1840378f040d" /> <img width="1473" height="288" alt="image" src="https://github.com/user-attachments/assets/7514a117-a5f8-457f-94d0-d049f9d0d7f2" /> ___ ### **PR Type** Enhancement ___ ### **Description** - Add comprehensive dashboard management commands (list, delete, delete-dashboard, delete-folder) - Implement S3 connection pool tracking with active connection gauge metric - Add disk cache eviction and error tracking metrics - Enhance HTTP trace metrics with connection reuse and failure rate analysis - Support configurable MaxConnsPerHost for S3 object storage connections ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Dashboard Commands"] -->|create/list/delete| B["Grafana API"] C["HTTP Client"] -->|track connections| D["S3ConnActiveGauge"] E["Disk Cache"] -->|evict/error events| F["Cache Metrics"] G["HTTP Trace"] -->|connection lifecycle| H["Connection Pool Metrics"] I["Config"] -->|MaxConnsPerHost| C ``` <details><summary><h3>File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>8 files</summary><table> <tr> <td><strong>main.go</strong><dd><code>Refactor dashboard command into subcommands with full CRUD operations</code></dd></td> <td><a href="https://github.com/matrixorigin/matrixone/pull/23253/files#diff-ee6866f9fb95784ecffcdd1a9dbac5e896f6a743467c18a07fdd5c5cda357e7c">+314/-3</a>&nbsp; </td> </tr> <tr> <td><strong>disk_cache.go</strong><dd><code>Add disk cache eviction and error counter metrics</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/matrixorigin/matrixone/pull/23253/files#diff-2775f0987c87dd3548487f327c4b4f0b8a93e6adbea6fb51b8c9aeaddb543985">+7/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>http_client.go</strong><dd><code>Implement active connection tracking and periodic cleanup</code></dd></td> <td><a href="https://github.com/matrixorigin/matrixone/pull/23253/files#diff-2e7f85fbe5d0260fc0ce5c05baabf885a5c1a77ff1621db750c6bb87001cc719">+150/-4</a>&nbsp; </td> </tr> <tr> <td><strong>http_trace.go</strong><dd><code>Enhance HTTP trace with connection reuse and failure tracking</code></dd></td> <td><a href="https://github.com/matrixorigin/matrixone/pull/23253/files#diff-849e1f1fbb030fb73508f10940ae8d74cef2191e7e28b732805ccf06e3b36eef">+50/-2</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>grafana_dashboard.go</strong><dd><code>Implement dashboard list, delete, and folder deletion operations</code></dd></td> <td><a href="https://github.com/matrixorigin/matrixone/pull/23253/files#diff-b11e4d02a1c8126d2afb936cff26c8d95feaedeaea7173bef747c3f645ade1df">+317/-0</a>&nbsp; </td> </tr> <tr> <td><strong>grafana_dashboard_fs.go</strong><dd><code>Add cache evict/error row and enhance S3 connection metrics </code><br><code>visualization</code></dd></td> <td><a href="https://github.com/matrixorigin/matrixone/pull/23253/files#diff-150a2d0e9e7884f246192255bceb432b8eff0f74015e938db8e21f9bf0b6d6d8">+197/-32</a></td> </tr> <tr> <td><strong>fileservice.go</strong><dd><code>Define S3 active connection and disk cache error metrics</code>&nbsp; </dd></td> <td><a href="https://github.com/matrixorigin/matrixone/pull/23253/files#diff-40c4045e01211017f7ffa2d3be743d46f32abff373c9fc5ec5e4a5fee0eb4ae6">+40/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>metrics.go</strong><dd><code>Register new connection pool and disk cache metrics</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/matrixorigin/matrixone/pull/23253/files#diff-97ea305cc79979fa7db0a8fd618613cdda1c9bda32e6aa91847d8a754de671f3">+4/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>object_storage_arguments.go</strong><dd><code>Add MaxConnsPerHost configuration parameter for S3</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/matrixorigin/matrixone/pull/23253/files#diff-8fbae320ed4d3771224b3ea411e82bdc12c00823f6a8ffd713bb75e5c26987c8">+6/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>Makefile</strong><dd><code>Add list and delete dashboard make targets with documentation</code></dd></td> <td><a href="https://github.com/matrixorigin/matrixone/pull/23253/files#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52">+86/-8</a>&nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>DEV_README.md</strong><dd><code>Document dashboard list and delete operations with examples</code></dd></td> <td><a href="https://github.com/matrixorigin/matrixone/pull/23253/files#diff-71ef0b6c2617dd81654dca538af01d544e9cdfdb92239bf702e26f874427ca91">+37/-4</a>&nbsp; &nbsp; </td> </tr> </table></details></td></tr></tbody></table> </details> ___ --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
1 parent ccb29ae commit 139c1b7

File tree

11 files changed

+1205
-53
lines changed

11 files changed

+1205
-53
lines changed

Makefile

Lines changed: 86 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -371,21 +371,37 @@ dev-help:
371371
@echo " make dev-edit-common - Edit common configuration (all services)"
372372
@echo " make dev-setup-docker-mirror - Configure Docker registry mirror (for faster pulls)"
373373
@echo ""
374-
@echo "Dashboard Creation (via mo-tool):"
375-
@echo " make dev-create-dashboard - Create dashboard (default: local mode, port 3001)"
376-
@echo " make dev-create-dashboard-local - Create local dashboard (port 3001)"
377-
@echo " make dev-create-dashboard-cluster - Create cluster dashboard (port 3000, docker compose)"
378-
@echo " make dev-create-dashboard-k8s - Create K8S dashboard"
379-
@echo " make dev-create-dashboard-cloud-ctrl - Create cloud control-plane dashboard"
374+
@echo "Dashboard Management (via mo-tool):"
375+
@echo " Create:"
376+
@echo " make dev-create-dashboard - Create dashboard (default: local mode, port 3001)"
377+
@echo " make dev-create-dashboard-local - Create local dashboard (port 3001)"
378+
@echo " make dev-create-dashboard-cluster - Create cluster dashboard (port 3000, docker compose)"
379+
@echo " make dev-create-dashboard-k8s - Create K8S dashboard"
380+
@echo " make dev-create-dashboard-cloud-ctrl - Create cloud control-plane dashboard"
381+
@echo " List:"
382+
@echo " make dev-list-dashboard - List dashboards (default: local mode, port 3001)"
383+
@echo " make dev-list-dashboard-local - List local dashboards (port 3001)"
384+
@echo " make dev-list-dashboard-cluster - List cluster dashboards (port 3000)"
385+
@echo " make dev-list-dashboard-k8s - List K8S dashboards"
386+
@echo " make dev-list-dashboard-cloud-ctrl - List cloud control-plane dashboards"
387+
@echo " Delete:"
388+
@echo " make dev-delete-dashboard - Delete all dashboards in folder (default: local mode, port 3001)"
389+
@echo " make dev-delete-dashboard-local - Delete all local dashboards (port 3001)"
390+
@echo " make dev-delete-dashboard-cluster - Delete all cluster dashboards (port 3000)"
391+
@echo " make dev-delete-dashboard-k8s - Delete all K8S dashboards"
392+
@echo " make dev-delete-dashboard-cloud-ctrl - Delete all cloud control-plane dashboards"
380393
@echo ""
381394
@echo " Custom options:"
382395
@echo " DASHBOARD_PORT=3001 make dev-create-dashboard # Custom port"
383396
@echo " DASHBOARD_HOST=localhost make dev-create-dashboard # Custom host"
384397
@echo " DASHBOARD_MODE=cloud make dev-create-dashboard # Custom mode"
385398
@echo ""
386399
@echo " Or use mo-tool directly:"
387-
@echo " ./mo-tool dashboard --mode local --port 3001"
388-
@echo " ./mo-tool dashboard --mode cloud --port 3000"
400+
@echo " ./mo-tool dashboard create --mode local --port 3001"
401+
@echo " ./mo-tool dashboard list --mode cloud --port 3000"
402+
@echo " ./mo-tool dashboard delete --mode local --port 3001 # Delete all dashboards in folder"
403+
@echo " ./mo-tool dashboard delete-dashboard --uid <uid> # Delete single dashboard by UID"
404+
@echo " ./mo-tool dashboard delete-folder --mode local --port 3001 # Delete entire folder and all dashboards"
389405
@echo ""
390406
@echo "Examples:"
391407
@echo " make dev-build && make dev-up # Build and start"
@@ -790,6 +806,68 @@ dev-create-dashboard-k8s:
790806
dev-create-dashboard-cloud-ctrl:
791807
@$(MAKE) DASHBOARD_MODE=cloud-ctrl DASHBOARD_PORT=$(DASHBOARD_PORT) dev-create-dashboard
792808

809+
# List dashboards
810+
.PHONY: dev-list-dashboard
811+
dev-list-dashboard: mo-tool
812+
@echo "Listing dashboards..."
813+
@echo " Mode: $(DASHBOARD_MODE)"
814+
@echo " Host: $(DASHBOARD_HOST):$(DASHBOARD_PORT)"
815+
@echo " Username: $(DASHBOARD_USERNAME)"
816+
@./mo-tool dashboard list \
817+
--host $(DASHBOARD_HOST) \
818+
--port $(DASHBOARD_PORT) \
819+
--mode $(DASHBOARD_MODE) \
820+
--username $(DASHBOARD_USERNAME) \
821+
--password $(DASHBOARD_PASSWORD) \
822+
--datasource $(DASHBOARD_DATASOURCE)
823+
824+
.PHONY: dev-list-dashboard-local
825+
dev-list-dashboard-local:
826+
@$(MAKE) DASHBOARD_MODE=local DASHBOARD_PORT=3001 dev-list-dashboard
827+
828+
.PHONY: dev-list-dashboard-cluster
829+
dev-list-dashboard-cluster:
830+
@$(MAKE) DASHBOARD_MODE=cloud DASHBOARD_PORT=3000 DASHBOARD_DATASOURCE=Prometheus dev-list-dashboard
831+
832+
.PHONY: dev-list-dashboard-k8s
833+
dev-list-dashboard-k8s:
834+
@$(MAKE) DASHBOARD_MODE=k8s DASHBOARD_PORT=$(DASHBOARD_PORT) dev-list-dashboard
835+
836+
.PHONY: dev-list-dashboard-cloud-ctrl
837+
dev-list-dashboard-cloud-ctrl:
838+
@$(MAKE) DASHBOARD_MODE=cloud-ctrl DASHBOARD_PORT=$(DASHBOARD_PORT) dev-list-dashboard
839+
840+
# Delete dashboards
841+
.PHONY: dev-delete-dashboard
842+
dev-delete-dashboard: mo-tool
843+
@echo "Deleting dashboards..."
844+
@echo " Mode: $(DASHBOARD_MODE)"
845+
@echo " Host: $(DASHBOARD_HOST):$(DASHBOARD_PORT)"
846+
@echo " Username: $(DASHBOARD_USERNAME)"
847+
@./mo-tool dashboard delete \
848+
--host $(DASHBOARD_HOST) \
849+
--port $(DASHBOARD_PORT) \
850+
--mode $(DASHBOARD_MODE) \
851+
--username $(DASHBOARD_USERNAME) \
852+
--password $(DASHBOARD_PASSWORD) \
853+
--datasource $(DASHBOARD_DATASOURCE)
854+
855+
.PHONY: dev-delete-dashboard-local
856+
dev-delete-dashboard-local:
857+
@$(MAKE) DASHBOARD_MODE=local DASHBOARD_PORT=3001 dev-delete-dashboard
858+
859+
.PHONY: dev-delete-dashboard-cluster
860+
dev-delete-dashboard-cluster:
861+
@$(MAKE) DASHBOARD_MODE=cloud DASHBOARD_PORT=3000 DASHBOARD_DATASOURCE=Prometheus dev-delete-dashboard
862+
863+
.PHONY: dev-delete-dashboard-k8s
864+
dev-delete-dashboard-k8s:
865+
@$(MAKE) DASHBOARD_MODE=k8s DASHBOARD_PORT=$(DASHBOARD_PORT) dev-delete-dashboard
866+
867+
.PHONY: dev-delete-dashboard-cloud-ctrl
868+
dev-delete-dashboard-cloud-ctrl:
869+
@$(MAKE) DASHBOARD_MODE=cloud-ctrl DASHBOARD_PORT=$(DASHBOARD_PORT) dev-delete-dashboard
870+
793871
###############################################################################
794872
# Local Development with MinIO Storage
795873
###############################################################################

0 commit comments

Comments
 (0)