Skip to content

Commit 5f57f31

Browse files
XuPeng-SHXuPeng-SH
authored
enhance mo-diag tool (#22630)
enhance mo-diag tool Approved by: @LeftHandCold
1 parent 800a555 commit 5f57f31

File tree

6 files changed

+1309
-348
lines changed

6 files changed

+1309
-348
lines changed

‎clients/python/README.md‎

Lines changed: 71 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -475,8 +475,24 @@ MO-DIAG[test]> show_indexes ivf_health_demo_docs
475475
- Rows: 17
476476
- Compressed Size: 3.09 KB
477477
- Original Size: 6.83 KB
478+
479+
*************************** 3. row ***************************
480+
Index Name: idx_embedding_ivf_v2
481+
Algorithm: ivfflat
482+
Table Type: entries
483+
Physical Table: __mo_index_secondary_0199e725-0a7c-77f4-8d0b-48fd8258098a
484+
Columns: embedding
485+
Statistics:
486+
- Objects: 1
487+
- Rows: 1,000
488+
- Compressed Size: 156.34 KB
489+
- Original Size: 176.07 KB
490+
491+
Total: 3 index tables (1 ivfflat with 3 physical tables)
478492
```
479493

494+
**Note**: IVF indexes have 3 physical tables (metadata, centroids, entries)
495+
480496
**`show_all_indexes [database]`**
481497
- Health report for all tables with secondary indexes in the database
482498
- Row count consistency checks
@@ -486,13 +502,22 @@ MO-DIAG[test]> show_indexes ivf_health_demo_docs
486502
MO-DIAG[test]> show_all_indexes
487503
488504
📊 Index Health Report for Database 'test':
489-
═══════════════════════════════════════════════════════════════════════════
505+
========================================================================================================================
490506
491-
🟢 HEALTHY TABLES (1):
492-
─────────────────────────────────────────────────────────────────────────
507+
✓ HEALTHY (3 tables)
508+
------------------------------------------------------------------------------------------------------------------------
493509
Table Name | Indexes | Row Count | Notes
494-
──────────────────────────────────────────────────────────────────────────
495-
ivf_health_demo_docs | 1 | ✓ 1000 rows | IVF: 17 centroids, 1000 vectors
510+
------------------------------------------------------------------------------------------------------------------------
511+
cms_all_content_chunk_info | 3 | ✓ 32,712 rows | -
512+
demo_mixed_indexes | 4 | ✓ 20 rows | -
513+
ivf_health_demo_docs | 1 | 300 rows | IVF: 17 centroids, 300 vectors
514+
515+
========================================================================================================================
516+
Summary:
517+
✓ 3 healthy tables
518+
Total: 3 tables with indexes
519+
520+
💡 Tip: Use 'verify_counts <table>' or 'show_ivf_status' for detailed diagnostics
496521
```
497522

498523
**`verify_counts <table> [database]`**
@@ -540,24 +565,49 @@ ivf_health_demo_docs | idx_embedding_ivf_v2 | embedding
540565
```
541566
MO-DIAG[test]> show_table_stats ivf_health_demo_docs -a -d
542567
568+
📊 Detailed Table Statistics for 'test.ivf_health_demo_docs':
569+
======================================================================================================================================================
570+
543571
Table: ivf_health_demo_docs
544572
Objects: 1 | Rows: 1,000 | Null: 0 | Original: 176.03 KB | Compressed: 156.24 KB
545573
546574
Objects:
547-
Object Name | Rows | Null Cnt | Original Size | Compressed Size
548-
─────────────────────────────────────────────────────────────────────────────────────────────────────
549-
0199e729-642e-71e0-b338-67c4980ee294_00000 | 1000 | 0 | 176.03 KB | 156.24 KB
575+
Object Name | Rows | Null Cnt | Original Size | Compressed Size
576+
----------------------------------------------------------------------------------------------------------------------------------------------------
577+
0199e729-642e-71e0-b338-67c4980ee294_00000 | 1000 | 0 | 176.03 KB | 156.24 KB
550578
551579
Index: idx_embedding_ivf_v2
552-
└─ Physical Table (metadata): __mo_index_secondary_0199e725-0a7a-77b8-b689-ccdd0a33f581
580+
└─ (metadata): __mo_index_secondary_0199e725-0a7a-77b8-b689-ccdd0a33f581:272851
553581
Objects: 1 | Rows: 7 | Null: 0 | Original: 1.98 KB | Compressed: 940 B
554582
555583
Objects:
556-
Object Name | Rows | Null Cnt | Original Size | Compressed Size
557-
─────────────────────────────────────────────────────────────────────────────────────────────────────
558-
0199e729-642a-7d36-ac37-0ae17325f7ec_00000 | 7 | 0 | 1.98 KB | 940 B
584+
Object Name | Rows | Null Cnt | Original Size | Compressed Size
585+
----------------------------------------------------------------------------------------------------------------------------------------------------
586+
0199e729-642a-7d36-ac37-0ae17325f7ec_00000 | 7 | 0 | 1.98 KB | 940 B
587+
588+
└─ (centroids): __mo_index_secondary_0199e725-0a7b-706e-8f0a-a50edc3621a1:272852
589+
Objects: 1 | Rows: 17 | Null: 0 | Original: 6.83 KB | Compressed: 3.09 KB
590+
591+
Objects:
592+
Object Name | Rows | Null Cnt | Original Size | Compressed Size
593+
----------------------------------------------------------------------------------------------------------------------------------------------------
594+
0199e729-6431-794d-9c13-d10f2e4a59e7_00000 | 17 | 0 | 6.83 KB | 3.09 KB
595+
596+
└─ (entries): __mo_index_secondary_0199e725-0a7c-77f4-8d0b-48fd8258098a:272853
597+
Objects: 1 | Rows: 1,000 | Null: 0 | Original: 696.11 KB | Compressed: 626.37 KB
598+
599+
Objects:
600+
Object Name | Rows | Null Cnt | Original Size | Compressed Size
601+
----------------------------------------------------------------------------------------------------------------------------------------------------
602+
0199e729-6438-7db1-bcb6-7b74c59b0aee_00000 | 1000 | 0 | 696.11 KB | 626.37 KB
559603
```
560604

605+
**Note**:
606+
- Hierarchical structure: Table → Index → Physical Tables → Objects
607+
- IVF index has 3 physical tables: metadata (7 rows), centroids (17 rows), entries (1,000 rows)
608+
- Physical table name format: `table_name:table_id`
609+
- Use `-t` flag to include tombstone statistics for deleted/updated data
610+
561611
#### Database Operations
562612

563613
**`flush_table <table> [database]`**
@@ -573,15 +623,21 @@ MO-DIAG[test]> flush_table ivf_health_demo_docs
573623
📋 Found 3 index physical tables
574624
575625
Index: idx_embedding_ivf_v2
576-
✓ metadata: __mo_index_secondary_xxx
577-
✓ centroids: __mo_index_secondary_yyy
578-
✓ entries: __mo_index_secondary_zzz
626+
✓ metadata: __mo_index_secondary_0199e725-0a7a-77b8-b689-ccdd0a33f581
627+
✓ centroids: __mo_index_secondary_0199e725-0a7b-706e-8f0a-a50edc3621a1
628+
✓ entries: __mo_index_secondary_0199e725-0a7c-77f4-8d0b-48fd8258098a
579629
580630
📊 Summary:
581631
Main table: ✓ flushed
582632
Index tables: 3/3 flushed successfully
583633
```
584634

635+
**Note**:
636+
- Automatically discovers and flushes all index physical tables
637+
- For IVF indexes: flushes metadata, centroids, and entries tables
638+
- Requires sys user privileges
639+
- Recommended before backups or after bulk operations
640+
585641
**`tables [database]`**
586642
- List all tables in current or specified database
587643

‎clients/python/README_USER.md‎

Lines changed: 71 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -310,8 +310,24 @@ MO-DIAG[test]> show_indexes ivf_health_demo_docs
310310
- Rows: 17
311311
- Compressed Size: 3.09 KB
312312
- Original Size: 6.83 KB
313+
314+
*************************** 3. row ***************************
315+
Index Name: idx_embedding_ivf_v2
316+
Algorithm: ivfflat
317+
Table Type: entries
318+
Physical Table: __mo_index_secondary_0199e725-0a7c-77f4-8d0b-48fd8258098a
319+
Columns: embedding
320+
Statistics:
321+
- Objects: 1
322+
- Rows: 1,000
323+
- Compressed Size: 156.34 KB
324+
- Original Size: 176.07 KB
325+
326+
Total: 3 index tables (1 ivfflat with 3 physical tables)
313327
```
314328

329+
**Note**: IVF indexes have 3 physical tables (metadata, centroids, entries)
330+
315331
**`show_all_indexes [database]`**
316332
- Health report for all tables with secondary indexes in the database
317333
- Row count consistency checks
@@ -321,13 +337,22 @@ MO-DIAG[test]> show_indexes ivf_health_demo_docs
321337
MO-DIAG[test]> show_all_indexes
322338
323339
📊 Index Health Report for Database 'test':
324-
═══════════════════════════════════════════════════════════════════════════
340+
========================================================================================================================
325341
326-
🟢 HEALTHY TABLES (1):
327-
─────────────────────────────────────────────────────────────────────────
342+
✓ HEALTHY (3 tables)
343+
------------------------------------------------------------------------------------------------------------------------
328344
Table Name | Indexes | Row Count | Notes
329-
──────────────────────────────────────────────────────────────────────────
330-
ivf_health_demo_docs | 1 | ✓ 1000 rows | IVF: 17 centroids, 1000 vectors
345+
------------------------------------------------------------------------------------------------------------------------
346+
cms_all_content_chunk_info | 3 | ✓ 32,712 rows | -
347+
demo_mixed_indexes | 4 | ✓ 20 rows | -
348+
ivf_health_demo_docs | 1 | 300 rows | IVF: 17 centroids, 300 vectors
349+
350+
========================================================================================================================
351+
Summary:
352+
✓ 3 healthy tables
353+
Total: 3 tables with indexes
354+
355+
💡 Tip: Use 'verify_counts <table>' or 'show_ivf_status' for detailed diagnostics
331356
```
332357

333358
**`verify_counts <table> [database]`**
@@ -375,24 +400,49 @@ ivf_health_demo_docs | idx_embedding_ivf_v2 | embedding
375400
```
376401
MO-DIAG[test]> show_table_stats ivf_health_demo_docs -a -d
377402
403+
📊 Detailed Table Statistics for 'test.ivf_health_demo_docs':
404+
======================================================================================================================================================
405+
378406
Table: ivf_health_demo_docs
379407
Objects: 1 | Rows: 1,000 | Null: 0 | Original: 176.03 KB | Compressed: 156.24 KB
380408
381409
Objects:
382-
Object Name | Rows | Null Cnt | Original Size | Compressed Size
383-
─────────────────────────────────────────────────────────────────────────────────────────────────────
384-
0199e729-642e-71e0-b338-67c4980ee294_00000 | 1000 | 0 | 176.03 KB | 156.24 KB
410+
Object Name | Rows | Null Cnt | Original Size | Compressed Size
411+
----------------------------------------------------------------------------------------------------------------------------------------------------
412+
0199e729-642e-71e0-b338-67c4980ee294_00000 | 1000 | 0 | 176.03 KB | 156.24 KB
385413
386414
Index: idx_embedding_ivf_v2
387-
└─ Physical Table (metadata): __mo_index_secondary_0199e725-0a7a-77b8-b689-ccdd0a33f581
415+
└─ (metadata): __mo_index_secondary_0199e725-0a7a-77b8-b689-ccdd0a33f581:272851
388416
Objects: 1 | Rows: 7 | Null: 0 | Original: 1.98 KB | Compressed: 940 B
389417
390418
Objects:
391-
Object Name | Rows | Null Cnt | Original Size | Compressed Size
392-
─────────────────────────────────────────────────────────────────────────────────────────────────────
393-
0199e729-642a-7d36-ac37-0ae17325f7ec_00000 | 7 | 0 | 1.98 KB | 940 B
419+
Object Name | Rows | Null Cnt | Original Size | Compressed Size
420+
----------------------------------------------------------------------------------------------------------------------------------------------------
421+
0199e729-642a-7d36-ac37-0ae17325f7ec_00000 | 7 | 0 | 1.98 KB | 940 B
422+
423+
└─ (centroids): __mo_index_secondary_0199e725-0a7b-706e-8f0a-a50edc3621a1:272852
424+
Objects: 1 | Rows: 17 | Null: 0 | Original: 6.83 KB | Compressed: 3.09 KB
425+
426+
Objects:
427+
Object Name | Rows | Null Cnt | Original Size | Compressed Size
428+
----------------------------------------------------------------------------------------------------------------------------------------------------
429+
0199e729-6431-794d-9c13-d10f2e4a59e7_00000 | 17 | 0 | 6.83 KB | 3.09 KB
430+
431+
└─ (entries): __mo_index_secondary_0199e725-0a7c-77f4-8d0b-48fd8258098a:272853
432+
Objects: 1 | Rows: 1,000 | Null: 0 | Original: 696.11 KB | Compressed: 626.37 KB
433+
434+
Objects:
435+
Object Name | Rows | Null Cnt | Original Size | Compressed Size
436+
----------------------------------------------------------------------------------------------------------------------------------------------------
437+
0199e729-6438-7db1-bcb6-7b74c59b0aee_00000 | 1000 | 0 | 696.11 KB | 626.37 KB
394438
```
395439

440+
**Note**:
441+
- Hierarchical structure: Table → Index → Physical Tables → Objects
442+
- IVF index has 3 physical tables: metadata (7 rows), centroids (17 rows), entries (1,000 rows)
443+
- Physical table name format: `table_name:table_id`
444+
- Use `-t` flag to include tombstone statistics for deleted/updated data
445+
396446
#### Database Operations
397447

398448
**`flush_table <table> [database]`**
@@ -408,15 +458,21 @@ MO-DIAG[test]> flush_table ivf_health_demo_docs
408458
📋 Found 3 index physical tables
409459
410460
Index: idx_embedding_ivf_v2
411-
✓ metadata: __mo_index_secondary_xxx
412-
✓ centroids: __mo_index_secondary_yyy
413-
✓ entries: __mo_index_secondary_zzz
461+
✓ metadata: __mo_index_secondary_0199e725-0a7a-77b8-b689-ccdd0a33f581
462+
✓ centroids: __mo_index_secondary_0199e725-0a7b-706e-8f0a-a50edc3621a1
463+
✓ entries: __mo_index_secondary_0199e725-0a7c-77f4-8d0b-48fd8258098a
414464
415465
📊 Summary:
416466
Main table: ✓ flushed
417467
Index tables: 3/3 flushed successfully
418468
```
419469

470+
**Note**:
471+
- Automatically discovers and flushes all index physical tables
472+
- For IVF indexes: flushes metadata, centroids, and entries tables
473+
- Requires sys user privileges
474+
- Recommended before backups or after bulk operations
475+
420476
**`tables [database]`**
421477
- List all tables in current or specified database
422478

0 commit comments

Comments
 (0)