YSQL: Add `attributes (jsonb)` to `yb_tablet_metadata` by keisku · Pull Request #30294 · yugabyte/yugabyte-db

keisku · 2026-02-12T07:33:13Z

Testing

Upgrade ysql.

./bin/yb-ctl --rf 3 create
./build/latest/bin/yb-admin upgrade_ysql

Able to see active_ssts_size and wals_size in the yb_tablet_metadata.

./bin/ysqlsh -c "\d yb_tablet_metadata"
            View "pg_catalog.yb_tablet_metadata"
     Column      |  Type   | Collation | Nullable | Default
-----------------+---------+-----------+----------+---------
 tablet_id       | text    |           |          |
 oid             | oid     |           |          |
 db_name         | text    |           |          |
 relname         | text    |           |          |
 start_hash_code | integer |           |          |
 end_hash_code   | integer |           |          |
 leader          | text    |           |          |
 replicas        | text[]  |           |          |
 attributes      | jsonb   |           |          |

Run this right after creating a cluster

./bin/ysqlsh -c "
SELECT *
FROM yb_tablet_metadata
WHERE relname = 'test_sizes';
"
 tablet_id | oid | db_name | relname | start_hash_code | end_hash_code | leader | replicas | attributes
-----------+-----+---------+---------+-----------------+---------------+--------+----------+------------
(0 rows)

Create a test table

./bin/ysqlsh -c "CREATE TABLE test_sizes (k INT PRIMARY KEY, v TEXT);"
CREATE TABLE

See values in attributes.

./bin/ysqlsh -P pager=off -c "
SELECT *
FROM yb_tablet_metadata
WHERE relname = 'test_sizes';
"
            tablet_id             |  oid  | db_name  |  relname   | start_hash_code | end_hash_code |     leader     |                    replicas                    |                                                                                                    attributes
----------------------------------+-------+----------+------------+-----------------+---------------+----------------+------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 5fdb14d4b4f547729f3d18153910137b | 16384 | yugabyte | test_sizes |               0 |         65536 | 127.0.0.1:5433 | {127.0.0.1:5433,127.0.0.2:5433,127.0.0.3:5433} | {"replicas": {"127.0.0.1:5433": {"wal_sizes": 1048576, "active_sst_sizes": 0}, "127.0.0.2:5433": {"wal_sizes": 1048576, "active_sst_sizes": 0}, "127.0.0.3:5433": {"wal_sizes": 1048576, "active_sst_sizes": 0}}}
(1 row)

Checked the UI.

Insert 29000 rows.

./bin/ysqlsh -c "
INSERT INTO test_sizes
SELECT g, repeat('x', 100000)
FROM generate_series(1001, 30000) g;
"
INSERT 0 29000

active_ssts_size and wals_size in the attributes increased.

./bin/ysqlsh -P pager=off -c "
SELECT tablet_id, attributes
FROM yb_tablet_metadata
WHERE relname = 'test_sizes';
"
            tablet_id             |                                                                                                            attributes
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 140d3712996b4b3182d9fc1f897a3fbc | {"replicas": {"127.0.0.1:5433": {"wal_sizes": 0, "active_sst_sizes": 0}, "127.0.0.2:5433": {"wal_sizes": 2902590966, "active_sst_sizes": 139142511}, "127.0.0.3:5433": {"wal_sizes": 2902590911, "active_sst_sizes": 139142511}}}
 da0d353355d6449da1a10714fcbc6c54 | {"replicas": {"127.0.0.1:5433": {"wal_sizes": 0, "active_sst_sizes": 0}, "127.0.0.2:5433": {"wal_sizes": 2902590966, "active_sst_sizes": 139142511}, "127.0.0.3:5433": {"wal_sizes": 2902590911, "active_sst_sizes": 139142511}}}
(2 rows)

Checked the UI.

Note

Medium Risk
Changes the yb_get_tablet_metadata system function/view signature and adds a new migration, which could impact clients or tests that depend on the previous column set. Also extends master-to-pggate protobuf plumbing for per-replica size metrics, so mismatched versions or edge cases around missing metrics could surface.

Overview
Adds an attributes (jsonb) column to yb_tablet_metadata (and underlying yb_get_tablet_metadata() record) that returns per-replica active_sst_sizes and wal_sizes keyed by replica address.

Plumbs replica size data from master tablet metadata through protobuf (TabletStatusPB), pggate descriptors, and Postgres JSONB construction, and includes a new YSQL migration plus updated regression tests and docs/examples to reflect the new column.

^{Written by Cursor Bugbot for commit 79d4847. This will update automatically on new commits. Configure here.}

netlify · 2026-02-12T07:34:31Z

✅ Deploy Preview for infallible-bardeen-164bc9 ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`79d4847`
🔍 Latest deploy log	https://app.netlify.com/projects/infallible-bardeen-164bc9/deploys/69b245e722b6c20007bd2dfb
😎 Deploy Preview	https://deploy-preview-30294--infallible-bardeen-164bc9.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

jasonyb

docs/content/stable/explore/observability/yb-tablet-metadata.md:

diff --git a/docs/content/stable/explore/observability/yb-tablet-metadata.md b/docs/content/stable/explore/observability/yb-tablet-metadata.md
index 112465c01b..9963e1a468 100644
--- a/docs/content/stable/explore/observability/yb-tablet-metadata.md
+++ b/docs/content/stable/explore/observability/yb-tablet-metadata.md
@@ -37,6 +37,8 @@ The following table describes the columns of the `yb_tablet_metadata` view.
 | end_hash_code | int | Ending hash code (exclusive) for the tablet. (NULL for range-sharded tables.) |
 | leader | text | IP address, port of the leader node for the tablet. |
 | replicas | text[] | A list of replica IP addresses and port (includes leader) associated with the tablet. |
+| active_ssts_size | bigint[] | Per-replica SST files size in bytes, in the same order as replicas. |
+| wals_size | bigint[] | Per-replica WAL files size in bytes, in the same order as replicas. |
 
 ## Examples

yb.port.rules.out

diff --git a/src/postgres/src/test/regress/expected/yb.orig.tablet_metadata.out b/src/postgres/src/test/regress/expected/yb.orig.tablet_metadata.out
index b6981e2f4f..29f230dc80 100644
--- a/src/postgres/src/test/regress/expected/yb.orig.tablet_metadata.out
+++ b/src/postgres/src/test/regress/expected/yb.orig.tablet_metadata.out
@@ -15,6 +15,21 @@ ORDER BY start_hash_code NULLS FIRST;
  test_table_1 | yugabyte |           32768 |         65536
 (3 rows)
 
+-- Test that active_ssts_size and wals_size columns are present and are arrays
+SELECT
+    relname,
+    pg_typeof(active_ssts_size) AS sst_type,
+    pg_typeof(wals_size) AS wal_type,
+    array_length(active_ssts_size, 1) = array_length(replicas, 1) AS sizes_match_replicas
+FROM yb_tablet_metadata
+WHERE relname = 'test_table_1'
+ORDER BY start_hash_code NULLS FIRST
+LIMIT 1;
+   relname    | sst_type | wal_type | sizes_match_replicas 
+--------------+----------+----------+----------------------
+ test_table_1 | bigint[] | bigint[] | t
+(1 row)
+
 -- Test that we are able to join with yb_servers()
 SELECT
     ytm.relname,

src/yb/yql/pgwrapper/pg_mini-test.cc

src/postgres/src/include/catalog/pg_proc.dat

src/yb/yql/pggate/ybc_pg_typedefs.h

src/yb/yql/pgwrapper/pg_mini-test.cc

src/yb/tablet/tablet.proto

src/postgres/src/backend/utils/misc/pg_yb_utils.c

hbhanawat · 2026-02-13T05:31:17Z

@keisku Instead of adding two fields, we would want a single json column for all the tablet attributes. This will ensure that the other tablet properties could be added in future without modifying the schema of the table cc: @IshanChhangani @GauravSingh-yb @jasonyb

keisku · 2026-02-13T06:36:15Z

Sorry there are so many things to review. I appreciate your help.

Instead of adding two fields, we would want a single json column for all the tablet attributes.

I am working on this.

keisku · 2026-02-13T08:11:21Z

@hbhanawat @jasonyb

I've addressed review points and updated the PR title/description accordingly 👍

keisku · 2026-02-26T03:32:18Z

@jasonyb any chance you could take a look at this PR again?

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Free Tier Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable autofix in the Cursor dashboard.}

src/yb/yql/pgwrapper/ysql_migrations/V101__29665__yb_tablet_metadata_sizes.sql

jasonyb

sorry for the wait. I was OOO

jasonyb · 2026-03-03T01:35:10Z

src/postgres/src/backend/utils/misc/pg_yb_utils.c

+			qsort_arg(sort_idxs, nreplicas, sizeof(int),
+					  replica_index_cmp, tablet->replicas);


given jsonb does not preserve ordering, you can leave the replicas code untouched and do the attributes code separately

src/yb/yql/pgwrapper/ysql_migrations/V101__29665__yb_tablet_metadata_sizes.sql

keisku · 2026-03-11T00:24:42Z

@jasonyb Updated the code to address your review points!

jasonyb

You also need to modify src/postgres/src/include/catalog/pg_yb_migration.dat. And the migration number should now be 101 because of a9a4c01.

jasonyb · 2026-03-11T21:14:26Z

src/postgres/src/backend/utils/misc/pg_yb_utils.c

+			/*
+			 * Build JSONB attributes before sorting replicas, because the
+			 * parallel size arrays must stay aligned with the replica order.
+			 * JSONB does not preserve key order so sorting is unnecessary.


What is "parallel size arrays"? I don't see a reason why you need to handle attributes before replicas. The sort is on a local list variable. If there is no strong reason, the order should be the expected values[8] then [9]

Sorry for the confusion. I missed this in my self-review. Fixed order to values[8], values[9] and removed the misleading comment.

Rename active_ssts_size/wals_size to active_sst_sizes/wal_sizes across proto, pggate, catalog manager, system views, and migration. Use uint64 datums instead of CString to match the bigint[] column type. Make the migration resilient to OID changes by looking up yb_get_tablet_metadata by name instead of hardcoded OID. Move size column validation from C++ mini-test to pg_regress, covering array alignment, non-negative sizes, and replica-tserver correspondence.

jasonyb

Please read src/postgres/src/tools/pgindent/README and run yb_pgindent and only keep changes relevant to your changes.

fix test failure:
./yb_build.sh fastdebug --java-test 'org.yb.pgsql.TestPgRegressRules#testPgRegressRules' --gcc12

diff -U3 ${TEST_TMPDIR}/pgregress_output/yb_pg_rules_schedule/expected/yb.port.rules.out ${TEST_TMPDIR}/pgregress_output/yb_pg_rules_schedule/results/yb.port.rules.out
--- ${TEST_TMPDIR}/pgregress_output/yb_pg_rules_schedule/expected/yb.port.rules.out	2026-03-12 23:45:22.191187135 +0000
+++ ${TEST_TMPDIR}/pgregress_output/yb_pg_rules_schedule/results/yb.port.rules.out	2026-03-12 23:45:22.135187232 +0000
@@ -2786,8 +2786,9 @@
     t.start_hash_code,
     t.end_hash_code,
     t.leader,
-    t.replicas
-   FROM ((yb_get_tablet_metadata() t(tablet_id, object_uuid, namespace, object_name, type, start_hash_code, end_hash_code, leader, replicas)
+    t.replicas,
+    t.attributes
+   FROM ((yb_get_tablet_metadata() t(tablet_id, object_uuid, namespace, object_name, type, start_hash_code, end_hash_code, leader, replicas, attributes)
      LEFT JOIN pg_class c ON ((c.relname = t.object_name)))
      LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
   WHERE (((t.namespace = 'system'::text) AND (t.object_name = 'transactions'::text)) OR ((t.type = 'YSQL'::text) AND (n.nspname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name]))));

Also saw many pgvector test failures such as the following, but it is possibly unrelated to your change:
./yb_build.sh fastdebug --cxx-test pgwrapper_pg_vector_index-test --gtest_filter 'PgVectorIndexTest.BootstrapFlushedVectorIndexes/ColocatedPackingV2' --gcc12

I0312 23:42:53.120318  9706 pg_vector_index-test.cc:454] RowsMatch:    Query: SELECT * FROM test AS t  ORDER BY embedding <-> '[0.0, 0.0, 0.0]' LIMIT 5
I0312 23:42:53.209281  9706 pg_vector_index-test.cc:456] RowsMatch:   Result: []
I0312 23:42:53.209326  9706 pg_vector_index-test.cc:457] RowsMatch: Expected: [1, [1, 2, 3], 2, [2, 4, 6], 3, [3, 6, 9], 4, [4, 8, 12], 5, [5, 10, 15]]
I0312 23:42:53.209331  9706 pg_vector_index-test.cc:460] RowsMatch: Wrong number of results: 0, while 5 expected
../../src/yb/yql/pgwrapper/pg_vector_index-test.cc:478: Failure
Value of: RowsMatch(conn, filter, expected, limit)
  Actual: false
Expected: true

jasonyb · 2026-03-12T21:48:01Z

src/postgres/src/backend/utils/misc/pg_yb_utils.c

+					jb_key.val.string.len = 16;
+					pushJsonbValue(&jb_state, WJB_KEY, &jb_key);
+					jb_val.type = jbvNumeric;
+					jb_val.val.numeric = DatumGetNumeric(


Error (S&RX) bad_opening_paren
There should be no linebreak after (

jasonyb · 2026-03-12T21:48:04Z

src/postgres/src/backend/utils/misc/pg_yb_utils.c

+					jb_key.val.string.len = 9;
+					pushJsonbValue(&jb_state, WJB_KEY, &jb_key);
+					jb_val.type = jbvNumeric;
+					jb_val.val.numeric = DatumGetNumeric(


Error (S&RX) bad_opening_paren
There should be no linebreak after (

jasonyb · 2026-03-12T21:48:24Z

src/postgres/src/backend/utils/misc/pg_yb_utils.c

 #endif
 #include "storage/procarray.h"
 #include "tcop/utility.h"
+#include "utils/array.h"


Suggested change

#include "utils/array.h"

keisku force-pushed the issue-29665 branch 2 times, most recently from 5822cce to 0f7c9c3 Compare February 12, 2026 12:22

keisku marked this pull request as ready for review February 12, 2026 12:22

keisku force-pushed the issue-29665 branch 2 times, most recently from 2acbff4 to b463c61 Compare February 12, 2026 22:56

keisku mentioned this pull request Feb 12, 2026

Include tablet size in yb_tablet_metadata #29665

Open

jasonyb reviewed Feb 13, 2026

View reviewed changes

keisku changed the title ~~YSQL: Add {active_ssts,wals}_size to yb_tablet_metadata~~ YSQL: Add attributes (jsonb) to yb_tablet_metadata Feb 13, 2026

keisku force-pushed the issue-29665 branch from ba894ed to 90c4746 Compare February 13, 2026 08:05

keisku force-pushed the issue-29665 branch from 90c4746 to 620364f Compare February 26, 2026 03:31

cursor bot reviewed Feb 26, 2026

View reviewed changes

src/yb/yql/pgwrapper/ysql_migrations/V101__29665__yb_tablet_metadata_sizes.sql Show resolved Hide resolved

jasonyb reviewed Mar 3, 2026

View reviewed changes

keisku force-pushed the issue-29665 branch from 620364f to 75ac6d7 Compare March 11, 2026 00:23

keisku requested a review from jasonyb March 11, 2026 00:24

jasonyb requested changes Mar 11, 2026

View reviewed changes

keisku added 7 commits March 12, 2026 12:59

YSQL: Add {active_ssts,wals}_size to yb_tablet_metadata

23b6d2f

Expose tablet replica attributes as JSONB

e3e742f

Use v100

65c6a25

Simplify replica sorting in yb_get_tablet_metadata

1e2ff4a

Bump tablet metadata migration to V101

2eb26ed

Fix values[] assignment order in yb_tablet_metadata_sizes

79d4847

keisku force-pushed the issue-29665 branch from 75ac6d7 to 79d4847 Compare March 12, 2026 04:49

jasonyb requested changes Mar 13, 2026

View reviewed changes

		qsort_arg(sort_idxs, nreplicas, sizeof(int),
		replica_index_cmp, tablet->replicas);

Conversation

keisku commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for infallible-bardeen-164bc9 ready!

Uh oh!

jasonyb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hbhanawat commented Feb 13, 2026

Uh oh!

keisku commented Feb 13, 2026

Uh oh!

keisku commented Feb 13, 2026

Uh oh!

keisku commented Feb 26, 2026

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jasonyb left a comment

Choose a reason for hiding this comment

Uh oh!

jasonyb Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

keisku commented Mar 11, 2026

Uh oh!

jasonyb left a comment

Choose a reason for hiding this comment

Uh oh!

jasonyb Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

keisku Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

jasonyb left a comment

Choose a reason for hiding this comment

Uh oh!

jasonyb Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

jasonyb Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

jasonyb Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

keisku commented Feb 12, 2026 •

edited

Loading

netlify bot commented Feb 12, 2026 •

edited

Loading