Skip to content

Comments

HDDS-13884. Populate block file metadata with helper fields#9263

Draft
ptlrs wants to merge 1 commit intoapache:masterfrom
ptlrs:HDDS-13884-Populate-block-file-metadata-with-helper-fields
Draft

HDDS-13884. Populate block file metadata with helper fields#9263
ptlrs wants to merge 1 commit intoapache:masterfrom
ptlrs:HDDS-13884-Populate-block-file-metadata-with-helper-fields

Conversation

@ptlrs
Copy link
Contributor

@ptlrs ptlrs commented Nov 7, 2025

What changes were proposed in this pull request?

Saving the following fields as block file metadata can help with debugging and improve the visibility into the system.

  • volume/bucket/key name
  • parentId/objectId
  • Original key creation timestamp

Storing such information can help answer questions about any arbitrary block file such as

  • For which key was this block file created?
  • Regardless of replication/ EC reconstruction/OS filesystem timestamps , when was this block file originally created for the very first time?

Today some of these questions can be answered only if the chain of metadata from OM to Datanodes is intact. Others can't be answered at all.

This PR:

  1. Saves metadata to block files for both Ratis and EC replicated files.
  2. The IO clients supply the extra metadata that cannot be retrieved on the datanode side easily.
  3. The datanode saves the information to the block file metadata table in RocksDB

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13884

How was this patch tested?

New tests added

CI: https://github.com/ptlrs/ozone/actions/runs/19176103214

rpatel@Mac DS-db9b617d-a5c6-4c90-b1f8-481710227e98 % ~/Github/ozone/hadoop-ozone/dist/target/ozone-2.2.0-SNAPSHOT/bin/ozone debug ldb --db container.db scan --cf block_data
{ "1|115816896921600001": {
  "blockID" : {
    "containerBlockID" : {
      "containerID" : 1,
      "localID" : 115816896921600001
    },
    "blockCommitSequenceId" : 2
  },
  "metadata" : {
    "BUCKET_NAME" : "obs-27a9b2b4-edda-428f-b9f7-828444858936",
    "CREATION_TIME" : "2025-11-08T04:51:59.172Z",
    "KEY_NAME" : "ca3490e7-04e7-44d9-8a92-096b2e1ee948",
    "OBJECT_ID" : "-9223372036854775040",
    "PARENT_OBJECT_ID" : "0",
    "TYPE" : "KEY",
    "VOLUME_NAME" : "b04c6e09-5128-4204-8da6-ca5c9072afe7"
  },
  "chunkList" : {
    "memoizedHashCode" : 0,
    "memoizedSize" : -1,
    "unknownFields" : {
      "fields" : { }
    },
    "bitField0_" : 15,
    "chunkName_" : {
      "hash" : 0,
      "bytes" : "MTE1ODE2ODk2OTIxNjAwMDAxX2NodW5rXzE="
    },
    "offset_" : 0,
    "len_" : 45,
    "metadata_" : [ ],
    "checksumData_" : {
      "memoizedHashCode" : 0,
      "memoizedSize" : -1,
      "unknownFields" : {
        "fields" : { }
      },
      "bitField0_" : 3,
      "type_" : 2,
      "bytesPerChecksum_" : 16384,
      "checksums_" : [ {
        "hash" : 0,
        "bytes" : "5X5F/Q=="
      } ],
      "memoizedIsInitialized" : 1
    },
    "stripeChecksum_" : {
      "hash" : 0,
      "bytes" : ""
    },
    "memoizedIsInitialized" : 1
  },
  "size" : 45
}
 }
rpatel@Mac DS-db9b617d-a5c6-4c90-b1f8-481710227e98 % pwd
/Users/rpatel/Github/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-7acd6bfe-c321-40a4-a886-885c46a0b278/ozone-metadata/datanode-7/data-0/hdds/7acd6bfe-c321-40a4-a886-885c46a0b278/DS-db9b617d-a5c6-4c90-b1f8-481710227e98

@swamirishi swamirishi self-requested a review November 10, 2025 17:40
@github-actions
Copy link

github-actions bot commented Dec 2, 2025

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

@github-actions github-actions bot added the stale label Dec 2, 2025
@ptlrs
Copy link
Contributor Author

ptlrs commented Dec 3, 2025

Hi @swamirishi @errose28, can you take a look at this PR?

@github-actions github-actions bot removed the stale label Dec 4, 2025
@github-actions
Copy link

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

@github-actions github-actions bot added the stale label Dec 25, 2025
@github-actions
Copy link

github-actions bot commented Jan 1, 2026

Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.

@github-actions github-actions bot closed this Jan 1, 2026
@ptlrs
Copy link
Contributor Author

ptlrs commented Jan 8, 2026

Hi @errose28 @adoroszlai can you please reopen this PR and review it?

@adoroszlai adoroszlai reopened this Jan 9, 2026
@adoroszlai adoroszlai requested a review from errose28 January 9, 2026 17:23
@github-actions github-actions bot removed the stale label Jan 10, 2026
@github-actions
Copy link

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

@github-actions github-actions bot added the stale label Jan 31, 2026
@ptlrs
Copy link
Contributor Author

ptlrs commented Feb 1, 2026

Not stale

@github-actions github-actions bot removed the stale label Feb 2, 2026
@github-actions
Copy link

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

@github-actions github-actions bot added the stale label Feb 23, 2026
@ptlrs
Copy link
Contributor Author

ptlrs commented Feb 23, 2026

Not stale

@github-actions github-actions bot removed the stale label Feb 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants