Skip to content

Conversation

@shirshanka
Copy link
Contributor

Summary

This PR clarifies Kafka topic retention requirements in the documentation, addressing common questions from users deploying DataHub in environments with Kafka retention restrictions.

Changes

  • Mark MetadataGraphEvent_v4 as deprecated (schema removed in 2021, PR fix(model): Remove unused PDL from pre-nocode days #3659)
  • Fix duplicate MetadataGraphEvent_v4 entry in kafka-config.md
  • Add description for PlatformEvent_v1 topic
  • Update DataHubUpgradeHistory_v1 retention guidance:
    • Recommend 7-30 days instead of infinite retention
    • Clarify that infinite retention is not required when DATAHUB_SYSTEM_UPDATE_WAIT_FOR_SYSTEM_UPDATE=false
    • Explain that GMS only reads the last message on startup, not the entire history

Context

This PR addresses questions raised in the Slack community about Kafka topic retention requirements for production deployments. Users deploying DataHub at scale often face platform team restrictions on Kafka retention policies. The documentation previously suggested infinite retention for DataHubUpgradeHistory_v1, which is not actually required in most configurations.

Key Findings:

DataHubUpgradeHistory_v1:

  • GMS seeks to the last message on startup (end offset - 1) to verify the upgrade version
  • If the topic is empty, GMS starts normally when DATAHUB_SYSTEM_UPDATE_WAIT_FOR_SYSTEM_UPDATE=false (which is the default in many configs)
  • 7-30 day retention is sufficient for most deployments

MetadataGraphEvent_v4:

Testing

  • Documentation changes only, no code changes
  • Markdown formatting validated by pre-commit hooks

Related Links

- Mark MetadataGraphEvent_v4 as deprecated (removed in 2021, PR #3659)
- Fix duplicate MetadataGraphEvent_v4 entry in kafka-config.md
- Add description for PlatformEvent_v1
- Update DataHubUpgradeHistory_v1 retention guidance:
  - Recommend 7-30 days instead of infinite retention
  - Clarify that infinite retention is not required when DATAHUB_SYSTEM_UPDATE_WAIT_FOR_SYSTEM_UPDATE=false
  - GMS only reads the last message on startup, not the entire history

These changes help users deploying DataHub in environments with Kafka retention
restrictions while maintaining system reliability.
@github-actions github-actions bot added the docs Issues and Improvements to docs label Oct 30, 2025
@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Oct 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Issues and Improvements to docs needs-review Label for PRs that need review from a maintainer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants