docs(kafka): clarify Kafka topic retention requirements #15149
      
        
          +7
        
        
          −8
        
        
          
        
      
    
  
  Add this suggestion to a batch that can be applied as a single commit.
  This suggestion is invalid because no changes were made to the code.
  Suggestions cannot be applied while the pull request is closed.
  Suggestions cannot be applied while viewing a subset of changes.
  Only one suggestion per line can be applied in a batch.
  Add this suggestion to a batch that can be applied as a single commit.
  Applying suggestions on deleted lines is not supported.
  You must change the existing code in this line in order to create a valid suggestion.
  Outdated suggestions cannot be applied.
  This suggestion has been applied or marked resolved.
  Suggestions cannot be applied from pending reviews.
  Suggestions cannot be applied on multi-line comments.
  Suggestions cannot be applied while the pull request is queued to merge.
  Suggestion cannot be applied right now. Please check back later.
  
    
  
    
Summary
This PR clarifies Kafka topic retention requirements in the documentation, addressing common questions from users deploying DataHub in environments with Kafka retention restrictions.
Changes
MetadataGraphEvent_v4as deprecated (schema removed in 2021, PR fix(model): Remove unused PDL from pre-nocode days #3659)MetadataGraphEvent_v4entry in kafka-config.mdPlatformEvent_v1topicDataHubUpgradeHistory_v1retention guidance:DATAHUB_SYSTEM_UPDATE_WAIT_FOR_SYSTEM_UPDATE=falseContext
This PR addresses questions raised in the Slack community about Kafka topic retention requirements for production deployments. Users deploying DataHub at scale often face platform team restrictions on Kafka retention policies. The documentation previously suggested infinite retention for
DataHubUpgradeHistory_v1, which is not actually required in most configurations.Key Findings:
DataHubUpgradeHistory_v1:
DATAHUB_SYSTEM_UPDATE_WAIT_FOR_SYSTEM_UPDATE=false(which is the default in many configs)MetadataGraphEvent_v4:
Testing
Related Links