Deduplication with Cloud PubSub and Cloud Dataflow on Google Cloud Platform

This is the source code that accompanies the solution: Deduplication of messages with Cloud PubSub and Cloud Dataflow. This sample code demonstrates three approaches for deduplication:

• PubSubIO: com.google.examples.dfdedup.DedupWithPubSubIO
• Distinct transform: com.google.examples.dfdedup.DedupWithDistinct
• Custom state based deduplication: com.google.examples.dfdedup.DedupWithStateAndGC

End to end pipeline

You can run the following end to end pipeline to explore deduplication behavior across all three approaches:

Setting up resources

NOTE: If you're new to GCP, please see quickstarts for Cloud PubSub, BigQuery and Cloud Dataflow

BigQuery

Use the schema files under bqschemas/ to create

Cloud PubSub

Running Python-based the data generator

Blah blah