Merge branch 'main' into ishamehramixpanel-patch-5

Author: myronkaifung

.github/workflows/cspell.yaml

@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: streetsidesoftware/cspell-action@v6
+      - uses: streetsidesoftware/cspell-action@v7
         with:
           # Define glob patterns to filter the files to be checked. Use a new line between patterns to define multiple patterns.
           # The default is to check ALL files that were changed in in the pull_request or push.

openapi/src/docs/ingestion/track-event-deduplication.md

@@ -1,7 +1,28 @@
-Event deduplication allows a project to send the same exact event while only recording that event once.
-Deduplication only occurs when a subset of the event data is exactly identical.
+Mixpanel provides an event deduplication mechanism to ensure that duplicate events do not skew your analytics. Deduplication is essential when events may be sent multiple times due to network retries, client-side batching, or integration with multiple data sources.
+
+<br />
+
+## How Deduplication Works
+
+Mixpanel deduplicates events using a combination of four key event properties:
+
+- Event Name (`event`)
+- Distinct ID (`distinct_id`)
+- Timestamp (`time`)
+- Insert ID (`$insert_id`)
+
+If all four of these properties are identical across two or more events, Mixpanel considers them duplicates and will only show the most recent version of that event in your reports. This applies regardless of whether the events are sent via SDKs, APIs, or other integrations. 
+
+The `$insert_id` should be a randomly generated, unique value for each event to ensure proper deduplication. If `$insert_id` are reused, events may be unintentionally deduplicated.
+
+Only the four key event properties listed above are used for deduplication. Additional event properties are not considered for the deduplication mechanism. For example, if two events share the same Event Name, Distinct ID, Timestamp, and Insert ID, but have different $city value, they are still considered duplicate events.
+
+### Deduplication Example
+
+Deduplication occurs when a subset of the event data (event name, distinct_id, timestamp, $insert_id) is identical. Other event properties are not considered.
 
 **Required [Event Object](doc:data-model#anatomy-of-an-event) attributes**
+
 [block:parameters]
 {
   "data": {
@@ -13,6 +34,7 @@ Deduplication only occurs when a subset of the event data is exactly identical.
     "0-2": "A name for the event. For example, \"Signed up\", or \"Uploaded Photo\".",
     "1-0": "**properties**",
     "1-1": "<span style=\"font-family: courier\">Object</span></br><span style=\"color: red\">required</span>",
+    "1-2": "",
     "2-0": "**properties.distinct_id**",
     "2-1": "<span style=\"font-family: courier\">String</span></br><span style=\"color: red\">required</span>",
     "2-2": "The value of `distinct_id` will be treated as a string, and used to uniquely identify a user associated with your event. If you provide a distinct_id property with your events, you can track a given user through funnels and distinguish unique users for retention analyses. You should always send the same distinct_id when an event is triggered by the same user.",
@@ -27,32 +49,56 @@ Deduplication only occurs when a subset of the event data is exactly identical.
     "5-2": "A unique UUID tied to exactly one occurrence of an event."
   },
   "cols": 3,
-  "rows": 6
+  "rows": 6,
+  "align": [
+    "left",
+    "left",
+    "left"
+  ]
 }
 [/block]
 
-In other words, each event containing an $insert_id is checked for duplication after being minimized to the following shape:
+
+In other words, each event containing an `$insert_id` is checked for duplication after being minimized to the following shape:
 
 ```json
 {
-  "event": "Back to Back",
+  "event": "Item Purchased",
   "properties": {
-    "token": "project_token",
-    "distinct_id": "aubrey@thesix.views",
+    "token": "my_project_token",
+    "distinct_id": "user123xyz",
     "time": 1601412131000,
     "$insert_id": "88B7hahbaschhhB66cbsg"
   },
 }
 ```
 
-If this simplified object is an exact match to any other simplified event it is marked as a duplicate. Ingested events that have been marked as a duplicate will be deleted within 24 hours.
+If this minimized event object is an exact match to any other minimized event object, it is marked as a duplicate. Ingested events that have been marked as a duplicates will be deduplicated.
 
-If an event is sent to the Ingestion API without an `$insert_id` one will be generated for it. However, it will not qualify for the deduplication process.
+If an event is sent to the Ingestion API without an `$insert_id`, one will be generated for it. However, it will not qualify for the deduplication process.
 
-[block:callout]
-{
-  "type": "warning",
-  "title": "Deduplication does not rewrite data",
-  "body": "Using $insert_id is only used to prevent duplicate event data. It cannot be used to update, replace, or delete existing events."
-}
-[/block]
+## Deduplication Mechanisms
+
+Mixpanel uses two main deduplication processes:
+
+### Query-Time Deduplication
+
+- When: Happens immediately when you query data in the Mixpanel UI.
+- How: If multiple events share the same event_name, distinct_id, timestamp, and $insert_id, only the most recent version of the event is shown in reports (based on the API ingestion time). This ensures that duplicate events do not affect your analytics in real time.
+- Scope: This deduplication is visible in the Mixpanel UI and reports, but not in raw data exports. Raw event export will contain all data as they were ingested, without any deduplication.
+
+### Compaction-Time Deduplication
+
+- When: Runs periodically in the backend, typically after a few hours and again after about 20 days, once data ingestion for a day is complete.
+- How: During compaction, Mixpanel scans for events with the same event name, distinct_id, and $insert_id (timestamp does not need to match exactly, just the same calendar day). The older event is deleted, and only the latest remains in storage.
+- Scope: This process helps reduce storage of duplicate events and may affect event counts if duplicates were present with different timestamps
+
+<br />
+
+## Important Notes
+
+**Raw Event Export** - Deduplication is not applied to raw data exports. If you export events via the API, you may see duplicates. It is recommended to apply the same deduplication logic (event name, distinct_id, timestamp, $insert_id) to your exported data
+
+**Insert ID Best Practice** - Always generate a unique $insert_id for each event. Reusing $insert_id (e.g., setting it to the user’s distinct_id) can cause unintended deduplication and data loss
+
+**Deduplication Timing** - Query-time deduplication is immediate. Compaction-time deduplication timing is not guaranteed and may take hours to days to complete.

openapi/src/ingestion.openapi.yaml

@@ -101,7 +101,7 @@ paths:
                       time:
                         type: integer
                         title: time
-                        description: The time at which the event occurred, in seconds or milliseconds since UTC epoch.
+                        description: The time at which the event occurred, in seconds or milliseconds since UTC epoch. If the time value is set in the future, it will be overwritten with the current present time at ingestion.
                       distinct_id:
                         type: string
                         title: distinct_id
@@ -163,7 +163,7 @@ paths:
                       time:
                         type: integer
                         title: time
-                        description: The time at which the event occurred, in seconds or milliseconds since UTC epoch.
+                        description: The time at which the event occurred, in seconds or milliseconds since UTC epoch. If the time value is set in the future, it will be overwritten with the current present time at ingestion.
                       distinct_id:
                         type: string
                         title: distinct_id