Create div_sampling.py

For DIV (diversity sampling for InternVid-10M-DIV), we aim to sample video clips from all long videos available to maximize data diversity. This was done by counting the frequencies of long videos in the segmented clip pool and sampling clips with probabilities inverse to these frequencies.

Author: shepnerd

Data/InternVid/div_sampling.py

@@ -0,0 +1,14 @@
+from collections import Counter
+import json
+import random
+import numpy as np
+data = json.load(open("/path/to/to_sample"))
+video_id = set([x["video"].split("/")[-1][:11] for x in data])
+video_id_counter = Counter([x["video"].split("/")[-1][:11] for x in data])
+sampling_weights = [1.0 / video_id_counter[x["video"].split("/")[-1][:11]] for x in data]
+np.random.seed(42)
+sampling_weights = np.array(sampling_weights)
+sampling_weights = sampling_weights / sampling_weights.sum()
+sampled_index = np.random.choice(len(data), 10647458, replace=False, p=sampling_weights)
+data = [data[i] for i in sampled_index]
+json.dump(data, open("/path/to/sampled", "w"))