Overview of sampling dates of the entire Viridian dataset

[szhan]

Before running sc2ts on the entire dataset, we should check whether there are dates with an unusually high number of samples, like the one seen on 2020 New Year Eve (see jeromekelleher/sc2ts#225).

[szhan]

We may ultimately want to allow user to specify a list of skip dates.

[szhan]

Basically fix jeromekelleher/sc2ts#227

[szhan]

It seems like 2020's New Year's Eve is the only egregious collection date, having more than twice the number of samples as other collection dates. Note that I am counting only the samples having full-precision collection dates, ignoring the less precise dates (e.g. 2020 and 2022-09), so there are 3,960,704 samples here.

Some actual counts.

Counter({Timestamp('2020-12-31 00:00:00'): 40027,
         Timestamp('2021-11-29 00:00:00'): 18348,
         Timestamp('2021-12-29 00:00:00'): 18198,
         Timestamp('2021-12-06 00:00:00'): 16528,
         Timestamp('2022-02-07 00:00:00'): 16443,
         Timestamp('2021-12-15 00:00:00'): 16361,
         Timestamp('2021-12-07 00:00:00'): 15998,
         Timestamp('2021-12-30 00:00:00'): 15768,
         Timestamp('2022-01-04 00:00:00'): 15653,
         Timestamp('2022-01-31 00:00:00'): 15521,
         Timestamp('2021-12-14 00:00:00'): 15283,
         Timestamp('2022-01-19 00:00:00'): 15273,
         Timestamp('2022-02-08 00:00:00'): 15229,
         Timestamp('2021-11-22 00:00:00'): 15138,
         Timestamp('2021-11-30 00:00:00'): 15043,
         Timestamp('2021-12-28 00:00:00'): 14599,
         Timestamp('2021-12-27 00:00:00'): 14494,
         Timestamp('2021-11-08 00:00:00'): 14462,
         Timestamp('2022-01-03 00:00:00'): 14384,
         Timestamp('2022-01-24 00:00:00'): 14327,
         Timestamp('2021-11-16 00:00:00'): 14283,
         Timestamp('2021-12-01 00:00:00'): 14264,
         Timestamp('2021-12-08 00:00:00'): 14155,
         Timestamp('2022-02-02 00:00:00'): 14087,
         Timestamp('2022-01-21 00:00:00'): 13846,
...
         Timestamp('2022-12-04 00:00:00'): 352,
         Timestamp('2020-07-19 00:00:00'): 351,
         Timestamp('2020-04-25 00:00:00'): 349,
         Timestamp('2020-08-02 00:00:00'): 349,
         ...})

[jeromekelleher]

Excellent - let's just add a skip-date argument to the CLI so and add 2020-12-31 in our runscripts.

I guess one thing to follow up on just to be sure would be to do some analysis on the pango lineages from 2021-11-29. Do they all look roughly like what should be sampled on that date?

[szhan]

I'm comparing the per-Pangolin label counts of samples collected on 2021-11-29 (the focal date) in the Viridian dataset versus those from the cov-lineage website (recorded here).

The cov-lineages website keeps track of the earliest and latest dates of each Pangolin lineage (earliest_date and latest_date in the table below), as well as the sample count per collection date (date_count in the table below). It seems that the focal collection date lies between the earliest and latest dates reported by the cov-lineages website for each Pangolin lineage.

pango_label	viridian_count	earliest_date	latest_date	date_count	is_focal_date_within_range
AY.4.2.2	821	2021-04-12	2022-03-13	842	True
AY.4.2	1972	2021-01-01	2022-08-15	2260	True
AY.4	6187	2020-05-11	2023-02-22	8330	True
AY.4.5	28	2021-01-11	2022-04-07	177	True
AY.122	346	2020-05-11	2024-07-23	2939	True
AY.4.2.1	564	2020-12-21	2022-03-09	583	True
AY.4.8	63	2021-07-13	2022-01-29	68	True
AY.43	768	2020-05-11	2023-03-08	5120	True
BA.1	104	2020-06-25	2024-01-20	286	True
AY.46.5	202	2021-02-06	2022-02-03	196	True
AY.9.2	50	2020-10-15	2022-12-17	375	True
AY.120	253	2020-10-26	2022-09-28	331	True
BA.1.19	10	2020-06-01	2022-05-30	20	True
AY.34.1	58	2020-04-01	2022-12-17	160	True
AY.5	309	2020-11-15	2022-12-17	503	True
B.1.617.2	279	2020-03-27	2023-10-09	1834	True
AY.98	579	2020-10-14	2022-12-17	643	True
AY.36	54	2020-08-11	2022-02-23	208	True
AY.44	725	2020-09-09	2022-12-21	2082	True
AY.6	90	2021-03-09	2022-05-23	131	True
AY.4.2.3	56	2021-04-30	2022-04-12	187	True
AY.111	122	2021-02-05	2022-02-23	142	True
BA.1.1	13	2020-01-27	2024-06-25	60	True
AY.25.1.2	1	2021-08-03	2023-07-17	4	True
AY.34	54	2020-11-18	2022-08-30	174	True
AY.4.9	15	2020-08-11	2022-12-17	70	True
BA.1.17	26	2020-11-06	2023-08-30	55	True
AY.4.10	10	2021-09-10	2022-01-28	42	True
AY.39.1	22	2021-01-06	2022-09-28	117	True
AY.129	10	2020-05-11	2022-12-17	195	True
AY.33	7	2020-10-26	2022-12-17	141	True
AY.127	106	2021-01-09	2022-12-17	655	True
AY.126	80	2020-10-14	2022-05-15	725	True
AY.4.2.5	27	2021-10-22	2022-01-27	27	True
AY.125	27	2020-08-11	2022-12-17	335	True
AY.42	18	2020-10-15	2022-12-17	194	True
AY.98.1	40	2020-11-04	2022-12-17	637	True
AY.121	50	2020-05-11	2022-10-14	475	True
AY.90	25	2021-05-02	2022-01-19	25	True
AY.3	598	2020-08-01	2022-09-27	1497	True
AY.5.4	3	2021-04-30	2022-12-17	56	True
AY.7	18	2021-05-01	2022-01-15	17	True
AY.100	343	2020-08-31	2022-08-04	1038	True
AY.4.3	8	2021-04-30	2022-01-29	54	True
AY.25	421	2020-07-21	2023-05-03	1240	True
AY.4.11	4	2021-05-03	2022-12-17	10	True
AY.46.6	13	2020-08-11	2022-11-09	420	True
AY.109	13	2021-06-25	2022-03-24	26	True
AY.39	194	2021-01-08	2022-10-23	610	True
AY.118	35	2021-02-09	2022-09-13	150	True
BA.1.17.2	14	2020-10-23	2023-10-01	36	True
AY.61	1	2021-04-22	2022-01-14	5	True
AY.121.1	10	2021-01-02	2022-03-01	93	True
BA.1.14	2	2021-01-10	2023-01-13	7	True
AY.34.2	8	2021-07-12	2022-02-03	38	True
AY.123.1	2	2021-01-12	2022-02-13	17	True
AY.103	1284	2020-05-14	2023-02-01	3695	True
AY.4.13	3	2021-05-03	2022-03-04	27	True
AY.108	4	2021-05-25	2022-04-12	11	True
AY.25.1	301	2020-09-11	2022-12-29	1228	True
AY.45	1	2021-01-01	2022-07-04	11	True
AY.65	1	2020-09-11	2022-04-02	7	True
AY.46	7	2020-06-01	2022-09-15	94	True
AY.43.6	2	2021-08-31	2022-01-24	149	True
BA.1.15	4	2020-11-10	2024-01-17	27	True
AY.124	25	2020-05-11	2022-12-17	123	True
AY.84	1	2020-05-11	2022-01-24	21	True
AY.46.2	2	2021-03-09	2022-01-27	20	True
AY.47	41	2020-11-13	2022-04-19	130	True
AY.20.1	2	2021-09-05	2022-01-28	7	True
AY.131	1	2021-08-12	2022-08-01	5	True
AY.43.8	3	2021-01-12	2022-02-15	58	True
AY.124.1.1	1	2021-06-26	2022-05-31	8	True
AY.23	1	2020-03-12	2022-11-09	44	True
AY.9	3	2021-01-20	2022-07-21	26	True
AY.85	1	2021-01-09	2022-03-28	27	True
AY.101	1	2021-05-01	2022-10-01	43	True
AY.119	170	2020-10-29	2022-10-24	420	True
AY.43.9	1	2021-05-03	2022-02-01	12	True
AY.7.1	3	2021-02-09	2022-02-27	55	True
BA.1.18	3	2020-11-13	2023-08-07	31	True
AY.123	2	2021-03-11	2022-03-17	87	True
AY.26	56	2021-01-07	2023-01-05	185	True
BA.1.13	5	2021-08-31	2022-10-04	10	True
B.1.640.1	1	2021-01-01	2022-06-03	58	True
AY.92	1	2021-04-10	2022-12-07	108	True
AY.4.15	1	2021-09-08	2022-01-13	30	True
AY.128	1	2021-05-15	2022-01-15	7	True
AY.73	1	2021-05-03	2022-02-05	13	True
AY.69	1	2021-05-15	2022-02-10	106	True
AY.117	117	2020-05-11	2022-06-02	235	True
AY.87	2	2021-05-18	2022-01-22	2	True
AY.33.2	1	2021-04-30	2023-01-13	25	True
AY.110	1	2021-03-09	2022-06-16	0	True
AY.119.2	134	2021-01-11	2022-05-26	205	True
AY.75	17	2021-01-06	2022-10-04	60	True
AY.20	44	2021-01-05	2022-04-06	230	True
AY.25.3	8	2021-07-15	2022-06-23	94	True
AY.3.1	33	2020-11-20	2022-01-30	86	True
AY.114	23	2021-05-17	2022-02-01	31	True
AY.4.4	1	2021-01-19	2022-02-22	33	True
AY.112.2	1	2021-01-10	2022-04-12	23	True
AY.116.1	10	2021-06-07	2022-02-01	24	True
AY.54	10	2021-02-09	2022-02-10	29	True
AY.107	8	2021-01-05	2022-05-29	26	True
AY.120.1	3	2020-12-27	2022-02-15	17	True
AY.113	29	2020-12-16	2022-12-21	138	True
AY.43.3	2	2021-08-11	2022-02-09	32	True
AY.122.4	9	2021-08-30	2022-04-04	37	True
AY.99.2	5	2021-02-28	2022-10-10	239	True
AY.64	2	2021-05-22	2021-12-29	15	True
AY.46.4	12	2020-11-09	2022-02-09	38	True
AY.1	4	2021-03-31	2023-04-22	20	True
AY.3.3	10	2021-09-01	2022-02-07	32	True
AY.119.1	1	2020-11-09	2022-05-30	11	True
AY.122.6	2	2021-05-03	2022-07-14	116	True
AY.46.1	13	2021-06-25	2022-11-23	21	True
AY.5.3	6	2021-07-02	2022-01-15	15	True
AY.25.1.1	9	2021-07-26	2021-12-30	7	True
AY.14	8	2021-03-30	2023-01-12	18	True
AY.39.2	3	2021-07-06	2022-05-05	22	True
AY.16	4	2020-10-26	2022-01-17	10	True
AY.74	1	2021-04-09	2023-08-20	27	True
AY.105	1	2021-05-12	2022-02-01	2	True
BA.2	2	2020-03-28	2024-08-06	5	True
AY.124.1	10	2021-06-21	2022-01-30	18	True
AY.43.5	7	2021-10-22	2022-01-24	13	True
BA.1.21	1	2021-01-09	2023-05-15	24	True
AY.72	1	2021-05-24	2022-02-08	10	True

[szhan]

The sample counts are pretty well correlated as well.

[jeromekelleher]

Great - so we're happy that the samples for this date (2021-11-29) look OK, and (by extension) the conclusion that we only need to filter the samples for 2020-12-31?

[szhan]

I just checked 2021-12-29(the other date with >18k samples) as well, and it looks okay. So, yes, I think we just need to skip 2020-12-31.

[szhan]

I guess we can close this now that we know there is really only one egregious collection date to skip.

[jeromekelleher]

Yep.

[jeromekelleher]

I moved this here as a conversation as I'm regularly coming back to look at the plot above and it's good to have the data somewhere easy to find