REFACTOR: Testing data collection pipeline - Migrating R code to Python

[lucasrodes]

We are currently refactoring the project to have a fully python-based pipeline for our COVID-19 testing data collection. This way, the testing pipeline will also take advantage of the main repository package cowidev.

Planned stages*:

• ✅ STAGE 1 - Migrate country scraping scripts from R to Python: Find these at scripts/scripts/testing/automations/.
  More details...

  • Albania Albania test python script #2288
  • Australia (test, australia) python and batch #2303
  • Azerbaijan Azerbaijan refactor #2123
  • Bahrain TEST: Add utils + migrate Bahrain to Python #2156
  • Belarus Testing: Belarus python port #2129
  • Bulgaria refactor(test,bulgaria): migrate to python #2158
  • Cape Verde refactor(test, Cape Verde): python migration #2348
  • Costa Rica refactor(test, costa rica): python migration #2341
  • Denmark refactor(test,denmark): python migration #2360
  • Dominican Republic refactor(test, Dominican Republic): python migration #2373; manual
  • Fiji refactor(test, Fiji): migrate to python #2354 (tests, fiji): collect cumulative total #2382
  • Hong Kong refactor(test, hong kong): python migration #2343
  • Hungary refactor(test, hungary): migrate to python #2351
  • Iceland refactor(test,Iceland): python migration #2361
  • India India refactor #2131
  • Iran (test, iran) python script #2272
  • Italy Refactor(test,italy) migrate to python #2203
  • Kenya Kenya refactor #2117
  • Lebanon Create lebanon.py #2120
  • Liechtenstein refactor(test): Liechtenstein #2255
  • Luxembourg refactor(test,Luxembourg): python migration #2363
  • Malawi data(tests): africa cdc scraper #2251
  • Malta Refactor(test,malta) migrate to python #2202
  • Mauritania data(tests): africa cdc scraper #2251
  • Mexico Testing Mexico #2279
  • Moldova Test moldova new script #2305
  • Mongolia Refactor(test,mongolia) migrate to python #2209
  • New Zealand Refactor(tests,new zealand) refactor python scripts #2210
  • Nigeria Nigeria refactor #2110
  • Pakistan (test, pakistan) python script #2311
  • Peru Peru testing data #2302
  • Philippines refactor(test, philippines): python migration #2349
  • Romania refactor(test,Romania): python migration #2364
  • Russia (test, Russia) python conversion #2315
  • Singapore (test, Singapore) python script #2310
  • South Korea Refactor(test,southkorea) migrate from r to python #2217
  • South Sudan data(tests): africa cdc scraper #2251
  • Sweden refactor(test, sweden): python migration #2331
  • Switzerland refactor(test, Switzerland): Python migration #2357
  • Tunisia Tunisia refactor #2116
  • Ukraine refactor(test, ukraine): python migration #2342
  • Uruguay Uruguay, batch version for number of tests performed daily. #2154

• ✅ STAGE 2 - Revisit R scripts in scripts/scripts/testing/ directory: Deprecate scripts no longer needed.
• ⬜ STAGE 3 - Migrate generate_dataset.R to Python: Could split this into two steps, as in vaccination data pipeline (process and generate). 👤 @lucasrodes @edomt
• ✅ STAGE 4 - Move all temporary data files from automated_sheets/ to scripts/output/ and update scripts accordingly. refactor(test): move files to new location #2415_
• 🔁 STAGE 5 - Move logic into cowidev: scraping scripts, generate_datset, etc. This means creating a new submodule cowidev.testing. 👤 @lucasrodes
  • Scraping part of the test pipeline. refactor(test): integrate scraping scripts into library #2402
  • Process & checks
  • Merging testing data
  • Merging with global dataset
• 🔁 STAGE 6 - Delete all files in scripts/scripts/testing/ except those extrictly necessary for legacy purposes (e.g. maintain links from published papers, similar as in scripts/scripts/vaccination/). 👤 @lucasrodes @edomt
• ⬜ STAGE 7 - Update docs. 👤 @lucasrodes @camappel

*There is no particular chronological order.

How can you help

We are currently at STAGE 5.

The development of STAGE 5 is happening in branch feature/tests-refactor. This is being carried by @lucasrodes.

Soon we will have a new tool cowid to manage all our pipelines, including the testing pipeline. In the first version you will be able to run all scraping scripts for testing data as:

$ cowid test get

Questions?

Comment in this thread

[bzki]

Hi, I was hoping to contribute to the testing code and I have written a working script for Belarus: https://github.com/bzki/covid-19-data/blob/belarus_test/scripts/scripts/testing/automations/incremental/belarus.py
master...bzki:belarus_test

Before I submit a pull request, a few questions:
(1) Should I ask for a code review/testing here, or will that be conducted as part of the pull request? I'm having some issues getting cowidev installed so am not able to run python3 run_python_scripts.py quick in its entirety, so it would be nice to have some detailed checking/feedback for how to improve.
(2) The Jamaica script provided as an example does not use the cowidev code, but it seems the recent python additions like Kenya do make use of them. Should I be making use of cowidev (e.g., get_soup) for these contributions?

Apologies if the answers are obvious -- learning as I go.

[lucasrodes]

Hi @bzki!
Thanks for your feedback and for being willing to contribute, we really appreciate this!

I think there is a couple of things that you and we can do on our ends

You can do two things:

• For (1), I suggest creating a new issue, explaining what is failing in your cowidev library installation, so we can look at that in detail.
• Also, you can also run the script individually (instead of running all the scripts), e.g. python belarus.py.

On my end:

• Regarding (1), I just refactored the instructions to install the library here.
• On point (2), I changed the reference scripts to Japan and Kenya/Nigeria, respectively. The latter make use of the cowidev library.

Also, feel free to create a PR whenever you consider that your script is functional, so we can review it. Please let me know if this helps!

[bzki]

Thanks! Running the script on its own via python belarus.py did indeed work fine. I will, however, work on getting cowidev installed (and create a new issue if I'm unable to figure it out) and conform the script with the more recent examples you highlighted. Thank you for your quick responses.

[bzki]

I was able to get cowidev installed. See my pull request here: #2129