Skip to content

Massively rework everything#11

Draft
cleong110 wants to merge 155 commits intoBibleNLP:mainfrom
cleong110:ebible_mapping
Draft

Massively rework everything#11
cleong110 wants to merge 155 commits intoBibleNLP:mainfrom
cleong110:ebible_mapping

Conversation

@cleong110
Copy link
Collaborator

@cleong110 cleong110 commented Jun 17, 2025 •
edited
Loading

Things changed include:

  • restructure the entire repo, renaming things, so that it can be pip installed and you can import any package from any other
  • deduplicate code
  • remove the GUI so it runs on remote machines
  • Get the scripture citations from metadata.xml
  • Bible verse reference lookups
  • Some pytesting to ensure code quality
  • Reformat everything with ruff code formatter
  • code for doing OCR segmentation
  • Tasks for dataset

@cleong110 cleong110 marked this pull request as draft June 17, 2025 00:10
@cleong110 cleong110 changed the title Ebible mapping Ebible mapping and Headless downloading Jun 17, 2025
cleong110 added 20 commits July 2, 2025 15:42
@cleong110
Run autoformatter on prepare webdataset script
ff510a3
@cleong110
Run autoformatter on upload to huggingface script
938cd56
@cleong110
some WIP updates to prepare webdataset: remove the GUI
4df3aaf
@cleong110
Lots of reformatting or cleanup, mostly
5222242
@cleong110
A couple scripts for eBible parsing
973d6eb
@cleong110
A whole slew of OCR changes
eff0c18
@cleong110
helper script to find the next OCR file to edit
31a1912
@cleong110
various more OCR segmentation scripts
f3302c3
@cleong110
massive changes to prepare_webdataset.py: rework/remove the segmenter…
68738db 
… code, rework the webdataset format including language codes etc
@cleong110
scripts I used to finally get dwpose working
a3bfa28
@cleong110
edit the run-dwpose script to save confidence
0c15540
@cleong110
add a little thumbnail extraction script
4f554f2
@cleong110
verse utils
2ab7b35
@cleong110
edit DAN 6:1 to be 5:31
9867d96
@cleong110
Verse citation processing
859fe70
@cleong110
interactive script to look at ocr csv with thumbnails
c18c40e
@cleong110
reformatting upload script
1f14785
@cleong110
update dwpose script to check if confidences is in there and if not, …
0b4f5b5 
…add it
@cleong110
bash script to run segmentation
d94a33c
@cleong110
formatting and deletion
249a2e3
cleong110 added 30 commits July 31, 2025 04:35
@cleong110
a bit of formatting and tqdm
bb731ee
@cleong110
evaluate segments now also outputs avg number of segments to be ranked
d4b9982
@cleong110
gather and grade predictions!
b746903
@cleong110
code to extract faces from _frame folders
2d19ae8
@cleong110
run_skintone now uses facecrops by default
86acd81
@cleong110
Updating some data analysis scripts including autosegmenter
c62391f
@cleong110
Various data analysis scripts
db85181
@cleong110
prediction scripts
1650af1
@cleong110
get unique verse counts and supervenn
a3807ae
@cleong110
read eng_vref to figure out which verses are in sign bibles but NOT b…
7b9e876 
…sb. Also supervenn
@cleong110
quick script that reads the extracted webdataset transcripts and calc…
ddf99fd 
…ulates stats
@cleong110
Optionally, DON'T generate the README
8a98344
@cleong110
CDL: some notes
fd69075
@cleong110
CDL: script to extract webdataset to a folder for analysis, renaming …
ee727dd 
…to avoid collisions
@cleong110
code to generate BRISQUE latex table
48aa968
@cleong110
Various improvements to language overlaps visualization
aec57fe
@cleong110
Improve extract-frames-with-poses script
eef4980
@cleong110
update script to download/analyze dataset via a streaming method
e6943ea
@cleong110
Update script for autosegmenter boxplots
4f97251
@cleong110
Some fixes for eBible utils, and add deprecation warnings on outdated…
fbe3175 
… code
@cleong110
Updated video_ocr script output filenames
3450c71
@cleong110
Script for finding invalid videos
6ce72a0
@cleong110
Add remove-large-files flag to prepare_webdataset script
4acdce5
@cleong110
Script that, given a webdataset folder, will flatten it one level up.
ffbc062
@cleong110
transcript similarity heatmap plot script
e1d3438
@cleong110
Notebook to generate snippets based on sign-spotting ground truth, fo…
2d4f00a 
…r qualitative analysis
@cleong110
Notebook to generate snippets based on sign-spotting ground truth, fo…
7461d70 
…r qualitative analysis
@cleong110
Merge branch 'ebible_mapping' of github.com:cleong110/sign-bibles-dat…
5ce1f76 
…aset into ebible_mapping
@cleong110
Add more gitignore
17b5ed1
@cleong110
A few more data analysis scripts
09e4799
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cleong110

Comments