HisVis2

Project Description

Work for Part 2 of HisVis Project. In this project, we developed a scene detection dataset and trained models for predicting the labels. The input data for this project are press photographs from the De Boer collection.

This project includes software to: - automatically resize and crop the images - automatically rotate them to the correct orientation - training a rotation model - updating a pre-trained Places-365 model to our data - training a logistic classifier on top of a CLIP model - making predictions for the scene categories using the places-365, CLIP model or both - making predictions for the categories (indoor / outdoor) - preparing output generated by the annotation software (VeleHanden) - analyzing the annotations - checking whether the algorithm can also be applied to uncropped images

Prerequisites

Python 3.7.11 See requirements.txt to set up a Conda environment. conda create --name <envname> --file requirements.txt

Contributing

Issue Tracker: https://github.com/melvinwevers/HisVis2/issues

License

CC-NY

Contact

Dr. Melvin Wevers (melvin.wevers(at)uva.nl)

Folder Structure

Data

Intermediary

extra_training_labels.csv- labels for which additional samples were need in annotation step 3.

Processed

Annotations

• annotation_data_step1.csv - processed database export
• annotation_data_step21.csv - processed database export
• annotation_data_step3.csv - processed database export

Training Data

• all_labels.txt - list of all labels used

Place training data from Zenodo here.

Raw

Test

Contains example test file

Docs

• data_sheet.md - data sheet for HisVis dataset
• label_sheet.md - label sheet documenting used labels
• model_card_indoor_outdoor_clip - model card for indoor-outdoor detection using CLIP
• model_card_indoor_outdoor_places - model card for indoor-outdoor detection using Places-365
• model_card_rotation - model card for correct rotation detection
• model_card_scene_detection_clip - model card for scene detection using CLIP
• model_card_scene_detection_places - model card for scene detection using Places-365

Notebooks

• 0.install_packages.ipynb - install required packages that could not be installed using pip / conda
• 1.prepare_rotation_training_data.ipynb - creating training data for the rotation model using correctly rotated data
• 2.train_model_rotation.ipynb - training model for identifying correct rotation
• 3.make_predictions_in_out.ipynb - predicting whether an image is taken indoor or outdoor using CLIP.
• 4.prepare_output_picturae.ipynb - preparing data exported from Vele Handen provided by Picturae.
• 5.analyze_annotations.ipynb - analysis of annotation effort and code to prepare new training data
• 6.check_prediction_crop_or_not.ipynb - evaluation of model performance on cropped and non-cropped images.

Output

• finding_learningrate.out - output from learning rate finder used to estimate optimal learning rate
• slurm-9991191.out stack trace for training models
  
• slurm-9992781.out stack trace for training models
  

Models

Place models.tar.gz from Zenodo here

indoor_outdoor

Models and evaluation of models for indoor-outdoor detection

Rotation

• rotation.pkl model used to predict correct rotation of images

Scenes

Models and evaluation for models for scene detection

src

• find_correct_rotation.py - Python script to rotate images until they are correctly oriented. This makes use of the rotation.pkl model. This model has been trained using the notebook train_model_rotation.ipynb
• helper.py - script with different helper functions
• make_predictions_in_out.py - script to make indoor / outdoor predictions using standard clip
• make_predictions.py - script to make scene detection predictions using fine-tuned clip, places365, or both
• remove_border.py - script to resize images and crop them out of the photo negative sheets
• train_clip.py - script to finetune CLIP
• train_places_model.py script to finetune places-365

Preparing images

run prepare_images.sh

Training the model

We can train the model using a Jupyter Notebook or a Python Script that can be called using the run.sh bash script.

Running predictions

Run run_prediction.sh