Oxford-IIIT Pet Classifier – Image Similarity Web App

A lightweight web application that classifies pet breeds using a TensorFlow/Keras model trained on the Oxford-IIIT Pet Dataset and an embedding + cosine-similarity pipeline. The trained model is served with Flask, and users can upload an image via a minimal HTML UI to get instant predictions and a confidence score.

This repository contains:

• A Colab-friendly training script that downloads & prepares the dataset, fine-tunes a MobileNetV2 backbone, and saves the model as pet_classifier_model.keras.
• An embedding pipeline that converts images into feature vectors and builds an embeddings.pkl file used at inference time.
• A Flask web app (app.py) that loads the embedding index and returns the closest class by cosine similarity, with a simple UI.

---

Features

• Transfer learning with MobileNetV2 on Oxford-IIIT Pets (37 classes)
• Embedding search with cosine similarity for fast, interpretable predictions
• Clean Flask UI for image upload and response
• Colab-first training flow; easy to reproduce without local dataset wrangling

---

Project Structure

.
├── app.py                       # Flask app (upload + predict)
├── embed_model.py               # Loads the .keras model and exposes an embedding model
├── create_embeddings.py         # Builds embeddings.pkl from classified_images/
├── embeddings.pkl               # (generated) embeddings + labels
├── pet_classifier_model.keras   # (generated) trained Keras model
├── static/
│   └── uploads/                 # Uploaded images via UI
├── templates/
│   └── index.html               # Simple HTML form + result page
├── train_model/
│   └── model_training.py        # Colab-friendly training script
├── requirements.txt             # Python dependencies
└── README.md

Note: classified_images/ is generated by the training script under the working directory it runs in. If you train in Colab, download the folder or rebuild embeddings locally using the same folder structure.

---

Requirements

• Python 3.9–3.11 (recommended)
• pip / venv
• GPU is not required for inference (helpful for training)

requirements.txt (covers both training & app):

flask
tensorflow
numpy
pillow
opencv-python
scikit-learn
matplotlib

Remove matplotlib if you only run the web app without training or plotting.

---

1) Train the Model (Google Colab)

The training script downloads images and annotations, organizes them by class, builds a TensorFlow data pipeline, and fine-tunes MobileNetV2.

1. Open train_model/model_training.py in Google Colab.

2. Run all cells. When finished, you will get:

   • pet_classifier_model.keras
   • A folder classified_images/ containing images grouped by class.

3. Download these two artifacts to your local project directory.

You can also fine-tune different backbones or adjust augmentation in the script if needed.

---

2) Build the Embedding Index

create_embeddings.py iterates over classified_images/, encodes each image with the embedding model derived from your .keras network, and saves two items into embeddings.pkl: a 2D array of embeddings and the corresponding label list.

python create_embeddings.py

• Make sure pet_classifier_model.keras is in the same directory as embed_model.py and create_embeddings.py.
• By default, the script expects classified_images/ at the project root. If yours is elsewhere, edit base_dir in create_embeddings.py.

This will generate:

embeddings.pkl  # (embeddings, labels)

---

3) Run the Web App (Flask)

1. Create and activate a virtual environment (recommended):

   python -m venv .venv
   # Windows
   .venv\Scripts\activate
   # macOS/Linux
   source .venv/bin/activate

2. Install dependencies:

   pip install -r requirements.txt

3. Ensure these files exist at the project root:

   • app.py
   • embeddings.pkl
   • embed_model.py
   • pet_classifier_model.keras
   • templates/index.html
   • static/uploads/ (folder exists)

4. Start the development server:

   python app.py

5. Open the UI in your browser (usually):

   http://127.0.0.1:5000/
   

Upload an image → see the predicted breed and a confidence (cosine similarity).

---

Inference Details & “Not Found” Option

By default, the app returns the closest class with its cosine-similarity score. If you want to display “Not Found” for low-confidence matches, add a threshold in predict_class:

label, score = best_label, confidence
THRESHOLD = 0.50  # tune 0.3–0.7 based on validation
if score < THRESHOLD:
    label = "Not Found"

Tip: Tune the threshold using a small validation set and monitor accuracy vs. rejection rate.

---

Troubleshooting

• ModuleNotFoundError: cv2 → Install opencv-python: pip install opencv-python
• ModuleNotFoundError: sklearn → Install scikit-learn: pip install scikit-learn
• ValueError: shapes (1, N) and (M, N) not aligned → Ensure embeddings were built with the same model you load in embed_model.py.
• Low confidence → Rebuild embeddings after any retraining; try a higher image resolution or stronger augmentation.
• Large memory → If embeddings.pkl is big, consider PCA/whitening or switch to FAISS for larger-scale nearest-neighbor search.

---

Notes

• The training script is Colab-ready; for local training, remove any !pip install ... lines and ensure all packages are present.
• For production use, you may expose a REST endpoint (e.g., /predict) in Flask or migrate serving to FastAPI + Uvicorn. The current app focuses on a simple HTML workflow.