[PPOCR det] Precision and Recall quickly achieves 1.0

[PhuongNDVN]

I am trying to do fine-tuning text detection model. Surprisingly, the model achieves both Precision and Recall to 1.0 quickly after few epochs. It happens when I just use default config value shrink_ratio = 0.45. The loss also does not decrease much (from 1.6 -> 1.4). Every evaluation after that Precision and Recall always = 1.0 and model of following epochs is chosen as best model. When I change shrink_ratio = 0.3, Precision and Recall reduce to 0.91xxx. My question is the method to measure Precision and Recall be able to trusted? I use 600 images to train and 150 images for evaluation as recommended in document

[GreatV]

Your observation that Precision and Recall reach 1.0 unusually fast during fine-tuning suggests potential issues with training configuration, dataset, or evaluation methodology. Here are possible explanations and solutions:

Possible Reasons:

1. Overfitting to a Small Dataset

   • You mentioned using only 600 images for training and 150 for evaluation. This is a relatively small dataset, which can lead to overfitting where the model memorizes the training set rather than generalizing.

2. Mismatch Between Training and Evaluation Data

   • If the training and validation datasets are highly similar (or even overlap), the model may perform perfectly on them but fail in generalization.

3. Evaluation Metric Calculation Issue

   • The method for calculating Precision and Recall in PaddleOCR depends on correctly set post-process parameters (box_thresh, thresh, etc.). If the thresholds are too lenient, they may artificially boost Precision/Recall.

4. Effect of shrink_ratio Parameter

   • You observed that reducing shrink_ratio from 0.45 to 0.3 led to a drop in Precision/Recall to 0.91.
   • A higher shrink_ratio (like 0.45) might overly shrink the text region, making it easier for the model to match ground truth, thus inflating metrics improperly.

Recommendations & Steps to Validate:

1. Increase Dataset Size & Add Augmentations

   • Try increasing the number of training images and introducing stronger augmentation techniques such as random cropping, rotation, and noise addition.

2. Double-check Evaluation Data Separation

   • Ensure your validation set is strictly separate from training data. Even small overlaps can lead to high metrics.

3. Adjust Post-processing Parameters

   • Experiment with thresh and box_thresh in DBPostProcess. For example:

     PostProcess:
       name: DBPostProcess
       thresh: 0.3  # Threshold for binarization
       box_thresh: 0.6  # Confidence threshold for valid boxes
       max_candidates: 1000
       unclip_ratio: 1.5

   • Increasing box_thresh (e.g., to 0.7 or 0.8) may give more realistic scores.

4. Monitor Loss Alongside Metrics

   • If the loss is not decreasing significantly (1.6 → 1.4) while Precision/Recall hit 1.0, this indicates potential overestimation of performance.

5. Check for Bugs in Evaluation Code

   • Debug by manually checking sample images where the model reports perfect detection to validate correctness.

Related Discussion Links:

• A similar discussion about hmean = 0 issue:
  ch_PP-OCRv3_det_student.yml文本检测微调hmean一直为0 #14250
  → Ensuring dataset size, proper preprocessing, and parameter tuning are common concerns.

Conclusion

Your high Precision/Recall scores likely result from overfitting, evaluation biases, or improper post-processing parameters. Adjust the dataset, parameter settings, and augmentation techniques to obtain a more reliable evaluation.

---

Response generated by 🤖 feifei-bot | chatgpt-4o-latest

⚠️ This is an AI-generated response. Please verify any critical information.