Contributing to enhance text data pipeline - normalization & filtering ideas

[codewithEshaYoutube]

Hi gloss-api maintainers

I’d like to contribute to the text data pipeline standards by adding a
modern, AI-enhanced workflow for automated text preprocessing.

The proposed pipeline covers:

• Text normalization (Unicode, punctuation, whitespace cleanup)
• Heuristic filtering (low-quality text removal)
• Deduplication (exact + near-duplicate detection)
• Sharding for scalable processing

Additionally, AI models can be integrated for:

• PII detection and redaction
• Quality scoring of text data
• Real-time backend ingestion via a FastAPI service

The goal is to create a reproducible, modular, and automated pipeline
that can handle large-scale datasets efficiently.

I’ve also prepared a simple diagram to illustrate the workflow.

Looking forward to your feedback and suggestions.