Migrating CT documents to Azure AI Foundry

Author: melghazali

articles/ai-services/translator/custom-translator/azure-ai-foundry/beginners-guide.md

@@ -0,0 +1,166 @@
+---
+title: Azure AI Translator Custom Translation for beginners
+titleSuffix: Azure AI services
+description: A user guide for understanding the end-to-end customized machine translation process.
+author: laujan
+manager: nitinme
+ms.service: azure-ai-translator
+ms.author: lajanuar
+ms.date: 01/27/2025
+ms.topic: overview
+---
+
+# Azure AI Translator Custom Translation for beginners
+
+ [Custom Translation](overview.md) enables you to a build translation system that reflects your business, industry, and domain-specific terminology and style. Training and deploying a custom system is easy and doesn't require any programming skills. The customized translation system seamlessly integrates into your existing applications, workflows, and websites and is available on Azure through the same cloud-based [Microsoft Text Translation API](../reference/v3-0-translate.md?tabs=curl) service that powers billions of translations every day.
+
+[Custom Translation](overview.md) empowers you to build a translation system that truly captures your business's unique language, industry terminology, and domain-specific style. With an intuitive interface, training, testing, and deploying your custom model is simple and requires no programming expertise. Seamlessly integrate your tailored translation system into your existing applications, workflows, and websites—all backed by the cloud-based [Azure AI Translator Text Translation API](../reference/v3-0-translate.md?tabs=curl) service that powers billions of translations each day.
+
+The platform enables users to build and publish custom translation systems to and from English. The Custom Translator supports more than 100 languages that map directly to the languages available for Neural machine translation (NMT). For a complete list, *see* [Translator language support](../language-support.md).
+
+## Is a custom translation model the right choice for you?
+
+A well-trained custom translation model excels at delivering accurate, domain-specific translations by learning from your previously translated in-domain documents. This approach ensures that your specialized terms and phrases are used in context, producing fluent, natural translations that respect the target language’s grammatical nuances.
+
+Keep in mind that developing a full custom translation model requires a substantial amount of training data—typically at least 10,000 parallel sentences. If you don’t have enough data to train a comprehensive model, you might consider building a dictionary-only model to capture essential terminology, or you can rely on the high-quality, out-of-the-box translations offered by the Text Translation API.
+
+Ultimately, if you need translations that reflect your industry’s specific language and you have ample training resources, a custom translation model can be the ideal choice for your organization.
+
+:::image type="content" source="media/how-to/for-beginners.png" alt-text="Screenshot illustrating the difference between custom and general models.":::
+
+## What does training a custom translation model involve?
+
+Building a custom translation model requires:
+
+* Understanding your use-case.
+
+* Obtaining in-domain translated data (preferably human translated).
+
+* Assessing translation quality or target language translations.
+
+## How do I evaluate my use-case?
+
+Having clarity on your use-case and what success looks like is the first step towards sourcing proficient training data. Here are a few considerations:
+
+* Is your desired outcome specified and how is it measured?
+
+* Is your business domain identified?
+
+* Do you have in-domain sentences of similar terminology and style?
+
+* Does your use-case involve multiple domains? If yes, should you build one translation system or multiple systems?
+
+* Do you have requirements impacting regional data residency at-rest and in-transit?
+
+* Are the target users in one or multiple regions?
+
+## How should I source my data?
+
+Finding in-domain quality data is often a challenging task that varies based on user classification. Here are some questions you can ask yourself as you evaluate what data is available to you:
+
+* Does your company have previous translation data available that you can use? Enterprises often have a wealth of translation data accumulated over many years of using human translation.
+
+* Do you have a vast amount of monolingual data? Monolingual data is data in only one language. If so, can you get translations for this data?
+
+* Can you crawl online portals to collect source sentences and synthesize target sentences?
+
+## What should I use for training material?
+
+| Source | What it does | Rules to follow |
+|---|---|---|
+| Bilingual training documents | Teaches the system your terminology and style. | **Be liberal**. Any in-domain human translation is better than machine translation. Add and remove documents as you go and try to improve the [BLEU score](concepts/bleu-score.md?WT.mc_id=aiml-43548-heboelma). |
+| Tuning documents | Trains the Neural Machine Translation parameters. | **Be strict**. Compose them to be optimally representative of what you are going to translate in the future. |
+| Test documents | Calculate the [BLEU score](concepts/bleu-score.md?WT.mc_id=aiml-43548-heboelma).| **Be strict**. Compose test documents to be optimally representative of what you plan to translate in the future. |
+| Phrase dictionary | Forces the given translation 100% of the time. | **Be restrictive**. A phrase dictionary is case-sensitive and any word or phrase listed is translated in the way you specify. In many cases, it's better to not use a phrase dictionary and let the system learn. |
+| Sentence dictionary | Forces the given translation 100% of the time. | **Be strict**. A sentence dictionary is case-insensitive and good for common in domain short sentences. For a sentence dictionary match to occur, the entire submitted sentence must match the source dictionary entry. If only a portion of the sentence matches, the entry doesn't match. |
+
+## What is a BLEU score?
+
+BLEU (Bilingual Evaluation Understudy) is an algorithm for evaluating the precision or accuracy of text that is machine translated from one language to another. Custom Translation uses the BLEU metric as one way of conveying translation accuracy.
+
+A BLEU score is a number between zero and 100. A score of zero indicates a low quality translation where nothing in the translation matched the reference. A score of 100 indicates a perfect translation that is identical to the reference. It's not necessary to attain a score of 100 - a BLEU score between 40 and 60 indicates a high-quality translation.
+
+[Read more](concepts/bleu-score.md?WT.mc_id=aiml-43548-heboelma)
+
+## What happens if I don't submit tuning or testing data?
+
+Tuning and test sentences are optimally representative of what you plan to translate in the future. If you don't submit any tuning or testing data, Custom Translation automatically excludes sentences from your training documents to use as tuning and test data.
+
+| System-generated | Manual-selection |
+|---|---|
+| Convenient. | Enables fine-tuning for your future needs.|
+| Good, if you know that your training data is representative of what you are planning to translate. | Provides more freedom to compose your training data.|
+| Easy to redo when you grow or shrink the domain. | Allows for more data and better domain coverage.|
+|Changes each training run.| Remains static over repeated training runs|
+
+## How is training material processed by Custom Translation?
+
+To prepare for training, documents undergo a series of processing and filtering steps. Knowledge of the filtering process can help with understanding the sentence count displayed as well as the steps you can take to prepare training documents for training with Custom Translation. The filtering steps are as follows:
+
+* ### Sentence alignment
+
+  If your document isn't in `XLIFF`, `XLSX`, `TMX`, or `ALIGN` format, Custom Translation aligns the sentences of your source and target documents to each other, sentence-by-sentence. Translator doesn't perform document alignment—it follows your naming convention for the documents to find a matching document in the other language. Within the source text, Custom Translation tries to find the corresponding sentence in the target language. It uses document markup like embedded HTML tags to help with the alignment.
+
+  If you see a large discrepancy between the number of sentences in the source and target documents, your source document can't be parallel, or couldn't be aligned. The document pairs with a large difference (>10%) of sentences on each side warrant a second look to make sure they're indeed parallel.
+
+* ### Tuning and testing data extraction
+
+  Tuning and testing data is optional. If you don't provide it, the system removes an appropriate percentage from your training documents to use for tuning and testing. The removal happens dynamically as part of the training process. Since this step occurs as part of training, your uploaded documents aren't affected. You can see the final used sentence counts for each category of data—training, tuning, testing, and dictionary—on the Model details page after training succeeds.
+
+* ### Length filter
+
+  * Removes sentences with only one word on either side.
+  * Removes sentences with more than 100 words on either side. Chinese, Japanese, Korean are exempt.
+  * Removes sentences with fewer than three characters. Chinese, Japanese, Korean are exempt.
+  * Removes sentences with more than 2,000 characters for Chinese, Japanese, Korean.
+  * Removes sentences with less than 1% alphanumeric characters.
+  * Removes dictionary entries containing more than 50 words.
+
+* ### White space
+
+  * Replaces any sequence of white-space characters including tabs and CR/LF sequences with a single space character.
+  * Removes leading or trailing space in the sentence.
+
+* ### Sentence end punctuation
+
+  * Replaces multiple sentence-end punctuation characters with a single instance. Japanese character normalization.
+
+  * Converts full width letters and digits to half-width characters.
+
+* ### Unescaped XML tags
+
+   Transforms unescaped tags into escaped tags:
+
+  | Tag | Becomes |
+  |---|---|
+  | \< | \<  |
+  | \> | \>  |
+  | \& | \& |
+
+* ### Invalid characters
+
+  Custom Translation removes sentences that contain Unicode character U+FFFD. The character U+FFFD indicates a failed encoding conversion.
+
+* ### Invalid HTML tags
+
+  Custom Translation removes valid tags during training. Invalid tags cause unpredictable results and should be manually removed. 
+
+## What steps should I take before uploading data?
+
+* Remove sentences with invalid encoding.
+* Remove Unicode control characters.
+* Align sentences (source-to-target), if feasible.
+* Remove source and target sentences that don't match the source and target languages.
+* When source and target sentences have mixed languages, ensure that untranslated words are intentional, for example, names of organizations and products.
+* Avoid teaching errors to your model by making certain that grammar and typography are correct.
+* Have one source sentence mapped to one target sentence. Although our training process handles source and target lines containing multiple sentences, one-to-one mapping is a best practice.
+* Remove invalid HTML tags before uploading training data.
+
+## How do I evaluate the results?
+
+After your model is successfully trained, you can view the model's BLEU score and baseline model BLEU score on the model details page. We use the same set of test data to generate both the model's BLEU score and the baseline BLEU score. This data helps you make an informed decision regarding which model would be better for your use-case.
+
+## Next steps
+
+> [!div class="nextstepaction"]
+> [Try create project](../azure-ai-foundry/how-to-custom-translation-create-project.md)

articles/ai-services/translator/custom-translator/azure-ai-foundry/concepts/bleu-score.md

@@ -0,0 +1,47 @@
+---
+title: "What is a BLEU score? - Custom Translation"
+titleSuffix: Azure AI services
+description: BLEU is a measurement of the differences between machine translation and human-created reference translations of the same source sentence.
+author: laujan
+manager: nitinme
+ms.service: azure-ai-translator
+ms.topic: conceptual
+ms.date: 01/27/2025
+ms.author: lajanuar
+ms.custom: cogserv-non-critical-translator
+#Customer intent: As an Custom Translation user, I want to understand how BLEU score works so that I understand system test outcome better.
+---
+
+# What is a BLEU score?
+
+[BLEU (Bilingual Evaluation Understudy)](https://en.wikipedia.org/wiki/BLEU) is a measurement of the difference between an automatic translation and human-created reference translations of the same source sentence.
+
+## Scoring process
+
+The BLEU algorithm compares consecutive phrases of the automatic translation
+with the consecutive phrases it finds in the reference translation, and counts
+the number of matches, in a weighted fashion. These matches are position
+independent. A higher match degree indicates a higher degree of similarity with
+the reference translation, and higher score. Intelligibility and grammatical correctness aren't taken into account.
+
+## How BLEU works?
+
+The BLEU score's strength is that it correlates well with human judgment. BLEU averages out
+individual sentence judgment errors over a test corpus, rather than attempting
+to devise the exact human judgment for every sentence.
+
+A more extensive discussion of BLEU scores is [here](https://youtu.be/-UqDljMymMg).
+
+BLEU results depend strongly on the breadth of your domain; consistency of
+test, training and tuning data; and how much data you have
+available for training. If your models are trained within a narrow domain, and
+your training data is consistent with your test data, you can expect a high
+BLEU score.
+
+>[!NOTE]
+>A comparison between BLEU scores is only justifiable when BLEU results are compared with the same Test set, the same language pair, and the same MT engine. A BLEU score from a different test set is bound to be different.
+
+## Next steps
+
+> [!div class="nextstepaction"]
+> [BLEU score evaluation](../how-to-custom-translation-test-model.md)

articles/ai-services/translator/custom-translator/azure-ai-foundry/concepts/customization.md

@@ -0,0 +1,39 @@
+---
+title: Translation Customization - Custom Translation
+titleSuffix: Azure AI services
+description: Use the Microsoft Translator Hub to build your own machine translation system using your preferred terminology and style.
+#services: cognitive-services
+author: laujan
+manager: nitinme
+ms.service: azure-ai-translator
+ms.topic: conceptual
+ms.date: 01/28/2025
+ms.author: lajanuar
+---
+
+# Customize your text translations
+
+Custom Translation is a feature of the Azure AI Translator service. Custom Translation allows users to customize Azure AI Translator's advanced neural machine translation when translating text using Translator (version 3 only).
+
+The feature can also be used to customize speech translation when used with [Azure AI Speech](../../../speech-service/index.yml).
+
+## Custom Translation
+
+With Custom Translator, you can build neural translation systems that understand the terminology used in your own business and industry. The customized translation system integrates into existing applications, workflows, and websites.
+
+### How does it work?
+
+Use your previously translated documents (leaflets, webpages, documentation, etc.) to build a translation system that reflects your domain-specific terminology and style, better than a standard translation system. Users can upload `TMX`,`XLIFF`,`TXT`, `DOCX`, and `XLSX` documents.  
+
+The system also accepts data that is parallel at the document level but isn't yet aligned at the sentence level. If users have access to versions of the same content in multiple languages but in separate documents, Custom Translator os able to automatically match sentences across documents. The system can also use monolingual data in either or both languages to complement the parallel training data to improve the translations.
+
+The customized system is then available through a regular call to Translator using the category parameter.
+
+Given the appropriate type and amount of training data it isn't uncommon to expect gains between 5 and 10, or even more `BLEU` points on translation quality by using Custom Translator.
+
+More details about the various levels of customization based on available data can be found in the [Custom Translation User Guide](../overview.md).
+
+## Next steps
+
+> [!div class="nextstepaction"]
+> [Set up a customized language system using Custom Translation](../overview.md)

articles/ai-services/translator/custom-translator/azure-ai-foundry/concepts/data-filtering.md

@@ -0,0 +1,65 @@
+---
+title: "Data Filtering - Custom Translation"
+titleSuffix: Azure AI services
+description: When you submit documents to be used for training a custom system, the documents undergo a series of processing and filtering steps.
+author: laujan
+manager: nitinme
+ms.service: azure-ai-translator
+ms.date: 01/28/2025
+ms.author: lajanuar
+ms.topic: conceptual
+ms.custom: cogserv-non-critical-translator
+#Customer intent: As a Custom Translation, I want to understand how data is filtered before training a model.
+---
+
+# Custom Translation Data filtering
+
+When you submit documents to be used for training, the documents undergo a series of processing and filtering steps. These steps are explained here. The knowledge of the filtering can help you understand the sentence count displayed in Custom Translation and the steps you can take yourself to prepare the documents for training with Custom Translation.
+
+## Sentence alignment
+
+If your document isn't in XLIFF, `TMX`, or ALIGN format, Custom Translation aligns the sentences of your source and target documents to each other, sentence by sentence. Custom Translation doesn't perform document alignment – it follows your naming of the documents to find the matching document of the other language. Within the document, Custom Translation tries to find the corresponding sentence in the other language. It uses document markup like embedded HTML tags to help with the alignment.
+
+If you see a large discrepancy between the number of sentences in the source and target documents, your documents can't be parallel. The document pairs with a large difference (>10%) of sentences on each side warrant a second look to make sure they're indeed parallel. Custom Translation shows a warning next to the document if the sentence count differs suspiciously.
+
+## Deduplication
+
+Custom Translation removes the sentences that are present in test and tuning documents from training data. The removal happens dynamically inside of the training run, not in the data processing step. Custom Translation reports the sentence count to you in the project overview before such removal. Deduplication doesn't apply if you choose to upload your own test and tuning documents.
+
+## Length filter
+
+* Remove sentences with only one word on either side.
+* Remove sentences with more than 100 words on either side.  Chinese, Japanese, Korean are exempt.
+* Remove sentences with fewer than three characters. Chinese, Japanese, Korean are exempt.
+* Remove sentences with more than 2,000 characters for Chinese, Japanese, Korean.
+* Remove sentences with less than 1% alpha characters.
+* Remove dictionary entries containing more than 50 words.
+
+## White space
+
+* Replace any sequence of white-space characters including tabs and CR/LF sequences with a single space character.
+* Remove leading or trailing space in the sentence
+
+## Sentence end punctuation
+
+Replace multiple sentence end punctuation characters with a single instance.
+
+## Japanese character normalization
+
+Convert full width letters and digits to half-width characters.
+
+## Unescaped XML tags
+
+Filtering transforms unescaped tags into escaped tags:
+* `<` becomes `<`
+* `>` becomes `>`
+* `&` becomes `&`
+
+## Invalid characters
+
+Custom Translation removes sentences that contain Unicode character U+FFFD. The character U+FFFD indicates a failed encoding conversion.
+
+## Next steps
+
+> [!div class="nextstepaction"]
+> [Learn how to train a model](../how-to-custom-translation-train-model.md)