feat(machinery): add Argos offline translation backend#18572
feat(machinery): add Argos offline translation backend#18572knk125 wants to merge 4 commits intoWeblateOrg:mainfrom
Conversation
|
Can you please add tests? |
There was a problem hiding this comment.
Pull request overview
Adds a new offline machine translation backend based on Argos Translate, integrating it into Weblate’s machinery registry and packaging extras so it can be enabled as a service.
Changes:
- Added
ArgosTranslationmachinery backend (weblate/machinery/argos.py) usingargostranslateinstalled models. - Registered the backend in
WEBLATE_MACHINERYso it’s discoverable/configurable. - Added an
argosoptional dependency extra and included it in theallextra.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
weblate/machinery/models.py |
Registers the new machinery class so it can be loaded by the machinery class loader. |
weblate/machinery/argos.py |
Implements Argos Translate-backed offline translation and language-pair support checks. |
pyproject.toml |
Adds argostranslate as an optional dependency and wires it into the all extra. |
| try: | ||
| translation = argostranslate.translate.translate(text, src, tgt) | ||
| if translation: | ||
| yield { | ||
| "text": translation, | ||
| "quality": self.max_score, | ||
| "service": self.name, | ||
| "source": text, | ||
| } | ||
| except Exception as e: | ||
| logger.debug( | ||
| "Argos translation failed for %s to %s: %s", | ||
| source_language, | ||
| target_language, | ||
| e, | ||
| ) | ||
| return |
There was a problem hiding this comment.
ArgosTranslation swallows all exceptions during translation and only logs a debug message, which can silently hide real misconfigurations or runtime failures (users will just see no suggestions). Let the exception propagate so Weblate can wrap/report it, or catch only expected exceptions and raise a MachineTranslationError (and/or call self.report_error(...)) so failures are visible and diagnosable.
| try: | ||
| import argostranslate.package | ||
| import argostranslate.translate | ||
| except ImportError: | ||
| argostranslate = None | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| class ArgosTranslation(MachineTranslation): | ||
| """Argos offline machine translation.""" | ||
|
|
||
| name = "Argos Translate" | ||
| settings_form = None | ||
|
|
||
| def is_supported(self, source_language, target_language): | ||
| """Supported if the language model package is installed in argostranslate.""" | ||
| if not argostranslate: | ||
| return False |
There was a problem hiding this comment.
This backend keeps is_available as the default True even when argostranslate isn't installed (it just sets the module variable to None). That allows the service to appear installable/enabled in the UI but it will never return translations. Consider setting is_available = argostranslate is not None (or similar) so the service is clearly unavailable when the dependency is missing.
| src = source_language.split("-")[0].lower() | ||
| tgt = target_language.split("-")[0].lower() | ||
|
|
||
| installed_languages = argostranslate.translate.get_installed_languages() | ||
|
|
||
| src_lang = next( | ||
| (lang for lang in installed_languages if lang.code == src), None | ||
| ) | ||
| tgt_lang = next( | ||
| (lang for lang in installed_languages if lang.code == tgt), None | ||
| ) |
There was a problem hiding this comment.
is_supported() calls argostranslate.translate.get_installed_languages() on every probe and then linearly scans the list twice. Since get_languages() can call is_supported() multiple times per request (language fallback probing), this can add noticeable overhead. Consider caching the installed languages / available pairs (e.g., with a cached_property + explicit cache invalidation) and using a dict lookup instead of repeated linear scans.
| class ArgosTranslation(MachineTranslation): | ||
| """Argos offline machine translation.""" | ||
|
|
||
| name = "Argos Translate" | ||
| settings_form = None | ||
|
|
||
| def is_supported(self, source_language, target_language): | ||
| """Supported if the language model package is installed in argostranslate.""" | ||
| if not argostranslate: | ||
| return False | ||
|
|
||
| src = source_language.split("-")[0].lower() | ||
| tgt = target_language.split("-")[0].lower() | ||
|
|
||
| installed_languages = argostranslate.translate.get_installed_languages() | ||
|
|
||
| src_lang = next( | ||
| (lang for lang in installed_languages if lang.code == src), None | ||
| ) | ||
| tgt_lang = next( | ||
| (lang for lang in installed_languages if lang.code == tgt), None | ||
| ) | ||
|
|
||
| if src_lang and tgt_lang: | ||
| return src_lang.get_translation(tgt_lang) is not None | ||
|
|
||
| return False | ||
|
|
||
| def download_translations( | ||
| self, | ||
| source_language, | ||
| target_language, | ||
| text: str, | ||
| unit: Unit | None, | ||
| user: User | None, | ||
| threshold: int = 75, | ||
| ) -> DownloadTranslations: |
There was a problem hiding this comment.
A new machinery backend is introduced, but there are no corresponding tests added. weblate/machinery/tests.py contains comprehensive coverage for other backends; adding tests that mock argostranslate.translate (to cover is_supported() and download_translations() behavior, and the dependency-missing path) would help prevent regressions.
| import argostranslate.package | ||
| import argostranslate.translate | ||
| except ImportError: | ||
| argostranslate = None |
There was a problem hiding this comment.
This logic should not be needed. When the module cannot be imported, the engine is skipped by Weblate.
knk125
force-pushed
the
feature/argos-translation
branch
from
a9deab0 to
1954957
Compare
Add Argos Translate Offline Machine Translation Backend (issiue: 13106)
Description
This Pull Request introduces a new offline machine translation backend utilizing Argos Translate.
Argos Translate is an open-source offline translation library written in Python. It is based on OpenNMT (Open Neural Machine Translation) and runs inference locally using CTranslate2, providing high-quality translations without relying on third-party external APIs, thereby ensuring complete data privacy.
Implementation Details
Dependency Addition: Added argostranslate as an optional Python dependency under the [all] and [argos] extras in
pyproject.toml
.
Backend Engine: Created
weblate/machinery/argos.py
, which implements the
ArgosTranslation
class extending the base
MachineTranslation
engine.
The engine leverages argostranslate.translate.get_installed_languages() to dynamically determine which language pairs are locally supported via the
is_supported
method.
The
download_translations
method handles querying the local CTranslate2 models synchronously and yielding structured translation suggestions.
Registration: Registered "weblate.machinery.argos.ArgosTranslation" into the WEBLATE_MACHINERY tuple in
weblate/machinery/models.py
.
Testing methodology
Initialized a component inside a local Weblate instance and explicitly enabled the Argos Translate service within the project's machinery_settings config.
Installed the English-Spanish Argos model via Python bindings.
Passed a mock English source string unit to the API endpoint and received accurate translations on the Weblate UI under the "Automatic suggestions" tab.
No external cloud API keys are required for this engine to function since inference executes entirely on the local device's CPU/GPU.