index.bs

<pre class='metadata'>
Title: Translator and Language Detector APIs
Shortname: translation
Level: None
Status: CG-DRAFT
Group: webml
Repository: webmachinelearning/translation-api
URL: https://webmachinelearning.github.io/translation-api
Former Editor: Domenic Denicola, Google https://google.com, d@domenic.me, https://domenic.me/
Editor: Reilly Grant, Google https://www.google.com, reillyg@google.com, w3cid 83788
Editor: Ehsan Toreini, Samsung https://samsung.com/, e.toreini@samsung.com, https://toreini.github.io, w3cid 115969
Abstract: The translator and language detector APIs gives web pages the ability to translate text between languages, and detect the language of such text.
Markup Shorthands: markdown yes, css no
Complain About: accidental-2119 yes, missing-example-ids yes
Assume Explicit For: yes
Default Biblio Status: current
Boilerplate: omit conformance
Indent: 2
Die On: warning
</pre>

<pre class="anchors">
urlPrefix: https://tc39.es/ecma402/; spec: ECMA-402
  type: dfn
    text: Unicode canonicalized locale identifier; url: sec-language-tags
  type: abstract-op
    text: LookupMatchingLocaleByBestFit; url: sec-lookupmatchinglocalebybestfit
urlPrefix: https://tc39.es/ecma262/; spec: ECMA-262
  type: dfn
    text: current realm; url: current-realm
urlPrefix: https://whatpr.org/webidl/1465.html; spec: WEBIDL
  type: interface
    text: QuotaExceededError; url: quotaexceedederror
  type: dfn; for: QuotaExceededError
    text: requested; url: quotaexceedederror-requested
    text: quota; url: quotaexceedederror-quota
</pre>

<h2 id="intro">Introduction</h2>

The translator and language detector APIs expose the ability to translate text between human languages, and detect the language of such text. They are complementary to any built-in browser UI features for these purposes, giving web developers the ability to trigger these operations programmatically and integrate them into their applications. This can be especially useful for operating on user input, or text retrieved from the network.

These APIs are designed to provide a high-level interface for translation and language detection, abstracting away the complexities of underlying machine learning models and their management. To deal with possible interoperability issues arising from different implementation strategies or language support, the API design guides developers toward checking the availability of languages their applications are dependent on, and including appropriate error-handling.

<h2 id="dependencies">Dependencies</h2>

This specification depends on the Infra Standard. [[!INFRA]]

As with the rest of the web platform, human languages are identified in these APIs by BCP 47 language tags, such as "`ja`", "`en-US`", "`sr-Cyrl`", or "`de-CH-1901-x-phonebk-extended`". The specific algorithms used for validation, canonicalization, and language tag matching are those from the <cite>ECMAScript Internationalization API Specification</cite>, which in turn defers some of its processing to <cite>Unicode Locale Data Markup Language (LDML)</cite>. [[BCP47]] [[!ECMA-402]] [[UTS35]].

These APIs are part of a family of APIs expected to be powered by machine learning models, which share common API surface idioms and specification patterns. Currently, the specification text for these shared parts lives in [[WRITING-ASSISTANCE-APIS#supporting]], and the common privacy and security considerations are discussed in [[WRITING-ASSISTANCE-APIS#privacy]] and [[WRITING-ASSISTANCE-APIS#security]]. Implementing these APIs requires implementing that shared infrastructure, and conforming to those privacy and security considerations. But it does not require implementing or exposing the actual writing assistance APIs. [[!WRITING-ASSISTANCE-APIS]]

<h2 id="translator-api">The translator API</h2>

<xmp class="idl">
[Exposed=Window, SecureContext]
interface Translator {
  static Promise<Translator> create(TranslatorCreateOptions options);
  static Promise<Availability> availability(TranslatorCreateCoreOptions options);

  Promise<DOMString> translate(
    DOMString input,
    optional TranslatorTranslateOptions options = {}
  );
  ReadableStream translateStreaming(
    DOMString input,
    optional TranslatorTranslateOptions options = {}
  );

  readonly attribute DOMString sourceLanguage;
  readonly attribute DOMString targetLanguage;

  Promise<double> measureInputUsage(
    DOMString input,
    optional TranslatorTranslateOptions options = {}
  );
  readonly attribute unrestricted double inputQuota;
};
Translator includes DestroyableModel;

dictionary TranslatorCreateCoreOptions {
  required DOMString sourceLanguage;
  required DOMString targetLanguage;
};

dictionary TranslatorCreateOptions : TranslatorCreateCoreOptions {
  AbortSignal signal;
  CreateMonitorCallback monitor;
};

dictionary TranslatorTranslateOptions {
  AbortSignal signal;
};
</xmp>

<h3 id="translator-creation">Creation</h3>

<div algorithm>
  The static <dfn method for="Translator">create(|options|)</dfn> method steps are:

  1. Return the result of [=creating an AI model object=] given |options|, "{{translator}}", [=validate and canonicalize translator options=], [=compute translator options availability=], [=download the translation model=], [=initialize the translation model=], and [=create the translator object=].
</div>

<div algorithm>
  To <dfn>validate and canonicalize translator options</dfn> given a {{TranslatorCreateCoreOptions}} |options|, perform the following steps. They mutate |options| in place to canonicalize language tags, and throw an exception if any are invalid.

  1. [=Validate and canonicalize language tags=] given |options| and "{{TranslatorCreateCoreOptions/sourceLanguage}}".

  1. [=Validate and canonicalize language tags=] given |options| and "{{TranslatorCreateCoreOptions/targetLanguage}}".
</div>

<div algorithm>
  To <dfn>download the translation model</dfn>, given a {{TranslatorCreateCoreOptions}} |options|:

  1. [=Assert=]: these steps are running [=in parallel=].

  1. Initiate the download process for everything the user agent needs to translate text from |options|["{{TranslatorCreateCoreOptions/sourceLanguage}}"] to |options|["{{TranslatorCreateCoreOptions/targetLanguage}}"].

    This could include both a base translation model and specific language arc material, or perhaps material for multiple language arcs if an intermediate language is used.

  1. If the download process cannot be started for any reason, then return false.

  1. Return true.
</div>

<div algorithm>
  To <dfn>initialize the translation model</dfn>, given a {{TranslatorCreateCoreOptions}} |options|:

  1. [=Assert=]: these steps are running [=in parallel=].

  1. Perform any necessary initialization operations for the AI model backing the user agent's capabilities for translating from |options|["{{TranslatorCreateCoreOptions/sourceLanguage}}"] to |options|["{{TranslatorCreateCoreOptions/targetLanguage}}"].

    This could include loading the model into memory, or loading any fine-tunings necessary to support the specific options in question.

  1. If initialization failed for any reason, then return a [=DOMException error information=] whose [=DOMException error information/name=] is "{{OperationError}}" and whose [=DOMException error information/details=] contain appropriate detail.

  1. Return null.
</div>

<div algorithm>
  To <dfn>create the translator object</dfn>, given a [=ECMAScript/realm=] |realm| and a {{TranslatorCreateCoreOptions}} |options|:

  1. [=Assert=]: these steps are running on |realm|'s [=ECMAScript/surrounding agent=]'s [=agent/event loop=].

  1. Let |inputQuota| be the amount of input quota that is available to the user agent for future [=translate|translation=] operations. (This value is [=implementation-defined=], and may be +∞ if there are no specific limits beyond, e.g., the user's memory, or the limits of JavaScript strings.)

  1. Return a new {{Translator}} object, created in |realm|, with

    <dl class="props">
      : [=Translator/source language=]
      :: |options|["{{TranslatorCreateCoreOptions/sourceLanguage}}"]

      : [=Translator/target language=]
      :: |options|["{{TranslatorCreateCoreOptions/targetLanguage}}"]

      : [=Translator/input quota=]
      :: |inputQuota|
    </dl>
</div>

<h3 id="translator-availability">Availability</h3>

<div algorithm>
  The static <dfn method for="Translator">availability(|options|)</dfn> method steps are:

  1. Return the result of [=computing AI model availability=] given |options|, "{{translator}}", [=validate and canonicalize translator options=], and [=compute translator options availability=].
</div>

<div algorithm>
  To <dfn>compute translator options availability</dfn> given a {{TranslatorCreateCoreOptions}} |options|, perform the following steps. They return either an {{Availability}} value or null, and they mutate |options| in place to update language tags to their best-fit matches.

  1. [=Assert=]: this algorithm is running [=in parallel=].

  1. Let |availabilities| be the user agent's [=translator language arc availabilities=].

  1. If |availabilities| is null, then return null.

  1. [=map/For each=] |languageArc| → |availability| in |availabilities|:

    1. Let |sourceLanguageBestFit| be [$LookupMatchingLocaleByBestFit$](« |languageArc|'s [=language arc/source language=] », « |options|["{{TranslatorCreateCoreOptions/sourceLanguage}}"] »).

    1. Let |targetLanguageBestFit| be [$LookupMatchingLocaleByBestFit$](« |languageArc|'s [=language arc/target language=] », « |options|["{{TranslatorCreateCoreOptions/targetLanguage}}"] »).

    1. If |sourceLanguageBestFit| and |targetLanguageBestFit| are both not undefined, then:

      1. Set |options|["{{TranslatorCreateCoreOptions/sourceLanguage}}"] to |sourceLanguageBestFit|.\[[locale]].

      1. Set |options|["{{TranslatorCreateCoreOptions/targetLanguage}}"] to |targetLanguageBestFit|.\[[locale]].

      1. Return |availability|.

  1. If (|options|["{{TranslatorCreateCoreOptions/sourceLanguage}}"], |options|["{{TranslatorCreateCoreOptions/targetLanguage}}"]) [=language arc/can be fulfilled by the identity translation=], then return "{{Availability/available}}".

     <p class="note">Such cases could also return "{{Availability/downloadable}}", "{{Availability/downloading}}", or "{{Availability/available}}" because of the above steps, if the user agent has specific entries in its [=translator language arc availabilities=] for the given language arc. However, the identity translation is always available, so this step ensures that we never return "{{Availability/unavailable}}" for such cases.

     <div class="example" id="example-identity-translation">
      <p>One [=language arc=] that [=language arc/can be fulfilled by the identity translation=] is (`"en-US"`, `"en-GB"`). It is conceivable that an implementation might support a specialized model for this translation, which would show up in the [=translator language arc availabilities=].

      <p>On the other hand, it's pretty unlikely that an implementation has any specialized model for the [=language arc=] ("`en-x-asdf`", "`en-x-xyzw`"). In such a case, this step takes over, and later calls to the [=translate=] algorithm will use the identity translation.

      <p>Note that when this step takes over, |options|["{{TranslatorCreateCoreOptions/sourceLanguage}}"] and |options|["{{TranslatorCreateCoreOptions/targetLanguage}}"] are not modified, so if this algorithm is being called from {{Translator/create()}}, that means the resulting {{Translator}} object's {{Translator/sourceLanguage}} and {{Translator/targetLanguage}} properties will return the original inputs, and not some canonicalized form.
     </div>

  1. Return "{{Availability/unavailable}}".
</div>

A <dfn>language arc</dfn> is a [=tuple=] of two strings, a <dfn for="language arc">source language</dfn> and a <dfn for="language arc">target language</dfn>. Each item is a [=Unicode canonicalized locale identifier=].

<div algorithm>
  The <dfn>translator language arc availabilities</dfn> are given by the following steps. They return a [=map=] from [=language arcs=] to {{Availability}} values, or null.

  1. [=Assert=]: this algorithm is running [=in parallel=].

  1. If there is some error attempting to determine what language arcs the user agent [=model availability/can support=] translating text between, which the user agent believes to be transient (such that re-querying could stop producing such an error), then return null.

  1. Return a [=map=] from [=language arcs=] to {{Availability}} values, where each key is a [=language arc=] that the user agent supports translating text between, filled according to the following constraints:

    * If the user agent [=model availability/currently supports=] translating text from the [=language arc/source language=] to the [=language arc/target language=] of the [=language arc=], then the map must contain an [=map/entry=] whose [=map/key=] is that [=language arc=] and whose [=map/value=] is "{{Availability/available}}".

    * If the user agent believes it will be able to [=model availability/support=] translating text from the [=language arc/source language=] to the [=language arc/target language=] of the [=language arc=], but only after finishing a download that is already ongoing, then the map must contain an [=map/entry=] whose [=map/key=] is that [=language arc=] and whose [=map/value=] is "{{Availability/downloading}}".

    * If the user agent believes it will be able to [=model availability/support=] translating text from the [=language arc/source language=] to the [=language arc/target language=] of the [=language arc=], but only after performing a not-currently ongoing download, then the map must contain an [=map/entry=] whose [=map/key=] is that [=language arc=] and whose [=map/value=] is "{{Availability/downloadable}}".

    * The [=map/keys=] must not include any [=language arcs=] that [=language arc/overlap=] with the other [=map/keys=].
</div>

<div class="example" id="example-language-arc-support">
  Let's suppose that the user agent's [=translator language arc availabilities=] are as follows:

  * ("`en`", "`zh-Hans`") → "{{Availability/available}}"
  * ("`en`", "`zh-Hant`") → "{{Availability/downloadable}}"

  The use of [$LookupMatchingLocaleByBestFit$] means that {{Translator/availability()}} will probably give the following answers:

    <xmp class="language-js">
    function a(sourceLanguage, targetLanguage) {
      return ai.translator.availability({ sourceLanguage, targetLanguage }):
    }

    await a("en", "zh-Hans") === "available";
    await a("en", "zh-Hant") === "downloadable";

    await a("en", "zh") === "available";            // zh will best-fit to zh-Hans

    await a("en", "zh-TW") === "downloadable";      // zh-TW will best-fit to zh-Hant
    await a("en", "zh-HK") === "available";         // zh-HK will best-fit to zh-Hans
    await a("en", "zh-CN") === "available";         // zh-CN will best-fit to zh-Hans

    await a("en-US", "zh-Hant") === "downloadable"; // en-US will best-fit to en
    await a("en-GB", "zh-Hant") === "downloadable"; // en-GB will best-fit to en

    // Even very unexpected subtags will best-fit to en or zh-Hans
    await a("en-Braille-x-lolcat", "zh-Hant") === "downloadable";
    await a("en", "zh-BR-Kana") === "available";
    </xmp>
</div>

<div algorithm>
  A [=language arc=] |arc| <dfn for="language arc">overlaps</dfn> with a [=set=] of [=language arcs=] |otherArcs| if the following steps return true:

  1. Let |sourceLanguages| be the [=set=] composed of the [=language arc/source languages=] of each [=set/item=] in |otherArcs|.

  1. If [$LookupMatchingLocaleByBestFit$](|sourceLanguages|, « |arc|'s [=language arc/source language=] ») is not undefined, then return true.

  1. Let |targetLanguages| be the [=set=] composed of the [=language arc/target languages=] of each [=set/item=] in |otherArcs|.

  1. If [$LookupMatchingLocaleByBestFit$](|targetLanguages|, « |arc|'s [=language arc/target language=] ») is not undefined, then return true.

  1. Return false.
</div>

<div class="example" id="example-language-arc-overlap">
  The [=language arc=] ("`en`", "`fr`") [=language arc/overlaps=] with « ("`en`", "`fr-CA`") », so the user agent's [=translator language arc availabilities=] cannot contain both of these [=language arcs=] at the same time.

  Instead, a typical user agent will either support only one English-to-French language arc (presumably ("`en`", "`fr`")), or it could support multiple non-overlapping English-to-French language arcs, such as ("`en`", "`fr-FR`"), ("`en`", "`fr-CA`"), and ("`en`", "`fr-CH`").

  In the latter case, if the web developer requested to create a translator using <code highlight="js">ai.translator.create({ sourceLanguage: "en", targetLanguage: "fr" })</code>, the [$LookupMatchingLocaleByBestFit$] algorithm would choose one of the three possible language arcs to use (presumably ("`en`", "`fr-FR`")).
</div>

<div algorithm>
  A [=language arc=] |arc| <dfn for="language arc">can be fulfilled by the identity translation</dfn> if the following steps return true:

  1. If [$LookupMatchingLocaleByBestFit$](« |arc|'s [=language arc/source language=] », « |arc|'s [=language arc/target language=] ») is not undefined, then return true.

  1. If [$LookupMatchingLocaleByBestFit$](« |arc|'s [=language arc/target language=] », « |arc|'s [=language arc/source language=] ») is not undefined, then return true.

  1. Return false.
</div>

<h3 id="the-Translator-class">The {{Translator}} class</h3>

Every {{Translator}} has a <dfn for="Translator">source language</dfn>, a [=string=], set during creation.

Every {{Translator}} has a <dfn for="Translator">target language</dfn>, a [=string=], set during creation.

Every {{Translator}} has an <dfn for="Translator">input quota</dfn>, a number, set during creation.

<hr>

The <dfn attribute for="Translator">sourceLanguage</dfn> getter steps are to return [=this=]'s [=Translator/source language=].

The <dfn attribute for="Translator">targetLanguage</dfn> getter steps are to return [=this=]'s [=Translator/target language=].

The <dfn attribute for="Translator">inputQuota</dfn> getter steps are to return [=this=]'s [=Translator/input quota=].

<hr>

<div algorithm>
  The <dfn method for="Translator">translate(|input|, |options|)</dfn> method steps are:

  1. Let |operation| be an algorithm step which takes arguments |chunkProduced|, |done|, |error|, and |stopProducing|, and [=translates=] |input| given [=this=]'s [=Translator/source language=], [=this=]'s [=Translator/target language=], [=this=]'s [=Translator/input quota=], |chunkProduced|, |done|, |error|, and |stopProducing|.

  1. Return the result of [=getting an aggregated AI model result=] given [=this=], |options|, and |operation|.
</div>

<div algorithm>
  The <dfn method for="Translator">translateStreaming(|input|, |options|)</dfn> method steps are:

  1. Let |operation| be an algorithm step which takes arguments |chunkProduced|, |done|, |error|, and |stopProducing|, and [=translates=] |input| given [=this=]'s [=Translator/source language=], [=this=]'s [=Translator/target language=], [=this=]'s [=Translator/input quota=], |chunkProduced|, |done|, |error|, and |stopProducing|.

  1. Return the result of [=getting a streaming AI model result=] given [=this=], |options|, and |operation|.
</div>

<div algorithm>
  The <dfn method for="Translator">measureInputUsage(|input|, |options|)</dfn> method steps are:

  1. Let |measureUsage| be an algorithm step which takes argument |stopMeasuring|, and returns the result of [=measuring translator input usage=] given |input|, [=this=]'s [=Translator/source language=], [=this=]'s [=Translator/target language=], and |stopMeasuring|.

  1. Return the result of [=measuring AI model input usage=] given [=this=], |options|, and |measureUsage|.
</div>

<h3 id="translator-translation">Translation</h3>

<h4 id="translator-algorithm">The algorithm</h4>

<div algorithm>
  To <dfn>translate</dfn> given:

  * a [=string=] |input|,
  * a [=Unicode canonicalized locale identifier=] |sourceLanguage|,
  * a [=Unicode canonicalized locale identifier=] |targetLanguage|,
  * a number |inputQuota|,
  * an algorithm |chunkProduced| that takes a string and returns nothing,
  * an algorithm |done| that takes no arguments and returns nothing,
  * an algorithm |error| that takes [=error information=] and returns nothing, and
  * an algorithm |stopProducing| that takes no arguments and returns a boolean,

  perform the following steps:

  1. [=Assert=]: this algorithm is running [=in parallel=].

  1. Let |requested| be the result of [=measuring translator input usage=] given |input|, |sourceLanguage|, |targetLanguage|, and |stopProducing|.

  1. If |requested| is null, then return.

  1. If |requested| is an [=error information=], then:

    1. Perform |error| given |requested|.

    1. Return.

  1. [=Assert=]: |requested| is a number.

  1. If |requested| is greater than |inputQuota|, then:

    1. Let |errorInfo| be a [=quota exceeded error information=] with a [=quota exceeded error information/requested=] of |requested| and a [=quota exceeded error information/quota=] of |inputQuota|.

    1. Perform |error| given |errorInfo|.

    1. Return.

    <p class="note">In reality, we expect that implementations will check the input usage against the quota as part of the same call into the model as the translation itself. The steps are only separated in the specification for ease of understanding.

  1. In an [=implementation-defined=] manner, subject to the following guidelines, begin the processs of translating |input| from |sourceLanguage| into |targetLanguage|.

     If |input| is the empty string, or otherwise consists of no translatable content (e.g., only contains whitespace, or control characters), then the resulting translation should be |input|. In such cases, |sourceLanguage| and |targetLanguage| should be ignored.

     If (|sourceLanguage|, |targetLanguage|) [=language arc/can be fulfilled by the identity translation=], then the resulting translation should be |input|.

     The translation process must conform to the guidance given in [[WRITING-ASSISTANCE-APIS#privacy]] and [[WRITING-ASSISTANCE-APIS#security]], notably including (but not limited to) [[WRITING-ASSISTANCE-APIS#privacy-user-input]] and [[WRITING-ASSISTANCE-APIS#security-runtime]].

  1. While true:

    1. Wait for the next chunk of translated text to be produced, for the translation process to finish, or for the result of calling |stopProducing| to become true.

    1. If such a chunk is successfully produced:

      1. Let it be represented as a [=string=] |chunk|.

      1. Perform |chunkProduced| given |chunk|.

    1. Otherwise, if the translation process has finished:

      1. Perform |done|.

      1. [=iteration/Break=].

    1. Otherwise, if |stopProducing| returns true, then [=iteration/break=].

    1. Otherwise, if an error occurred during translation:

      1. Let the error be represented as a [=DOMException error information=] |errorInfo| according to the guidance in [[#translator-errors]].

      1. Perform |error| given |errorInfo|.

      1. [=iteration/Break=].
</div>

<h4 id="translator-usage">Usage</h4>

<div algorithm>
  To <dfn>measure translator input usage</dfn>, given:

  * a [=string=] |input|,
  * a [=Unicode canonicalized locale identifier=] |sourceLanguage|,
  * a [=Unicode canonicalized locale identifier=] |targetLanguage|, and
  * an algorithm |stopMeasuring| that takes no arguments and returns a boolean,

  perform the following steps:

  1. [=Assert=]: this algorithm is running [=in parallel=].

  1. Let |inputToModel| be the [=implementation-defined=] string that would be sent to the underlying model in order to [=translate=] |input| from |sourceLanguage| to |targetLanguage|.

    <p class="note" id="note-input-to-language-detector-model">This might be just |input| itself, if |sourceLanguage| and |targetLanguage| were loaded into the model during initialization. Or it might consist of more, e.g. appropriate quota usage for encoding the languages in question, or some sort of wrapper prompt to a language model.

    If during this process |stopMeasuring| starts returning true, then return null.

    If an error occurs during this process, then return an appropriate [=DOMException error information=] according to the guidance in [[#translator-errors]].

  1. Return the amount of input usage needed to represent |inputToModel| when given to the underlying model. The exact calculation procedure is [=implementation-defined=], subject to the following constraints.

    The returned input usage must be nonnegative and finite. It must be 0, if there are no usage quotas for the translation process (i.e., if the [=Translator/input quota=] is +∞). Otherwise, it must be positive and should be roughly proportional to the [=string/length=] of |inputToModel|.

    <p class="note" id="note-translator-input-usage">This might be the number of tokens needed to represent |input| in a <a href="https://arxiv.org/abs/2404.08335">language model tokenization scheme</a>, or it might be |input|'s [=string/length=]. It could also be some variation of these which also counts the usage of any prefixes or suffixes necessary to give to the model.

    If during this process |stopMeasuring| starts returning true, then instead return null.

    If an error occurs during this process, then instead return an appropriate [=DOMException error information=] according to the guidance in [[#translator-errors]].
</div>

<h4 id="translator-errors">Errors</h4>

When translation fails, the following possible reasons may be surfaced to the web developer. This table lists the possible {{DOMException}} [=DOMException/names=] and the cases in which an implementation should use them:

<table class="data">
  <thead>
    <tr>
      <th>{{DOMException}} [=DOMException/name=]
      <th>Scenarios
  <tbody>
    <tr>
      <td>"{{NotAllowedError}}"
      <td>
        <p>Translation is disabled by user choice or user agent policy.
    <tr>
      <td>"{{NotReadableError}}"
      <td>
        <p>The translation output was filtered by the user agent, e.g., because it was detected to be harmful, inaccurate, or nonsensical.
    <tr>
      <td>"{{UnknownError}}"
      <td>
        <p>All other scenarios, including if the user agent believes it cannot translate and also meet the requirements given in [[WRITING-ASSISTANCE-APIS#privacy]] and [[WRITING-ASSISTANCE-APIS#security]]. Or, if the user agent would prefer not to disclose the failure reason.
</table>

<p class="note">This table does not give the complete list of exceptions that can be surfaced by the translator API. It only contains those which can come from certain [=implementation-defined=] steps.

<h3 id="translator-permissions-policy">Permissions policy integration</h3>

Access to the translator API is gated behind the [=policy-controlled feature=] "<dfn permission>translator</dfn>", which has a [=policy-controlled feature/default allowlist=] of <code>[=default allowlist/'self'=]</code>.

<h2 id="language-detector-api">The language detector API</h2>

<xmp class="idl">
[Exposed=Window, SecureContext]
interface LanguageDetector {
  static Promise<LanguageDetector> create(
    optional LanguageDetectorCreateOptions options = {}
  );
  static Promise<Availability> availability(
    optional LanguageDetectorCreateCoreOptions options = {}
  );

  Promise<sequence<LanguageDetectionResult>> detect(
    DOMString input,
    optional LanguageDetectorDetectOptions options = {}
  );

  readonly attribute FrozenArray<DOMString>? expectedInputLanguages;

  Promise<double> measureInputUsage(
    DOMString input,
    optional LanguageDetectorDetectOptions options = {}
  );
  readonly attribute unrestricted double inputQuota;
};
LanguageDetector includes DestroyableModel;

dictionary LanguageDetectorCreateCoreOptions {
  sequence<DOMString> expectedInputLanguages;
};

dictionary LanguageDetectorCreateOptions : LanguageDetectorCreateCoreOptions {
  AbortSignal signal;
  CreateMonitorCallback monitor;
};

dictionary LanguageDetectorDetectOptions {
  AbortSignal signal;
};

dictionary LanguageDetectionResult {
  DOMString detectedLanguage;
  double confidence;
};
</xmp>

<h3 id="language-detector-creation">Creation</h3>

<div algorithm>
  The static <dfn method for="LanguageDetector">create(|options|)</dfn> method steps are:

  1. Return the result of [=creating an AI model object=] given |options|, "{{language-detector}}", [=validate and canonicalize language detector options=], [=compute language detector options availability=], [=download the language detector model=], [=initialize the language detector model=], and [=create the language detector object=].
</div>

<div algorithm>
  To <dfn>validate and canonicalize language detector options</dfn> given a {{LanguageDetectorCreateCoreOptions}} |options|, perform the following steps. They mutate |options| in place to canonicalize language tags, and throw an exception if any are invalid.

  1. [=Validate and canonicalize language tags=] given |options| and "{{LanguageDetectorCreateCoreOptions/expectedInputLanguages}}".
</div>

<div algorithm>
  To <dfn>download the language detector model</dfn>, given a {{LanguageDetectorCreateCoreOptions}} |options|:

  1. [=Assert=]: these steps are running [=in parallel=].

  1. Initiate the download process for everything the user agent needs to detect the languages of input text, including all the languages in |options|["{{LanguageDetectorCreateCoreOptions/expectedInputLanguages}}"].

     This could include both a base language detection model, and specific fine-tunings or other material to help with the languages identified in |options|["{{LanguageDetectorCreateCoreOptions/expectedInputLanguages}}"].

  1. If the download process cannot be started for any reason, then return false.

  1. Return true.
</div>

<div algorithm>
  To <dfn>initialize the language detector model</dfn>, given a {{LanguageDetectorCreateCoreOptions}} |options|:

  1. [=Assert=]: these steps are running [=in parallel=].

  1. Perform any necessary initialization operations for the AI model backing the user agent's capabilities for detecting the languages of input text.

     This could include loading the model into memory, or loading any fine-tunings necessary to support the languages identified in |options|["{{LanguageDetectorCreateCoreOptions/expectedInputLanguages}}"].

  1. If initialization failed for any reason, then return a [=DOMException error information=] whose [=DOMException error information/name=] is "{{OperationError}}" and whose [=DOMException error information/details=] contain appropriate detail.

  1. Return null.
</div>

<div algorithm>
  To <dfn>create the language detector object</dfn>, given a [=ECMAScript/realm=] |realm| and a {{LanguageDetectorCreateCoreOptions}} |options|:

  1. [=Assert=]: these steps are running on |realm|'s [=ECMAScript/surrounding agent=]'s [=agent/event loop=].

  1. Let |inputQuota| be the amount of input quota that is available to the user agent for future [=detect languages|language detection=] operations. (This value is [=implementation-defined=], and may be +∞ if there are no specific limits beyond, e.g., the user's memory, or the limits of JavaScript strings.)

  1. Return a new {{LanguageDetector}} object, created in |realm|, with

    <dl class="props">
      : [=LanguageDetector/expected input languages=]
      :: the result of [=creating a frozen array=] given |options|["{{LanguageDetectorCreateCoreOptions/expectedInputLanguages}}"] if it [=set/is empty|is not empty=]; otherwise null

      : [=LanguageDetector/input quota=]
      :: |inputQuota|
    </dl>
</div>

<h3 id="language-detector-availability">Availability</h3>

<div algorithm>
  The static <dfn method for="LanguageDetector">availability(|options|)</dfn> method steps are:

  1. Return the result of [=computing AI model availability=] given |options|, "{{language-detector}}", [=validate and canonicalize language detector options=], and [=compute language detector options availability=].
</div>

<div algorithm>
  To <dfn>compute language detector options availability</dfn> given a {{LanguageDetectorCreateCoreOptions}} |options|, perform the following steps. They return either an {{Availability}} value or null, and they mutate |options| in place to update language tags to their best-fit matches.

  1. [=Assert=]: this algorithm is running [=in parallel=].

  1. If there is some error attempting to determine what language detection capabilities the user agent [=model availability/can support=], which the user agent believes to be transient (such that re-querying could stop producing such an error), then return null.

  1. Let |partition| be the result of [=getting the language availabilities partition=] given the purpose of detecting text written in that language.

  1. Return the result of [=computing language availability=] given |options|["{{LanguageDetectorCreateCoreOptions/expectedInputLanguages}}"] and |partition|.
</div>

<h3 id="the-LanguageDetector-class">The {{LanguageDetector}} class</h3>

Every {{LanguageDetector}} has an <dfn for="LanguageDetector">expected input languages</dfn>, a <code>{{FrozenArray}}&lt;{{DOMString}}></code> or null, set during creation.

Every {{LanguageDetector}} has an <dfn for="LanguageDetector">input quota</dfn>, a number, set during creation.

<hr>

The <dfn attribute for="LanguageDetector">expectedInputLanguages</dfn> getter steps are to return [=this=]'s [=LanguageDetector/expected input languages=].

The <dfn attribute for="LanguageDetector">inputQuota</dfn> getter steps are to return [=this=]'s [=LanguageDetector/input quota=].

<hr>

<div algorithm>
  The <dfn method for="LanguageDetector">detect(|input|, |options|)</dfn> method steps are:

    1. Let |global| be [=this=]'s [=relevant global object=].

  1. [=Assert=]: |global| is a {{Window}} object.

  1. If |global|'s [=associated Document=] is not [=Document/fully active=], then return [=a promise rejected with=] an "{{InvalidStateError}}" {{DOMException}}.

  1. Let |signals| be « [=this=]'s [=DestroyableModel/destruction abort controller=]'s [=AbortController/signal=] ».

  1. If |options|["`signal`"] [=map/exists=], then [=set/append=] it to |signals|.

  1. Let |compositeSignal| be the result of [=creating a dependent abort signal=] given |signals| using {{AbortSignal}} and [=this=]'s [=relevant realm=].

  1. If |compositeSignal| is [=AbortSignal/aborted=], then return [=a promise rejected with=] |compositeSignal|'s [=AbortSignal/abort reason=].

  1. Let |promise| be [=a new promise=] created in [=this=]'s [=relevant realm=].

  1. Let |abortedDuringOperation| be false.

    <p class="note">This variable will be written to from the [=event loop=], but read from [=in parallel=].

  1. [=AbortSignal/add|Add the following abort steps=] to |compositeSignal|:

    1. Set |abortedDuringOperation| to true.

    1. [=Reject=] |promise| with |compositeSignal|'s [=AbortSignal/abort reason=].

  1. Let |inputQuota| be [=this=]'s [=LanguageDetector/input quota=].

  1. [=In parallel=]:

    1. Let |stopProducing| be the following steps:

      1. Return |abortedDuringOperation|.

    1. Let |result| be the result of [=detecting languages=] given |input|, |inputQuota|, and |stopProducing|.

    1. [=Queue a global task=] on the [=AI task source=] given |global| to perform the following steps:

      1. If |abortedDuringOperation| is true, then abort these steps.

      1. Otherwise, if |result| is an [=error information=], then [=reject=] |promise| with the result of [=converting error information into an exception object=] given |result|.

      1. Otherwise:

        1. [=Assert=]: |result| is a [=list=] of {{LanguageDetectionResult}} dictionaries. (It is not null, since in that case |abortedDuringOperation| would have been true.)

        1. [=Resolve=] |promise| with |result|.
</div>

<div algorithm>
  The <dfn method for="LanguageDetector">measureInputUsage(|input|, |options|)</dfn> method steps are:

  1. Let |measureUsage| be an algorithm step which takes argument |stopMeasuring|, and returns the result of [=measuring language detector input usage=] given |input| and |stopMeasuring|.

  1. Return the result of [=measuring AI model input usage=] given [=this=], |options|, and |measureUsage|.
</div>

<h3 id="language-detector-language-detection">Language detection</h3>

<h4 id="language-detector-algorithm">The algorithm</h4>

<div algorithm>
  To <dfn>detect languages</dfn> given a [=string=] |input|, a number |inputQuota|, and an algorithm |stopProducing| that takes no arguments and returns a boolean, perform the following steps. They will return either null, an [=error information=], or a [=list=] of {{LanguageDetectionResult}} dictionaries.

  1. [=Assert=]: this algorithm is running [=in parallel=].

  1. Let |requested| be the result of [=measuring language detector input usage=] given |input| and |stopProducing|.

  1. If |requested| is null or an [=error information=], then return |requested|.

  1. [=Assert=]: |requested| is a number.

  1. If |requested| is greater than |inputQuota|, then return a [=quota exceeded error information=] with a [=quota exceeded error information/requested=] of |requested| and a [=quota exceeded error information/quota=] of |inputQuota|.

    <p class="note">In reality, we expect that implementations will check the input usage against the quota as part of the same call into the model as the language detection itself. The steps are only separated in the specification for ease of understanding.

  1. Let |partition| be the result of [=getting the language availabilities partition=] given the purpose of detecting text written in that language.

  1. Let |currentlyAvailableLanguages| be |partition|["{{Availability/available}}"].

  1. In an [=implementation-defined=] manner, subject to the following guidelines, let |rawResult| and |unknown| be the result of detecting the languages of |input|.

    |rawResult| must be a [=map=] which has a [=map/key=] for each language in |currentlyAvailableLanguages|. The [=map/value=] for each such key must be a number between 0 and 1. This value must represent the implementation's confidence that |input| is written in that language.

    |unknown| must be a number between 0 and 1 that represents the implementation's confidence that |input| is not written in any of the languages in |currentlyAvailableLanguages|.

    The [=map/values=] of |rawResult|, plus |unknown|, must sum to 1. Each such value, or |unknown|, may be 0.

    If the implementation believes |input| to be written in multiple languages, then it should attempt to apportion the values of |rawResult| and |unknown| such that they are proportionate to the amount of |input| written in each detected language. The exact scheme for apportioning |input| is [=implementation-defined=].

    <div class="example" id="example-multilingual-input">
      <p>If |input| is "`tacosを食べる`", the implementation might split this into "`tacos`" and "`を食べる`", and then detect the languages of each separately. The first part might be detected as English with confidence 0.5 and Spanish with confidence 0.5, and the second part as Japanese with confidence 1. The resulting |rawResult| then might be «[ "`en`" → 0.25, "`es`" → 0.25, "`ja`" → 0.5 ]» (with |unknown| set to 0).

      <p>The decision to split this into two parts, instead of e.g. the three parts "`tacos`", "`を`", and "`食べる`", was an [=implementation-defined=] choice. Similarly, the decision to treat each part as contributing to "half" of the result, instead of e.g. weighting by number of [=code points=], was [=implementation-defined=].

      <p>(Realistically, we expect that implementations will split on larger chunks than this, as generally more than 4-5 [=code points=] are necessary for most language detection models.)
    </div>

    If |stopProducing| returns true at any point during this process, then return null.

    If an error occurred during language detection, then return an [=error information=] according to the guidance in [[#language-detector-errors]].

    The detection process must conform to the guidance given in [[WRITING-ASSISTANCE-APIS#privacy]] and [[WRITING-ASSISTANCE-APIS#security]], notably including (but not limited to) [[WRITING-ASSISTANCE-APIS#privacy-user-input]] and [[WRITING-ASSISTANCE-APIS#security-runtime]].

  1. [=map/Sort in descending order=] |rawResult| with a less than algorithm which given [=map/entries=] |a| and |b|, returns true if |a|'s [=map/value=] is less than |b|'s [=map/value=].

  1. Let |results| be an empty [=list=].

  1. Let |cumulativeConfidence| be 0.

  1. [=map/For each=] |key| → |value| of |rawResult|:

    1. If |value| is 0, then [=iteration/break=].

    1. If |value| is less than |unknown|, then [=iteration/break=].

    1. [=list/Append=] «[ "{{LanguageDetectionResult/detectedLanguage}}" → |key|, "{{LanguageDetectionResult/confidence}}" → |value| ]» to |results|.

    1. Set |cumulativeConfidence| to |cumulativeConfidence| + |value|.

    1. If |cumulativeConfidence| is greater than or equal to 0.99, then [=iteration/break=].

  1. [=Assert=]: 1 &minus; |cumulativeConfidence| is greater than or equal to |unknown|.

  1. [=Assert=]: If |results|'s [=list/size=] is greater than 0, then |results|[|results|'s [=list/size=] - 1]["{{LanguageDetectionResult/confidence}}"] is greater than or equal to |unknown|.

  1. [=list/Append=] «[ "{{LanguageDetectionResult/detectedLanguage}}" → "`und`", "{{LanguageDetectionResult/confidence}}" → |unknown| ]» to |results|.

  1. Return |results|.

  <p class="note" id="note-language-detection-post-processing">Languages which are less than 1% likely, or contribute to less than 1% of the text, are considered more likely to be noise and so not worth returning to the web developer. Similarly, if the implementation is less sure about a language than it is about the text not being in any of the languages it knows, that language is probably not worth returning to the web developer.

  <p class="advisement" id="warning-language-detection-sum">Because of such omitted low-probability results, the sum of all confidence values returned to the web developer could be less than 1.
</div>

<h4 id="language-detector-usage">Usage</h4>

<div algorithm>
  To <dfn>measure language detector input usage</dfn>, given a [=string=] |input| and an algorithm |stopMeasuring| that takes no arguments and returns a boolean, perform the following steps:

  1. [=Assert=]: this algorithm is running [=in parallel=].

  1. Let |inputToModel| be the [=implementation-defined=] string that would be sent to the underlying model in order to [=detect languages=] given |input|.

    <p class="note" id="note-input-to-translator-model">This might be just |input| itself, or it might include some sort of wrapper prompt to a language model.

    If during this process |stopMeasuring| starts returning true, then return null.

    If an error occurs during this process, then return an appropriate [=DOMException error information=] according to the guidance in [[#language-detector-errors]].

  1. Return the amount of input usage needed to represent |inputToModel| when given to the underlying model. The exact calculation procedure is [=implementation-defined=], subject to the following constraints.

    The returned input usage must be nonnegative and finite. It must be 0, if there are no usage quotas for the translation process (i.e., if the [=LanguageDetector/input quota=] is +∞). Otherwise, it must be positive and should be roughly proportional to the [=string/length=] of |inputToModel|.

    <p class="note" id="note-language-detector-input-usage">This might be the number of tokens needed to represent |input| in a <a href="https://arxiv.org/abs/2404.08335">language model tokenization scheme</a>, or it might be |input|'s [=string/length=]. It could also be some variation of these which also counts the usage of any prefixes or suffixes necessary to give to the model.

    If during this process |stopMeasuring| starts returning true, then instead return null.

    If an error occurs during this process, then instead return an appropriate [=DOMException error information=] according to the guidance in [[#language-detector-errors]].
</div>

<h4 id="language-detector-errors">Errors</h4>

When language detection fails, the following possible reasons may be surfaced to the web developer. This table lists the possible {{DOMException}} [=DOMException/names=] and the cases in which an implementation should use them:

<table class="data">
  <thead>
    <tr>
      <th>{{DOMException}} [=DOMException/name=]
      <th>Scenarios
  <tbody>
    <tr>
      <td>"{{NotAllowedError}}"
      <td>
        <p>Language detection is disabled by user choice or user agent policy.
    <tr>
      <td>"{{UnknownError}}"
      <td>
        <p>All other scenarios, including if the user agent believes it cannot detect and also meet the requirements given in [[WRITING-ASSISTANCE-APIS#privacy]] and [[WRITING-ASSISTANCE-APIS#security]]. Or, if the user agent would prefer not to disclose the failure reason.
</table>

<p class="note">This table does not give the complete list of exceptions that can be surfaced by the language detector API. It only contains those which can come from certain [=implementation-defined=] steps.

<h3 id="language-detector-permissions-policy">Permissions policy integration</h3>

Access to the language detector API is gated behind the [=policy-controlled feature=] "<dfn permission>language-detector</dfn>", which has a [=policy-controlled feature/default allowlist=] of <code>[=default allowlist/'self'=]</code>.

<h2 id="privacy">Privacy considerations</h2>

Please see [[WRITING-ASSISTANCE-APIS#privacy]] for a discussion of privacy considerations for the translator and language detector APIs. That text was written to apply to all APIs sharing the same infrastructure, as noted in [[#dependencies]].

<h2 id="security">Security considerations</h2>

Please see [[WRITING-ASSISTANCE-APIS#security]] for a discussion of security considerations for the translator and language detector APIs. That text was written to apply to all APIs sharing the same infrastructure, as noted in [[#dependencies]].