.. role:: html(code)
   :language: html

.. role:: js(code)
   :language: javascript

=============================
Fluent for Firefox Developers
=============================


This tutorial is intended for Firefox engineers already familiar with the previous
localization systems offered by Gecko - `DTD`_ and  `StringBundle`_ - and assumes
prior experience with those systems.


Using Fluent in Gecko
=====================

`Fluent`_ is a modern localization system currently being progressively introduced into
the Gecko platform with a focus on quality, performance, maintenance and completeness.

In order to ensure that Fluent is ready for engineers to work with, the initial
migrations are performed manually with a lot of oversight from the involved
stakeholders.

In this initial phase, `Firefox Preferences`_ is being migrated as the first target
and, as a result, the first bindings to be stabilized are for chrome-privileged
XUL context.

From there we plan to focus on two areas:

 - `Unprivileged Contexts`_
 - `System Add-ons`_

The end goal is replacing all uses of DTD and StringBundle within Firefox's codebase.

If you want to use Fluent, and your code involves one of the areas currently unsupported,
we'd like to work with you on getting Fluent ready for your code.


Getting a Review
----------------

If you end up working on any patch which touches FTL files, we have a temporary
hook in place that will reject your patch unless you get an r+ from one of the following
L10n Drivers:

  - Francesco Lodolo (:flod)
  - Zibi Braniecki (:gandalf)
  - Axel Hecht (:pike)
  - Staś Małolepszy (:stas)


Major Benefits
==============

Not only was the previous system designed over 20 years ago using file formats
never intended for localization, but also the Web stack which Fluent ties into has
completely changed over the same period, and the domain of internationalization
got a powerful foundation in the form of `Unicode`_, `CLDR`_ and `ICU`_ which Fluent tightly
`interoperates with`__.

__ https://github.com/projectfluent/fluent/wiki/Fluent-and-Standards

While it is beyond the scope of this document to cover all the benefits of Fluent in detail,
below is an attempt to select some most observable changes for each group of consumers.


Developers
----------

 - Support for XUL, XHTML, HTML, Web Components, React, JS, Python and Rust
 - Strings are available in a single, unified localization context available for both DOM and runtime code
 - Full internationalization (i18n) support: date and time formatting, number formatting, plurals, genders etc.
 - Strong focus on `declarative API via DOM attributes`__
 - Extensible with custom formatters, Mozilla-specific APIs etc.
 - `Separation of concerns`__: localization details, and the added complexity of some languages, don't leak onto the source code and are no concern for developers
 - Compound messages link a single translation unit to a single UI element
 - `DOM Overlays`__ allow for localization of DOM fragments
 - Simplified build system model
 - No need for pre-processing instructions
 - Support for pseudolocalization

__ https://github.com/projectfluent/fluent/wiki/Get-Started
__ https://github.com/projectfluent/fluent/wiki/Design-Principles
__ https://github.com/projectfluent/fluent.js/wiki/DOM-Overlays


Product Quality
------------------

 - A robust, multilevel, `error fallback system`__ prevents XML errors and runtime errors
 - Simplified l10n API reduces the amount of l10n specific code and resulting bugs
 - Runtime localization allows for dynamic language changes and updates over-the-air
 - DOM Overlays increase localization security

Many other smaller improvements will be noticed by the users of the system over time
and, with the new foundation, the Fluent team is `currently working`__ on multiple highly
requested features which will further improve the experience of developing
localizable UIs.

__ https://github.com/projectfluent/fluent/wiki/Error-Handling
__ https://github.com/projectfluent/fluent/wiki/Roadmap


Fluent Translation List - FTL
=============================

Fluent introduces a new localization format designed specifically for easy readability
and localization features offered by the system.

At first glance the format resembles `.properties` file. It may look like this:

.. code-block:: properties

  home-page-header = Home Page

  # The label of a button opening a new tab
  new-tab-open = Open New Tab

But the FTL file format is significantly more powerful and the additional features
quickly add up. In order to familiarize yourself with the basic features,
consider reading through the `Fluent Syntax Guide`_ to understand
a more complex example like:

.. code-block:: properties

  ### These messages correspond to security and privacy user interface.
  ###
  ### Please, choose simple and non-threatening language when localizing
  ### to help user feel in control when interacting with the UI.

  ## General Section

  -brand-short-name = Firefox
      .gender = masculine
  
  pref-pane =
      .title =
          { PLATFORM() ->
              [windows] Options
             *[other] Preferences
          }
      .accesskey = C
  
  # Variables:
  #   $tabCount (Number) - number of container tabs to be closed
  containers-disable-alert-ok-button =
      { $tabCount ->
          [one] Close { $tabCount } Container Tab
         *[other] Close { $tabCount } Container Tabs
      }
  
  update-application-info =
      You are using { -brand-short-name } Version: { $version }.
      <span>Please, read the <a>privacy policy</a>.</span>

The above, of course, is a particular selection of complex strings intended to exemplify
the new features and concepts introduced by Fluent.

In order to ensure the quality of the output, a lot of new checks and tooling
has been added to the build system.
`Pontoon`_, the main localization tool used to translate Firefox, has been rebuilding
its user experience to support localizers in their work.


.. _fluent-tutorial-social-contract:

Social Contract
===============

Fluent uses the concept of a `social contract` between developer and localizers.
This contract is established by the selection of a unique identifier, called :js:`l10n-id`,
which carries a promise of being used in a particular place to carry a particular meaning.

The use of unique identifiers is not new for Firefox engineers, but it is important
to recognize that Fluent formalizes this relationship.

.. important::

  An important part of the contract is that the developer commits to treat the
  localization output as `opaque`. That means that no concatenations, replacements
  or splitting should happen after the translation is completed to generate the
  desired output.

In return, localizers enter the social contract by promising to provide an accurate
and clean translation of the messages that match the request.

In previous localization systems, developers were responsible for differentiating
string variants based on a platform via pre-processing instructions, or
selecting which strings should be formatted using `PluralForms.jsm`.

In Fluent, the developer is not to be bothered with inner logic and complexity that the
localization will use to construct the response. Whether `declensions`__ or other
variant selection techniques are used is up to a localizer and their particular translation.
From the developer perspective, Fluent returns a final string to be presented to
the user, with no l10n logic required in the running code.

__ https://en.wikipedia.org/wiki/Declension


Markup Localization
===================

Fluent fully replaces the use of `DTD`_ in localization.

To localize an element in Fluent, the developer adds a new message to
an FTL file and then has to associate an :js:`l10n-id` with the element
by defining a :js:`data-l10n-id` attribute:

.. code-block:: html

  <h1 data-l10n-id="home-page-header" />

  <button data-l10n-id="pref-pane" />

Fluent will take care of the rest, populating the element with the message value
in its content and all localizable attributes if defined.

The difference compared to the use of DTD is that the developer provides only a single
message to localize the whole element, rather than a separate entity for
the value and each of the attributes.

The other change is that the developer can localize a whole fragment of DOM:

.. code-block:: html

  <p data-l10n-id="update-application-info" data-l10n-args="{'version': '60.0'}">
    <span class="bold">
      <a href="http://www.mozilla.org/privacy" />
    </span>
  </p>

.. code-block:: properties

  -brand-short-name = Firefox
  update-application-info =
      You are using { -brand-short-name } Version: { $version }.
      <span>Please, read the <a>privacy policy</a>.</span>


Fluent will overlay the translation onto the source fragment preserving attributes like
:code:`class` and :code:`href` from the source and adding translations for the elements
inside. The resulting localized content will look like this:

.. code-block:: html

  <p data-l10n-id="update-application-info" data-l10n-args="{'version': '60.0'}">
    You are using Firefox Version: 60.0.
    <span class="bold">
      Please, read the <a href="http://www.mozilla.org/privacy">privacy policy</a>.
    </span>
  </p>


This operation is sanitized, and Fluent takes care of selecting which elements and
attributes can be safely provided by the localization.
The list of allowed elements and attributes is `maintained by the W3C`__, and if
the developer needs to allow for localization of additional attributes, they can
whitelist them using :code:`data-l10n-attrs` list:

.. code-block:: html

  <label data-l10n-id="search-input" data-l10n-attrs="style" />

The above example adds an attribute :code:`style` to be allowed on this
particular :code:`label` element.


External Arguments
------------------

Notice in the previous example the attribute :code:`data-l10n-args`, which is
a JSON object storing variables exposed by the developer to the localizer.

This is the main channel for the developer to provide additional variables
to be used in the localization.

Arguments are rarely needed for situations where it’s currently possible to use
DTD, since such variables would need to be computed from the code at runtime.
It's worth noting that, when the :code:`l10n-args` are set in
the runtime code, they are in fact encoded as JSON and stored together with
:code:`l10n-id` as an attribute of the element.

__ https://www.w3.org/TR/2011/WD-html5-20110525/text-level-semantics.html


Runtime Localization
====================

Fluent fully replaces the use of `StringBundle`_ in localization.

In almost every case the JS runtime code will operate on a particular document, either
XUL, XHTML or HTML.

If the document has its markup already localized, then Fluent exposes a new
attribute on the :js:`document` element - :js:`document.l10n`.

This property is an object of type :js:`DOMLocalization` which maintains the main
localization context for this document and exposes it to runtime code as well.

With a focus on `declarative localization`__, the primary method of localization is
to alter the localization attributes in the DOM. Fluent provides a method to facilitate this:

.. code-block:: javascript

  document.l10n.setAttributes(element, "new-panel-header");

This will set the :code:`data-l10n-id` on the element and translate it before the next
animation frame.

The reason to use this API over manually setting the attribute is that it also
facilitates encoding l10n arguments as JSON:

.. code-block:: javascript

  document.l10n.setAttributes(element "containers-disable-alert-ok-button", {
    tabCount: 5
  }

__ https://github.com/projectfluent/fluent/wiki/Good-Practices-for-Developers


Non-Markup Localization
-----------------------

In rare cases, when the runtime code needs to retrieve the translation and not
apply it onto the DOM, Fluent provides an API to retrieve it:

.. code-block:: javascript

  let [ msg ] = await document.l10n.formatValues([
    {id: "remove-containers-description"}
  ]);

  alert(msg);

This model is heavily discouraged and should be used only in cases where the
DOM annotation is not possible.

.. note::

  This API is currently only available as asynchronous. In case of Firefox,
  the only non-DOM localizable calls are used where the output goes to
  a third-party like Bluetooth, Notifications etc.
  All those cases should already be asynchronous.


Internationalization
====================

The majority of internationalization issues are implicitly handled by Fluent without
any additional requirement. Full Unicode support, `bidirectionality`__, and
correct number formatting work without any action required from either
developer or localizer.

__ https://github.com/projectfluent/fluent/wiki/BiDi-in-Fluent

.. code-block:: javascript

  document.l10n.setAttributes(element, "welcome-message", {
    userName: "اليسع",
    count: 5
  });

A message like this localized to American English will correctly wrap the user
name in directionality marks, allowing the layout engine to determine how to
display the bidirectional text.

On the other hand, the same message localized to Arabic will use the Eastern Arabic
numeral for number "5".


Plural Rules
------------

The most common localization feature is the ability to provide different variants
of the same string depending on plural categories.

Fluent replaces the use of the proprietary :code:`PluralForms.jsm` with a Unicode CLDR
standard called `Plural Rules`_.

In order to allow localizers to use it, all the developer has to do is to pass
an external argument number:

.. code-block:: javascript

  document.l10n.setAttributes(element, "unread-warning", { unreadCount: 5 });

Localizers can use the argument to build a multi variant message if their
language requires that:

.. code-block:: properties

  unread-warning =
      { $unreadCount ->
          [one] You have { $unreadCount } unread message
         *[other] You have { $unreadCount } unread messages
      }

Fluent guesses that, since the variant selection is performed based on a number,
its `plural category`__ should be retrieved.

If the given translation doesn't need pluralization for the string (for example
Japanese often will not), the localizer can replace it with:

.. code-block:: properties

  unread-warning = You have { $unreadCount } unread messages

and the message will preserve the social contract.

One additional feature is that the localizer can further improve the message by
specifying variants for particular values:

.. code-block:: properties

  unread-warning =
      { $unreadCount ->
          [0] You have no unread messages
          [1] You have one unread message
         *[other] You have { $unreadCount } unread messages
      }

The advantage here is that per-locale choices don't leak onto the source code
and the developer is not affected.


.. note::

  There is an important distinction between a variant keyed on plural category
  `one` and digit `1`. Although in English the two are synonymous, in other
  languages category `one` may be used for other numbers.
  For example in `Bosnian`__, category `one` is used for numbers like `1`, `21`, `31`
  and so on, and also for fractional numbers like `0.1`.

__ https://unicode.org/cldr/charts/latest/supplemental/language_plural_rules.html
__ https://unicode.org/cldr/charts/latest/supplemental/language_plural_rules.html#bs

Partial Arguments
-----------------

When it comes to formatting data, Fluent allows the developer to provide
a set of parameters for the formatter, and the localizer can fine tune some of them.
This technique is called `partial arguments`__.

For example, when formatting a date, the developer can just pass a JS :js:`Date` object,
but its default formatting will be pretty expressive. In most cases, the developer
may want to use some of the :js:`Intl.DateTimeFormat` options to select the default
representation of the date in string:

.. code-block:: javascript

  document.l10n.setAttributes(element, "welcome-message", {
  startDate: FluentDateTime(new Date(), {
      year: "numeric",
      month: "long",
      day: "numeric"
    })
  });

.. code-block:: properties

  welcome-message = Your session will start date: { $startDate }

In most cases, that will be enough and the date would get formatted in the current
Firefox as `February 28, 2018`.

But if in some other locale the string would get too long, the localizer can fine
tune the options as well:

.. code-block:: properties

  welcome-message = Początek Twojej sesji: { DATETIME($startDate, month: "short") }

This will adjust the length of the month token in the message to short and get formatted
in Polish as `28 lut 2018`.

At the moment Fluent supports two formatters that match JS Intl API counterparts:

 * **NUMBER**: `Intl.NumberFormat`__
 * **DATETIME**: `Intl.DateTimeFormat`__

With time more formatters will be added.

__ http://projectfluent.org/fluent/guide/functions.html#partial-arguments
__ https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/NumberFormat
__ https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat

Registering New L10n Files
==========================

In the previous system, a new localization file had to be registered in order to
add it in the `jar.mn` file for packaging.

Fluent uses a wildcard statement, packaging all localization resources into
their component's `/localization/` directory.

That means that, if a new file is added to a component of Firefox already
covered by Fluent like `browser`, it's enough to add the new file to the
repository in a path like `browser/locales/en-US/browser/component/file.ftl`, and
the toolchain will package it into `browser/localization/browser/component/file.ftl`.

At runtime Firefox uses a special registry for all localization data. It will
register the browser's `/localization/` directory and make all files inside it
available to be referenced.

To make the document localized using Fluent, all the developer has to do is add
a single polyfill for the Fluent API to the source and list the resources
that will be used:

.. code-block:: html

  <link rel="localization" href="branding/brand.ftl"/>
  <link rel="localization" href="browser/preferences/preferences.ftl"/>
  <script src="chrome://global/content/l10n.js"></script>

For performance reasons the :html:`<link/>` elements have to be specified above the
:html:`<script/>`, and the :html:`<script/>` itself has to be synchronous in order to ensure
that the localization happens before first paint.

This allows Fluent to trigger asynchronous resource loading early enough to
perform the initial DOM translation before the initial layout.

The URI provided to the :html:`<link/>` element are relative paths within the localization
system.

Notice that only the registration of the script is synchronous. All the I/O and
translation happen asynchronously.


Custom Contexts
===============

The above method creates a single localization context per document.
In almost all scenarios that's sufficient.

In rare edge cases where the developer needs to fetch additional resources, or
the same resources in another language, it is possible to create additional
contexts manually using the `Localization` class:

.. code-block:: javascript

  const { Localization } =
    ChromeUtils.import("resource://gre/modules/Localization.jsm", {});
  
  
  const myL10n = new Localization([
    "branding/brand.ftl",
    "browser/preferences/preferences.ftl"
  ]);
  
  
  let [isDefaultMsg, isNotDefaultMsg] =
    myL10n.formatValues({id: "is-default"}, {id: "is-not-default"});


.. admonition:: Example

  An example of a use case is the Preferences UI in Firefox, which uses the
  main context to localize the UI but also to build a search index.

  It is common to build such search index both in a current language and additionally
  in English, since a lot of documentation and online help exist only in English.

  A developer may create manually a new context with the same resources as the main one,
  but hardcode it to `en-US` and then build the search index using both contexts.

Designing Localizable APIs
==========================

When designing localizable APIs, the most important rule is to resolve localization as
late as possible. That means that instead of resolving strings somewhere deep in the
codebase and then passing them on, or even caching, it is highly recommended to pass
around :code:`l10n-id` or :code:`[l10n-id, l10n-args]` pairs until the top-most code
resolves them or applies them onto the DOM element.


Testing
=======

When writing tests that involve both I18n and L10n, the general rule is that
result strings are opaque. That means that the developer should not assume any particular
value and should never test against it.

In case of raw i18n the :js:`resolvedOptions` method on all :js:`Intl.*` formatters
makes it relatively easy. In case of localization, the recommended way is to test that
the code sets the right :code:`l10n-id`/:code:`l10n-args` attributes like this:

.. code-block:: javascript
  
  testedFunction();
  
  const l10nAttrs = document.l10n.getAttributes(element);
  
  deepEquals(l10nAttrs, {
    id: "my-expected-id",
    args: {
      unreadCount: 5
    }
  });

If the code really has to test for particular values in the localized UI, it is
always better to scan for a variable:

.. code-block:: javascript

  testedFunction();
  
  equals(element.textContent.contains("John"));

.. important::

  Testing against whole values is brittle and will break when we insert Unicode
  bidirectionality marks into the result string or adapt the output in other ways.


Pseudolocalization
==================

When working with a Fluent-backed UI, the developer gets a new tool to test their UI
against several classes of problems.

Pseudolocalization is a mechanism which transforms messages on the fly, using
specific logic to help emulate how the UI will look once it gets localized.

The three classes of potential problems that this can help with are:

 - Hardcoded strings.

   Turning on pseudolocalization should expose any strings that were left
   hardcoded in the source, since they won't get transformed.


 - UI space not adapting to longer text.

   Many languages use longer strings than English. For example, German strings
   may be 30% longer (or more). Turning on pseudolocalization is a quick way to
   test how the layout handles such locales.


 - Bidi adaptation.

   For many developers, testing the UI in right-to-left mode is hard. Mozilla
   offers a pref :js:`intl.uidirection` which switches the direction of the layout,
   but that doesn't expose problems related to right-to-left text.
   Pseudolocalization shows how a right-to-left locale will look like.

To turn on pseudolocalization, add a new string pref :js:`intl.l10n.pseudo` and
select the strategy to be used:

 - :js:`accented` - Ȧȧƈƈḗḗƞŧḗḗḓ Ḗḗƞɠŀīīşħ

   This strategy replaces all Latin characters with their accented equivalents,
   and duplicates some vowels to create roughly 30% longer strings.


 - :js:`bidi` - ɥsıʅƃuƎ ıpıԐ

   This strategy replaces all Latin characters with their 180 degree rotated versions
   and enforces right to left text flow using Unicode UAX#9 `Explicit Directional Embeddings`__.

__ https://www.unicode.org/reports/tr9/#Explicit_Directional_Embeddings

Inner Structure of Fluent
=========================

The inner structure of Fluent in Gecko is out of scope of this tutorial, but
since the class and file names may show up during debugging or profiling,
below is a list of major components, each with a corresponding file in `/intl/l10n`
modules in Gecko.


MessageContext
--------------

MessageContext is the lowest level API. It's fully synchronous, contains a parser for the
FTL file format and a resolver for the logic. It is not meant to be used by
consumers directly.

In the future we intend to offer this layer for standardization and it may become
part of the :js:`mozIntl.*` or even :js:`Intl.*` API sets.

That part of the codebase is also the first that we'll be looking to port to Rust.


Localization
------------

Localization is a higher level API which uses :js:`MessageContext` internally but
provides a full layer of compound message formatting and robust error fall-backing.

It is intended for use in runtime code and contains all fundamental localization
methods.


DOMLocalization
---------------

DOMLocalization extends :js:`Localization` with functionality to operate on HTML, XUL
and the DOM directly including DOM Overlays and Mutation Observers.


l10n.js
-------

l10n.js is a small runtime code which fetches the :html:`<link>` elements specified
in the document and initializes the main :js:`DOMLocalization` context
on :js:`document.l10n`.


L10nRegistry
------------

L10nRegistry is our resource management service. It replaces :js:`ChromeRegistry` and
maintains the state of resources packaged into the build and language packs,
providing an asynchronous iterator of :js:`MessageContext` objects for a given locale set
and resources that the :js:`Localization` class uses.


.. _Fluent: http://projectfluent.org/
.. _DTD: https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XUL/Tutorial/Localization
.. _StringBundle: https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XUL/Tutorial/Property_Files
.. _Firefox Preferences: https://bugzilla.mozilla.org/show_bug.cgi?id=1415730
.. _Unprivileged Contexts: https://bugzilla.mozilla.org/show_bug.cgi?id=1407418
.. _System Add-ons: https://bugzilla.mozilla.org/show_bug.cgi?id=1425104
.. _CLDR: http://cldr.unicode.org/
.. _ICU: http://site.icu-project.org/
.. _Unicode: https://www.unicode.org/
.. _Fluent Syntax Guide: http://projectfluent.org/fluent/guide/
.. _Pontoon: https://pontoon.mozilla.org/
.. _Plural Rules: http://cldr.unicode.org/index/cldr-spec/plural-rules
