Interscript

Evaluating existing transliteration technologies for Farsi, we found the alternative solutions doing little than just basic mappings, thus not delivering a quality solution.

In order to improve this, if no good quality transliteration data is available, there was no alternative to write our own transliteration code.

This implies not only building a highly complex solution, modelling mappings, solving multiple special cases and accounting for languages complexity, but also porting the solution into production, away many scientific and NLP tools necessary for an advanced design.

On the top of that, we have also experienced that linguist and developer’s had quite different focus and understanding, which is not always easy to integrate, thus creating misunderstandings and efficiency losses.

We decided to solve this challenge with an innovative strategy that it is aggressively data based:

This has been successfully applied to Farsi and we think the workflow portable to any other languages applying 1. + 2. where no large transliteration corpus is available for just 2..

Below, we present the approach in more details before reviewing our scores and discussing it in the end of the document.

Transliteration is the process of transforming a language into another script, transforming the letters to maintain the sounds.

Building our initial solution, we faced following problems:

As a result, not only the code takes a considerable amount of time to be written, it has little flexibility for tweaks or redesigns and even debugging.

For that reason, we have considered an alternative approach consisting in:

Furthermore, as production is done in javascript from Ruby, key libraries do not exist and would need to be fully retro-engineered.

Fortunately, neural network can be used here to learn the necessary set of complex rules if generating large quality datasets with our code and datasets. For doing this, we could partially rely there on some of the team’s previous R&D work secryst.

In a nutshell:

Our linguist worked out an extensive, original solution to combine various mappings, rules and transformations (lemmatization, , stemming, PoS tagging, …​)

Better than many description, the full architecture is available online with a diagram: transliteration strategy.

Experimenting with various approaches for this project, we decided to work with lucidchart. However, any technology allowing to create workflows could be an alternative and massaged.

Once exported into a csv format, the diagrams can be computerized and modelled as trees, associating rules to each node.

Among the problems that had to be solved:

We are using a functional approach with the data "states" getting transformed as they "flow" within the graphical, logical structures.

Details can be found in the original Julia code.

Nodes are associated to an action to be performed that is described in plain English. For instance, in the diagram example:

Our code can be found with more explanations here: learn-graph code.

The current code is based on a design with over 150 nodes, dealing with main and edge cases. This represents a very complex solution.

While the developer’s job consists in the implementation of the above commands (searches in tables, comparing and concatenating strings, …​), the linguist can produce various nodes with commands and organize/re-organize them on the graphical editor.

In more details:

Below, we show the code output in full verbose, debug mode. The linguist can track the computation steps and help to identify bugs and inaccuracies.

We also provide with a test to assess the strategy overall scores. The mismatches are outputted into a local directory for further analysis, for instance with the above mode.

Transformers are a modern neural network architecture (attention is all you need) used on transduction problems such as language modeling and translation. They can be naturally applied to the problem of learning to transliterate.

Various libraries can be found online. We also experimented with multiple approaches, characters or words-based. The current method implemented in production is the latter.

Several resources are available online to explain transformers.

As for other projects, after training, ONNX was used to port trained neural networks onto a universal format.

This work (training+ ONNX export) can be found in our python scripts.

In production, we found that various components (neural networks) of the transformers had to be exported, such as generator, tokenizers, encoder, and decoder.

They had then to be combined correctly in our native ruby code.

Our codes can be tested/benchmarked with a test data set that we have designed. We are reporting simple metrics ACCU as (word accuracy %):

It has been optimized on our test set and after quite some work, could reach a very decent score. However, the code does not not cover/fails with many sentence (>50%), which is bad for training nnets because of the patterns loss.

Exactly this issue motivated the rewriting of a new version of CODE 0.9, which in turn, because of its difficulty/challenge, led to the alternative graphical approach featured in this blog.

Even though the code was built just overtaking an expert’s diagram and despite the considerable amount of data passed around, we found CODE D ~2 times faster than the initial implementation.

One reason is Julia, which can be several times faster than python, but probably also the non-optimal implementation in CODE 0.9.

We find this interesting to report as none of the many possible performance improvements were needed to transliterate ~20 sentences / sec. on a small machine, which is fast enough to produce huge amount of transliteration data overnight.