10

I just opened a web page in Google Chrome, and it says "This page is in Japanese, would you like to translate it?".

Asking for a translation would presumably send the contents to Google, but how is the language identified in the first place? Is this done locally, in the browser? Or does this also send the page to Google? If so, should I not be asked for permission first? The page itself has no markup to indicate the language, and it is an internal intranet page, so that I am not at all sure that Google should be having access to its content.

1
10

The Chrome browser can identify, or at least guess, the page language by looking at a number of on page factors:

This can be done locally without any further internet connection or reporting to Google.

Translation of the content would definitely send the page content to Google servers for translation.

6
  • I was once viewing an XML file in Chrome, and it told me that I was viewing a page in... some European language, Belgian maybe? This would suggest to me that some analysis of the text occurs, and went a bit wrong in my case. I can't see why English XML would have had encoding that would indicate Belgian. But yes, for something like Japanese the encoding would be a dead giveaway. Jun 29 '11 at 5:17
  • How is UTF-8 a giveaway for Japanese?
    – Thilo
    Jun 29 '11 at 6:10
  • @CamJackson I'm sure it was not suggesting Belgian... Maybe it was Dutch and chrome found the single letter 'ij'
    – Peter Smit
    Jun 29 '11 at 6:38
  • 1
    It's not entirely by page encoding. I have a blog (in English) that gets a decent amount of Russian comment spam. Often when I am viewing my spam bucket Chrome prompts me to translate the page. It's obviously (to me) examining the contents and being triggered with the "other language" content is over a certain percentage threshold.
    – ale
    Jun 29 '11 at 13:05
  • 1
    @Thilo, it’s not UTF-8 that shows that it’s Japanese, but if a large percentage of the characters on a page are from the Japanese range of Unicode, then you’ve got your answer (automated language-detection is always guess work). Failing the obvious method of checking the character-encoding, I don’t think that it does (or at least needs to) send the page to a server for detection. Have you seen the size of chrome.dll recently? It’s huge! I haven’t looked through the (massive) code recently, but no doubt there is a function or two built-in for language-detection (it’s not that hard).
    – Synetech
    Jun 30 '11 at 1:24
4

The function is called DeterminePageLanguage. It's in the file components/translate/core/language_detection/language_detection_util.cc

Chrome first checks the HTML lang attribute and if it's not present it checks the Content-Language HTTP header. Then it gets a prediction from cld3.

The Compact Language Detector v3 (or CLD3) is a neural network model for language identification. The README states:

The inference code extracts character ngrams from the input text and computes the fraction of times each of them appears. For example, as shown in the figure below, if the input text is "banana", then one of the extracted trigrams is "ana" and the corresponding fraction is 2/4. The ngrams are hashed down to an id within a small range, and each id is represented by a dense embedding vector estimated during training.

The model averages the embeddings corresponding to each ngram type according to the fractions, and the averaged embeddings are concatenated to produce the embedding layer.

So essentially, they downloaded copies of a bunch of websites and paid someone to look at the text on those websites and say what language they're written in. Next they split the text into n-grams (groups of n letters) and so on and used a neural network to learn a mapping between n-gram distributions and languages.

So now they have 2 variables:

  • language which is set from either the HTML or the header (recall that the HTML attribute takes precedence if both are present)
  • cld_language which is a prediction based on the frequencies of groups of letters on the page

Then we hit this series of if-statements (I've edited out the part where they send analytics data about language mismatches)

  if (language.empty()) {
    return cld_language;
  }

  if (cld_language == kUnknownLanguageCode) {
    return language;
  }

  if (CanCLDComplementSubCode(language, cld_language)) {
    return cld_language;
  }

  if (IsSameOrSimilarLanguages(language, cld_language)) {
    return language;
  }

  if (MaybeServerWrongConfiguration(language, cld_language)) {
    return cld_language;
  }

  // Content-Language value might be wrong because CLD says that this page is
  // written in another language with confidence. In this case, Chrome doesn't
  // rely on any of the language codes, and gives up suggesting a translation.
  return kUnknownLanguageCode;

CLD3 is small and is run locally. In fact, it's open source and they distribute a pre-trained model (although the code for training the model and the data they used isn't available). You can use it in your projects.

There's even Python bindings (unofficial and unmaintained) for the original C++ code (you'll need to install Cython)

pip install cld3

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.