Multilingual Sites

If sites publish in multiple languages and use a plugin to present a list of language versions, wpextract can parse this and add links between translated versions in the output dataset.

Extraction Process

Extracting multilingual data is performed during the extract command. This data isn't available in the WordPress REST API response, so instead must be obtained from scraped HTML.

Obtaining the scraped HTML is relatively straightforward, as we already have a list of all posts from the download command.

One way this could be scraped is using jq to parse the downloaded posts file and produce a URL list, then wget to download each page:

$ cat posts.json | jq -r '.[] | .link' > url_list.txt
$ touch rejected.log
$ wget --adjust-extension --input-file=url_list.txt \
     --wait 1 --random-wait --force-directories \
     --rejected-log=rejected.log

When running the extract command, pass this directory as the --scrape-root argument. The scrape will be crawled to match URLs to downloaded HTML files following this process.

Supported Plugins

wpextract uses an extensible system of parsers to find language picker elements and extract their data.

Currently the following plugins are supported:

Polylang

Plugin Page · Website

Supports:

• Adding as a widget (e.g. to a sidebar)

  Example

  <div id="polylang-2" class="widget widget_polylang">
    <ul>
      <li
        class="lang-item lang-item-18 lang-item-en current-lang lang-item-first"
      >
        <a
          hreflang="en-US"
          href="https://example.org/current-lang-page/"
          lang="en-US"
        >
          <img
            src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAALCAMAAABBPP0LAAAAmVBMVEViZsViZMJiYrf9gnL8eWrlYkjgYkjZYkj8/PujwPybvPz4+PetraBEgfo+fvo3efkydfkqcvj8Y2T8UlL8Q0P8MzP9k4Hz8/Lu7u4DdPj9/VrKysI9fPoDc/EAZ7z7IiLHYkjp6ekCcOTk5OIASbfY/v21takAJrT5Dg6sYkjc3Nn94t2RkYD+y8KeYkjs/v7l5fz0dF22YkjWvcOLAAAAgElEQVR4AR2KNULFQBgGZ5J13KGGKvc/Cw1uPe62eb9+Jr1EUBFHSgxxjP2Eca6AfUSfVlUfBvm1Ui1bqafctqMndNkXpb01h5TLx4b6TIXgwOCHfjv+/Pz+5vPRw7txGWT2h6yO0/GaYltIp5PT1dEpLNPL/SdWjYjAAZtvRPgHJX4Xio+DSrkAAAAASUVORK5CYII="
            alt="English"
            style="width: 16px; height: 11px"
            width="16"
            height="11"
          />
          <span style="margin-left: 0.3em">English</span>
        </a>
      </li>
      <li class="lang-item lang-item-20 lang-item-fr">
        <a
          hreflang="fr-FR"
          href="https://example.org/fr/translation-page/"
          lang="fr-FR"
        >
          <img
            src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAALCAMAAABBPP0LAAAAbFBMVEVzldTg4ODS0tLxDwDtAwDjAADD0uz39/fy8vL3k4nzgna4yOixwuXu7u7s6+zn5+fyd2rvcGPtZljYAABrjNCpvOHrWkxegsqfs93NAADpUUFRd8THAABBa7wnVbERRKa8vLyxsLCoqKigoKClCvcsAAAAXklEQVR4AS3JxUEAQQAEwZo13Mk/R9w5/7UERJCIGIgj5qfRJZEpPyNfCgJTjMR1eRRnJiExFJz5Mf1PokWr/UztIjRGQ3V486u0HO55m634U6dMcf0RNPfkVCTvKjO16xHA8miowAAAAABJRU5ErkJggg=="
            alt="Français"
            style="width: 16px; height: 11px"
            width="16"
            height="11"
          />
          <span style="margin-left: 0.3em">Français</span>
        </a>
      </li>
      <li class="lang-item lang-item-22 lang-item-de no-translation">
        <a hreflang="de-DE" href="https://example.org/de/" lang="de-DE">
          <img
            src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAALCAIAAAD5gJpuAAABLElEQVR4AY2QgUZEQRSGz9ydmzbYkBWABBJYABHEFhJ6m0WP0DMEQNIr9AKrN8ne2Tt3Zs7MOdOZmRBEv+v34Tvub9R6fdNlAzU+snSME/wdjbjbbJ6EiEg6BA8102QbjKNpoMzw8v6qD/sOALbbT2MC1NgaAWOKOgxf5czY+4dbAX2G/THzcozLrvPV85IQyqVz0rvg2p9Pei4HjzSsiFbV4JgyhhxCjpGdZ0RhdikLB9/b8Qig7MkpSovR7Cp59q6CazaNFiTt4J82o6uvdMVwTsztKTXZod4jgOJJuqNAjFyGrBR8gM6XwKfIC4KanBSTZ0rClKh08D9DFh3egW7ebH7NcRDQWrz9rM2Ne+mDOXB2mZJ8agL19nwxR2iZXGm1gDbQKhDjd4yHb2oW/KR8xHicAAAAAElFTkSuQmCC"
            alt="Deutsch"
            style="width: 16px; height: 11px"
            width="16"
            height="11"
          />
          <span style="margin-left: 0.3em">Deutsch</span>
        </a>
      </li>
      <li class="lang-item lang-item-24 lang-item-es no-translation">
        <a hreflang="es-ES" href="https://example.org/es/" lang="es-ES">
          <img
            src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAALCAMAAABBPP0LAAAAflBMVEX/AAD9AAD3AADxAADrAAD/eXn9bGz8YWH8WVn6UVH5SEj5Pz/3NDT0Kir9/QD+/nL+/lT18lDt4Uf6+j/39zD39yf19R3n5wDxflXsZ1Pt4Y3x8zr0wbLs1NXz8xPj4wD37t3jmkvsUU/Bz6nrykm3vJ72IiL0FBTyDAvhAABEt4UZAAAAX0lEQVR4AQXBQUrFQBBAwXqTDkYE94Jb73+qfwVRcYxVQRBRToiUfoaVpGTrtdS9SO0Z9FR9lVy/g5c99+dKl30N5uxPuviexXEc9/msC7TOkd4kHu/Dlh4itCJ8AP4B0w4Qwmm7CFQAAAAASUVORK5CYII="
            alt="Español"
            style="width: 16px; height: 11px"
            width="16"
            height="11"
          />
          <span style="margin-left: 0.3em">Español</span>
        </a>
      </li>
      <li class="lang-item lang-item-26 lang-item-zh no-translation">
        <a hreflang="zh-CN" href="https://example.org/zh/" lang="zh-CN">
          <img
            src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAALCAMAAABBPP0LAAAAXVBMVEXUAADlQgDLAADBAADtgXn63Xjypnf1wHHpcG/oZmbmXVzlU1PjS0q1AAD981775VvwnVD2zkvhPz/fNzfdMjHcKyvaJyfsi0baISHYGhqqAADWExPTDQ2jAACfAAApGpDBAAAAWklEQVR4ATXIhQHDQBTDUMll2n/RMiU5/vQsAE4EsPbaKVOU+pXNwc/WKQXeDZMKu+psCXw/Z7efarmENd6GIwGpXhUvM4spxoiEbouRNT7Fmtaq+RG4wAqZZvceD8DeIelqAAAAAElFTkSuQmCC"
            alt="中文 (中国)"
            style="width: 16px; height: 11px"
            width="16"
            height="11"
          />
          <span style="margin-left: 0.3em">中文 (中国)</span>
        </a>
      </li>
      <li class="lang-item lang-item-41 lang-item-ar no-translation">
        <a hreflang="ar" href="https://example.org/ar/" lang="ar">
          <img
            src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAALCAMAAABBPP0LAAAANlBMVEUAYjMTYDs3R0AvV0NObzE3dSoTWzhAZjgyfEY0gl1EcDFqpIhKj28TVzaLs41ol1JSaF1JW1NzUHm9AAAAPUlEQVR4AY2MtQEAMAgE447tv2xKvuQqeEtRcikZ/9p6b9X/Mdfeaw4PnPvehQhNvpcnJYiInIqraqYpyAd1AAFxIEreLQAAAABJRU5ErkJggg=="
            alt="العربية"
            style="width: 16px; height: 11px"
            width="16"
            height="11"
          />
          <span style="margin-left: 0.3em">العربية</span>
        </a>
      </li>
    </ul>
  </div>
  

• Adding to the navbar as a custom dropdown¹

  Example

  <div class="header-lang_switcher switcher-ltr">
    <div class="current-lang-switcher">
      <img src="https://example.org/flag_en.svg" alt="flag-en" />
      <span>en</span>
    </div>
    <ul>
      <li class="lang-item lang-item-5 lang-item-fr lang-item-first">
        <a
          hreflang="fr-FR"
          href="https://example.org/fr/translation-page/"
          lang="fr-FR"
          >Français</a
        >
      </li>
      <li class="lang-item lang-item-7 lang-item-de no-translation">
        <a hreflang="de-DE" href="https://example.org/de/" lang="de-DE"
          >Deutsch</a
        >
      </li>
      <li class="lang-item lang-item-9 lang-item-es no-translation">
        <a hreflang="es-ES" href="https://example.org/es/" lang="es-ES"
          >Español</a
        >
      </li>
      <li class="lang-item lang-item-11 lang-item-it no-translation">
        <a hreflang="it-IT" href="https://example.org/it/" lang="it-IT"
          >Italiano</a
        >
      </li>
      <li class="lang-item lang-item-13 lang-item-zh no-translation">
        <a hreflang="zh-CN" href="https://example.org/zh/" lang="zh-CN"
          >中文 (中国)</a
        >
      </li>
      <li class="lang-item lang-item-15 lang-item-ar no-translation">
        <a hreflang="ar" href="https://example.org/ar/" lang="ar"
          >العربية</a
        >
      </li>
    </ul>
  </div>
  

Does not support:

• Methods which show the picker as a <select> element

Adding Support

Note

To use additional pickers, you must use WPextract as a library.

Support can be added by creating a new picker definition inheriting from LangPicker, and passing to the translation_pickers argument of WPExtractor

This parent class defines two abstract methods which must be implemented:

• LangPicker.get_root - returns the root element of the picker
• LangPicker.extract - find the languages, call LangPicker.set_current_lang and call LangPicker.add_translation for each

More complicated pickers may need to override additional methods of the class, but should still ultimately populate the LangPicker.translations and LangPicker.current_language attributes as the parent class does.

This section will show implementing a new picker with the following simplified markup:

Example picker markup

<ul class="translations">
  <li><a href="/page/" class="lang current-lang" lang="en">English</a></li>
  <li><a href="/de/seite/" class="lang" lang="de">Deutsch</a></li>
  <li><a href="/page/" class="lang no-translation" lang="fr">Français</a></li>
</ul>

The correct parse of this picker should set the current language to English, add German as a translation, and ignore French.

get_root()

Selector Support

The select() and select_one() methods use Soup Sieve internally.

This library supports many, but not all, CSS selectors. Supported selectors can be found here. Namespace selection is not supported as we use the lxml backend.

Using the self.page_doc attribute, a BeautifulSoup object representing the page, the root element of the picker should be found and returned.

The select_one method is used to find the root element, and will return None if no element is found, which will be interpreted as the picker not being present on the page.

If a value is returned, the self.root_el attribute will be populated with the result of this method.

Example get_root implementation

class MyPicker(LangPicker):
    ...
    def get_root(self):
        return self.page_doc.select_one('ul.translations')

extract()

Using the self.root_el attribute, the languages should be found and added to the dataset.

Be careful to avoid:

• Adding the current language as a translation
• Adding languages which are listed but don't have translations

Example extract implementation

class MyPicker(LangPicker):
    ...
    def extract(self):
        for lang_el in self.root_el.select('li'):
            lang_a = lang_el.select_one('a')
            if 'current-lang' in lang_a.get('class'):
                self.set_current_lang(lang_a.get('lang'))
            elif 'no-translation' not in lang_a.get('class'):
                self.add_translation(lang_a.get('href'), lang_a.get('lang'))

Parsing Robustness

BeautifulSoup's select and select_one methods will silently fail if no matching elements are found (returning None and an empty list respectively). In some cases this may be desirable, e.g. if the picker contains no languages, select_one returning an empty list will probably be the correct behaviour.

In cases where this isn't right, like when retrieving an element which should always be present, you can instead use the LangPicker._root_select() or LangPicker._root_select_one() methods. These will raise an error if no element(s) are found.

Currently, this error will be caught and the page in question will be skipped as if no translation picker could be found. In future, this may instead result in other pickers being tried instead.

Example more robust extract implementation

class MyPicker(LangPicker):
    ...
    def extract(self):
        # This should *always* be present, if it isn't this can't be the correct picker.
        current_lang_a = self._root_select_one('li a.current-lang')
        self.set_current_lang(current_lang_a.get('lang'))

        # This could be empty
        for lang_a in self.root_el.select('li a.lang'):
            if (
                'current-lang' not in lang_a.get('class')
                and 'no-translation' not in lang_a.get('class')
            ):
                self.add_translation(lang_a.get('href'), lang_a.get('lang'))

Using other selector methods

BeautifulSoup's CSS selector methods should cover most use cases. However, if you need to use more complex selection logic, you can generate the same error by using the LangPicker._build_extraction_fail_err method.

Performance

For each post in the site, the get_root() method of every LangPicker instance selected will be run, so the efficiency of this method is important.

• Try and make the test as specific as possible
• For more complex tests, consider splitting it up and failing early
• Minimise usage of dynamic patterns. SoupSieve internally caches the compiled form of each search pattern to improve performance, but any change will invalidate this. Consider splitting the static and dynamic parts of patterns to work around this.

Contributing Pickers

We welcome contributions via a GitHub PR so long as the picker is not overly specific to a single site.

---

1. This implementation may be overly customised to the site it was added to collect. ↩