Ir al contenido principal

Flatpak docs y más

flatpak-docs.png

La semana pasada traduje la documentación de Flatpak a español. Ya está disponible acá http://flatpak.readthedocs.io/es/latest/ .

Espero que sirva para promocionar la adopción de Flatpak. Si vas a seguir el tutorial traducido para empaquetar una app, me gustaría recibir tus comentarios.

También fue una buena oportunidad para refrescar los detalles.


Después de empaquetar GeoGebra, tuve una idea más clara de cómo funciona --extra-data. Esta opción suele usarse para enlazar datos externos que no se van a agrupar con el paquete, ya sea por licenciamiento u otros motivos (en el caso de GeoGebra la licencia no lo permite). Me pareció raro que la misma extra-data se tuviera que bajar cada vez que corría mis tests, incluso usando un repositorio local.

Entonces sugerí en IRC un directorio de cacheo para extra-data, en el espacio de usuario. Mirando los fuentes de flatpak encontré una línea TODO sobre almacenar datos parciales y resumir las descargas. Esta mejora va en la misma línea de lo que yo quería. Así que propuse el directorio de cacheo como un primer paso para esto https://github.com/flatpak/flatpak/pull/873 .

Building Electron apps offline for Flathub

electron-flathub.png

Electron is one way to make desktop applications using web technologies HTML/CSS/Javascript.

Flatpak is the new cool thing in Linux desktops. A technology that improves security, development workflow, and overcomes library differences between Linux distros by using versioned runtimes.

Doing a flatpak from an Electron app has already been addressed by Endless OS. They ship Slack for example, one of the most popular Electron apps. Internally, the flatpak bundles an entire web browser (Chromium) along with the application, so the final package is big.

A flatpak is usually built automatically with flatpak-builder. This command takes a single text file, the flatpak manifest, that has all the information to bring up a flatpak. flatpak-builder basically does these steps:

  1. Download sources
  2. Build and install modules
  3. Cleanup and finish application

flathub.org is an initiative to have a central hub of flatpaks. I'm very excited about flathub.org because it has the potential to make Linux users lives much easier.

flathub.org has an automatic build system. Each time an application updates the flatpak manifest, it is automatically rebuilt.

The problem with Electron apps (and Node.js projects in general) is that usually they download many dependencies at the build/install step. And flathub.org requires that the build process (step 2 above) doesn't access the network. It only allows network usage in the Download sources step. To me this restriction is acceptable. It is important to have the same result if the downloaded sources don't change. Builds have to be deterministic.

If we want to have Electron apps in flathub.org like Atom or Slack, we have to work around this: offline installation of Electron.

Offline Node.js install

The issue described above is no news in the Node.js world. More specifically is an issue in Node's most common package manager, npm. If the package manager can't warrant determinism, different developers may be developing under different environments. This is also a problem for continuous integration tools. And this is why Facebook developed their own package manager as a replacement for npm, Yarn. Also, this is the reason of important changes in latest npm release, v5 which has seen light a few days ago.

I set my final goal to: have one source entry per Node dependency in the flatpak manifest. The tarballs for each Node package are available either in https://registry.npmjs.org for npm and https://registry.yarnpkg.com for Yarn.

So I decided to take a look at both, Yarn and npm v5. This blog post was a good start to understand their different takes at determinism. Both are similar in that they translate the package.json with the project requirements to a yarn.lock (Yarn) or package-lock.json (npm v5). This lock file has a list of each dependency, a flattened version of the tree of dependencies. For example here is the head of my yarn.lock:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1

"@types/node@^7.0.18":
  version "7.0.29"
  resolved "https://registry.yarnpkg.com/@types/node/-/node-7.0.29.tgz#ccfcec5b7135c7caf6c4ffb8c7f33102340d99df"

abbrev@1:
  version "1.1.0"
  resolved "https://registry.yarnpkg.com/abbrev/-/abbrev-1.1.0.tgz#d0554c2256636e2f56e7c2e5ad183f859428d81f"

ajv@^4.9.1:
  version "4.11.8"
  resolved "https://registry.yarnpkg.com/ajv/-/ajv-4.11.8.tgz#82ffb02b29e662ae53bdc20af15947706739c536"
  dependencies:
    co "^4.6.0"
    json-stable-stringify "^1.0.1"

As you can see, there is a URL to each tarball, plus a checksum.

First I tried npm v5 and the offline install worked. But the offline mode works using a cache directory, not with the tarballs like I wanted.

Yarn on the other hand does store the tarballs in a directory, and uses that directory to skip downloads if instructed to do so. They call this directory the "Offline mirror". Note Yarn also has a caching mechanism but the offline install will still work if the cache is cleaned.

With that, I was able to integrate Yarn with Flatpak as follows:

  • I converted the flatpak manifest into a template.
  • Created a script that parses the yarn.lock looking for URLs to tarballs and adds them as sources into the template, creating the final flatpak manifest.
  • Because Yarn uses SHA1 and flatpak uses SHA256, the script also validates the downloaded files with the SHA1 and calculates their SHA256, which is also added to the manifest.

The resulting manifest has a long list of files. The following is an extract that corresponds with the piece of lock file above:

{
    "type": "file",
    "url": "https://registry.yarnpkg.com/@types/node/-/node-7.0.29.tgz",
    "sha256": "bf52962861e3000e2fa6607b9aff389ab7b996768a0e4440025b2f8b0877c97d",
    "dest": "electron/npm-offline-cache/",
    "dest-filename": "@types-node-7.0.29.tgz"
},
{
    "type": "file",
    "url": "https://registry.yarnpkg.com/abbrev/-/abbrev-1.1.0.tgz",
    "sha256": "6071627b4a4c6bdf8f59fe07c76c51c695b818976f46c0aa19f7c65d259d7791",
    "dest": "electron/npm-offline-cache/"
},
{
    "type": "file",
    "url": "https://registry.yarnpkg.com/ajv/-/ajv-4.11.8.tgz",
    "sha256": "5e645008a327dfa21293ea60d2e2b9aaa383b44553da1b4126a7dc5a2034fcb7",
    "dest": "electron/npm-offline-cache/"
},

With the Yarn offline mirror recreated in the build directory, the application can now be installed paassing --offline in the Makefile:

cd electron && node yarn.js install --non-interactive --offline --verbose
cd electron && node yarn.js run pack --non-interactive --offline

Actually this build will hang at yarn install --non-interactive, because…

One more thing, Electron binaries

Well… Node being based on web technologies it seems to be in its very nature to always assume a network connection :-)

Even if we pass --offline to either Yarn or nmp v5, the packages can do whatever they want in their post installation scripts. Like, for example, an HTTP request. This is the case for Electron, that tries to download a binary in its post-script. This binary is architecture dependent.

Fortunately, if the binary happens to be already in a specific location in the host, the post-script will skip the download. So I added them to the flatpak manifest too.

{
    "type": "file",
    "url": "https://github.com/electron/electron/releases/download/v1.6.11/electron-v1.6.11-linux-x64.zip",
    "sha256": "b3b5151f047657fbb26b812d1b02117f526455f39b353d476b9afddc2849d448",
    "only-arches": ["x86_64"]
},
{
    "type": "file",
    "url": "https://github.com/electron/electron/releases/download/v1.6.11/electron-v1.6.11-linux-ia32.zip",
    "sha256": "f3366ddc1adf8fe090cbfc2c2c4fca93b427429b5a1ded1aa3a42133e5a0e406",
    "only-arches": ["i386"]
},
{
    "type": "file",
    "url": "https://github.com/electron/electron/releases/download/v1.6.11/electron-v1.6.11-linux-arm.zip",
    "sha256": "0bb9b6dd5238723b61d66489ef88f32348b61f30c2683c636a6f2d417fb6c096",
    "only-arches": ["arm"]
},
{
    "type": "file",
    "url": "https://github.com/electron/electron/releases/download/v1.6.11/SHASUMS256.txt",
    "sha256": "541d84500c141dc7c4cd84d70229544814a49110f44eb1097d2fa49ef269669a",
    "dest-filename": "SHASUMS256.txt-1.6.11"
},

These files are moved to the right place by the module Makefile. Note the only-arches attribute tells flatpak-builder to install the file only if building for that specific architecture.

# Copy electron binaries to electron default cache directory
mkdir -p ~/.electron
mv electron-v1.6.11-linux-x64.zip ~/.electron || true
mv electron-v1.6.11-linux-ia32.zip ~/.electron || true
mv electron-v1.6.11-linux-arm.zip ~/.electron || true
mv SHASUMS256.txt-1.6.11 ~/.electron

And that's it! Now flatpak-builder builds an Electron app fully offline. My test case was to run the steps separately. With --download-only first, then turn off my network connection and run --build-only, after that worked I ran --finish-only.

Here you can find an example Electron application that will succesfully build offline: https://github.com/manuq/org.sugarizer.Sugarizer

Pending bits

This takes us one step closer to have Electron apps in flathub.org . Still there is more work to be done. You can refer to this discussion in the Endless forums for a background of what is missing. It comes down to:

  • Move io.atom.electron.BaseApp to being a project built on flathub.org
  • Migrate io.atom.electron.BaseApp to Freedesktop runtime v1.6