Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rationalise building section #98

Merged
merged 1 commit into from
Mar 22, 2018
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions docs/building-basics.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
Building Concepts and Setup
===========================

The :doc:`introduction` and page on :doc:`first-build` have already shown how applications get built as Flatpaks, and have introduced many of the core ideas. This page describes some of these ideas and concepts in more detail. In doing so, it provides guidance on picking a runtime, getting setup to build applications, and when to bundle dependencies yourself.

Runtimes
--------

As was described in the :doc:`introduction`, runtimes provide basic dependencies that can be used by applications. They also provide the environment that applications run in.

Flatpak requires each application to specify a runtime, and this runtime must be present on a system for it to run. Therefore, one of the first decisions you need to make when building an application with Flatpak, is which runtime it will use.

An overview of the runtimes that are available can be found in the :doc:`available-runtimes` page. There are deliberately only a small number of runtimes to choose from. Typically, runtimes are picked on the basis of which dependencies an application requires. If a runtime exists that provides libraries that you plan on using, this is usually the correct runtime to use!

.. tip::

Runtimes require regular maintenance, and application developers should generally not consider creating their own.

Runtimes are automatically installed for users when they install an application, and build tools can also automatically install them for you (``flatpak-builder``'s ``--install-deps-from`` option is useful for this). However, if you do need to manually install your chosen runtime, this can be done in the same way as installing an application, with the ``flatpak install`` command. For example, the command to install the GNOME 3.26 runtime is::

$ flatpak install flathub org.gnome.Platform//3.26

Software Development Kits (SDKs)
--------------------------------

Each runtime is paired with an SDK (Software Develpment Kit). For example, the GNOME 3.26 runtime is accompanied by the GNOME 3.26 SDK. The SDK includes the same things as the regular runtime, but also includes all the development resources and tools that are required to build an application, such as build and packaging tools, header files, compilers and debuggers.

Applications must be built with the SDK that corresponds to their runtime, so that applications that use the GNOME 3.26 runtime must be built with the GNOME 3.26 SDK.

Like runtimes, SDKs will sometimes be automatically installed for you, but if you do need to manually install them, they are installed in the same was as applications and runtimes, such as::

$ flatpak install flathub org.gnome.Sdk//3.26

Bundling
--------

One of the key advantages of Flatpak is that it allows application authors to bundle whatever libraries or dependencies that they want. This means that developers aren't constrained by which libraries are available through Linux distributions. It also provides flexibility, allowing particular versions of libraries to be used, or the use of libraries that have been patched.

While bundling is very powerful and flexible, it also places a greater maintenance burden on the application developer. Therefore, while it is possible to bundle as much as you would like, it is generally recommended to try and keep the number of bundled modules as low as possible. If a dependency is available as part of a runtime, it is generally better to use the version from the runtime rather than bundle it yourself.

The specifics of how to bundle libraries is covered in the :doc:`manifests` section.
7 changes: 4 additions & 3 deletions docs/building.rst
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
Building
========

This section includes documentation on how to build apps with Flatpak. It covers runtimes and SDKs in more detail, followed by details on how use ``flatpak-builder``.
This section contains detailed information on how to build applications as Flatpaks, including information on concepts, guidance on key decisions, detailed information on how to use ``flatpak-builder``, and how to write manifest files.

If you haven't already, it is recommended to run through :doc:`first-build` before reading this section.

.. toctree::
:maxdepth: 2

runtimes-sdks
building-basics
flatpak-builder
manifests
flatpak-builder-tutorial
156 changes: 62 additions & 94 deletions docs/first-build.rst
Original file line number Diff line number Diff line change
@@ -1,130 +1,98 @@
Building your first Flatpak
Building Your First Flatpak
===========================

This tutorial provides a quick introduction to building Flatpaks. In it, you will learn how to create a basic Flatpak application, which can be installed and run.
This tutorial provides a quick, step-by-step example of how to build a simple application as a Flatpak. It also takes the opportunity to explain a few key concepts.

Please follow the :doc:`setup` page before doing this tutorial.
In order to complete this tutorial, you should have followed the `setup guide on flatpak.org <http://flatpak.org/setup/>`_. You also need to have installed ``flatpak-builder``, which is usually available from the same repository as the ``flatpak`` package.

1. Install a runtime and the matching SDK
-----------------------------------------
1. Create a manifest
--------------------

Flatpak requires every app to specify a runtime that it uses for its basic
dependencies. Each runtime has a matching SDK (Software Development Kit), which
contains all the things that are in the runtime, plus headers and development
tools. This SDK is required to build apps for the runtime.

In this tutorial we will use the Freedesktop 1.6 runtime and SDK. To install these, run::

$ flatpak install flathub org.freedesktop.Platform//1.6 org.freedesktop.Sdk//1.6

2. Create the app
-----------------

The app that is going to be created for this tutorial is a simple script. To
create it, copy the following::

#!/bin/sh
echo "Hello world, from a sandbox"

Now paste this into an empty file and save it as `hello.sh`.

3. Add a manifest
-----------------

Each Flatpak is built using a manifest file which provides basic information about the application and instructions for how it is to be built. To add a manifest to the hello world app, add the following to an empty file:
The input to ``flatpak-builder`` is a JSON file that describes the parameters for building the application. This is called the manifest. The following example is the manifest for the GNOME Dictionary application:

.. code-block:: json

{
"app-id": "org.flatpak.Hello",
"runtime": "org.freedesktop.Platform",
"runtime-version": "1.6",
"sdk": "org.freedesktop.Sdk",
"command": "hello.sh",
"modules": [
"app-id": "org.gnome.Dictionary",
"runtime": "org.gnome.Platform",
"runtime-version": "3.26",
"sdk": "org.gnome.Sdk",
"command": "gnome-dictionary",
"finish-args": [
"--socket=x11",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realize you are trying to be minimal but this leaves some bad habits like not including --socket=wayland and --share=ipc for Gtk3 applications.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only reason those are the only sandbox permissions in the example is that they're what's in the manifest I inherited. I'd love to have a better demo with a sane set of options in it (that's #97)!

Copy link
Member

@TingPing TingPing Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All Gtk3 applications will have like 7 or so permissions to just be fully functional.

"--share=network"
],
"modules": [
{
"name": "gnome-dictionary",
"sources": [
{
"name": "hello",
"buildsystem": "simple",
"build-commands": [
"install -D hello.sh /app/bin/hello.sh"
],
"sources": [
{
"type": "file",
"path": "hello.sh"
}
]
"type": "archive",
"url": "https://download.gnome.org/sources/gnome-dictionary/3.26/gnome-dictionary-3.26.0.tar.xz",
"sha256": "387ff8fbb8091448453fd26dcf0b10053601c662e59581097bc0b54ced52e9ef"
}
]
]
}
]
}

Now save the file alongside `hello.sh` and call it `org.flatpak.Hello.json`.

In a more complex application, the manifest would list multiple modules. The
last one would typically be the application itself, and the earlier ones would
be dependencies that are bundled with the app because they are not part of the
runtime.
As can be seen, this manifest includes basic information about the application, including:

4. Build the application
------------------------
- ``app-id`` - the application ID
- ``runtime`` - the runtime, which provides the environment that the application will run in, as well as basic dependencies it can use
- ``sdk`` - the Software Development Kit, which is a version of the runtime with development files and headers; this is required to build the app
- ``command`` - the command that is used to run the application
- ``finish-args`` - configuration options which give the application access to resources outside of its sandbox; in this case, the application is being given network access and access to the X11 display server

Now that the app has a manifest, `flatpak-builder` can be used to build it.
This is done by specifying the manifest file and a target directory::
The next part of the manifest is the ``modules`` list. This describes each of the modules that are to be built as part of the build process. One of these modules is always the application. Others can be libraries and other resources that are bundled as part of the application.

$ flatpak-builder app-dir org.flatpak.Hello.json
Module sources can be of several types, including ``.tar`` or ``.zip`` archives, Git or Bzr repositories. In this case, there is only one module, which is a ``.tar`` file which will be downloaded and built.

This command will build each module that is listed in the manifest and install
it to the `/app` subdirectory, inside the `app-dir` directory.
The modules section of the Dictionary manifest is short, because only one module is built: the application itself.

5. Test the build
-----------------
To create a manifest for the Dictionary, create a file called ``org.gnome.Dictionary.json`` and paste the JSON from above into it.

To verify that the build was successful, run the following::
2. Run the build
----------------

$ flatpak-builder --run app-dir org.flatpak.Hello.json hello.sh
To use the manifest to build the Dictionary application, run the following command::

Congratulations, you've made an app!
$ flatpak-builder --repo=tutorial-repo dictionary org.gnome.Dictionary.json

6. Put the app in a repository
------------------------------
This will:

Before you can install and run the app, it first needs to be put in a
repository. This is done by passing the `--repo` argument to `flatpak-builder`::
* Create a new directory called dictionary
* Download and verify the Dictionary source code
* Build and install the source code, inside the SDK rather than the host system
* Finish the build, by setting permissions (in this case giving access to X11 and the network)
* Create a new repository called repo (if it doesn't exist) and export the resulting build into it

$ flatpak-builder --repo=repo --force-clean app-dir org.flatpak.Hello.json
``flatpak-builder`` will also do some other useful things, like creating a separately installable debug runtime (called ``org.gnome.Dictionary.Debug`` in this case) and a separately installable translation runtime (called ``org.gnome.Dictionary.Locale``).

This does the build again, and at the end exports the result to a local
directory called `repo`. Note that `flatpak-builder` keeps a cache of previous
builds in the `.flatpak-builder` subdirectory, so doing a second build like
this is very fast.
3. Add the new repository
-------------------------

This second time we passed in `--force-clean`, which means that the previously
created `app-dir` directory was deleted before the new build was started.
To test the application that has been built, you need to add the new repository that has been created::

7. Install the app
------------------
$ flatpak --user remote-add --no-gpg-verify --if-not-exists tutorial-repo tutorial-repo

Now we're ready to add the repository that was just created and install the
app. This is done with two commands::
4. Install the application
--------------------------

$ flatpak --user remote-add --no-gpg-verify tutorial-repo repo
$ flatpak --user install tutorial-repo org.flatpak.Hello
The next step is to install the Dictionary application from the repository. To do this, run::

The first command adds the repository that was created in the previous step.
The second command installs the app from the repository.
$ flatpak --user install tutorial-repo org.gnome.Dictionary

Both these commands use the `--user` argument, which means that the repository
and the app are added per-user rather than system-wide. This is useful for testing.
To check that the application has been successfully installed, you can compare the sha256 commit of the installed app with the commit ID that was printed by ``flatpak-builder``::

Note that the repository was added with `--no-gpg-verify`, since a GPG key
wasn't specified when the app was built. This is fine for testing, but for
official repositories you should sign them with a private GPG key.
$ flatpak info org.gnome.Dictionary
$ flatpak info org.gnome.Dictionary.Locale

8. Run the app
--------------
5. Run the application
----------------------

All that's left is to try the app. This can be done with the following command::
Finally, you can run the application that you've built::

$ flatpak run org.flatpak.Hello
$ flatpak run org.gnome.Dictionary

This runs the app, so that it prints `Hello world, from a sandbox`.
The rest of the documentation provides a complete guide to using ``flatpak-builder``. If you are new to Flatpak, it is recommended to start with the :doc:`introduction`.
52 changes: 0 additions & 52 deletions docs/flatpak-builder-tutorial.rst

This file was deleted.

32 changes: 22 additions & 10 deletions docs/flatpak-builder.rst
Original file line number Diff line number Diff line change
@@ -1,42 +1,54 @@
Flatpak Builder
===============

Flatpaks are built using the ``flatpak-builder`` tool. This allows a series of modules to be built into a single application bundle. These modules can include libraries and dependencies in addition to the application itself.
``flatpak-builder`` has already been introduced in :doc:`first-build`, which gave a basic example of how it can be used. This page provides additional detail on how to use ``flatpak-builder``, including the various command options that are available.

Modules can be built with a variety of build systems, including `autotools <https://www.gnu.org/software/automake/manual/html_node/Autotools-Introduction.html>`_, `cmake <https://cmake.org/>`_, `cmake-ninja <https://cmake.org/cmake/help/v3.0/generator/Ninja.html>`_, `meson <http://mesonbuild.com/>`_, and the so called `Build API <https://github.com/cgwalters/build-api/>`_. A "simple" build method is also available, which allows a series of commands to be specified.
Invocation
----------

The basic format used to invoke ``flatpak-builder`` is::

$ flatpak-builder <directory> <manifest>

Where ``<directory>`` is the path to the directory that the application will be built into, and ``<manifest>`` is the path to a manifest JSON file.

The contents of ``<directory>`` can be useful for testing and debugging purposes, but is generally treated as an artifact of the build process.

Exporting
---------

The result of the build process can be exported to a repository or automatically installed locally.
``flatpak-builder`` provides two options for exporting an application in order to run it. The first is to export to a repository, from which the application can be run. The second is to automatically install locally.

Exporting to a repository
`````````````````````````

The ``--repo`` option allows a repository to be specified, which the resulting application will be added to. This takes the format::
The ``--repo`` option allows a repository to be specified, for the application to be exported to. This takes the format::

$ flatpak-builder --repo=<repository-destination> application.id.json
$ flatpak-builder --repo=<repository> <directory> <manifest>

Here, ``<repository>`` is a path to a repository. If no repository exists at the specified location, the repository will be created. If the application is already in the specified repository, ``flatpak-builder`` will add the build as a new version of the existing application.

.. note::

By default, flatpak-builder splits off translations and debug information into separate `.Locale` and `.Debug` extensions. These extensions are automatically exported into a repository along with the application.
By default, ``flatpak-builder`` splits off translations and debug information into separate `.Locale` and `.Debug` extensions. These extensions are automatically exported into a repository along with the application.


Installing builds directly
``````````````````````````

Instead of exporting to a repository, the application bundle that is produced by ``flatpak-builder`` can be automatically installed locally::
Instead of exporting to a repository, the Flatpak that is produced by ``flatpak-builder`` can be automatically installed locally, using the ``--install`` option::

$ flatpak-builder --install <directory> <manifest>

$ flatpak-builder --install application.id.json
This approach has the advantage of skipping the separate install step that is needed when exporting to a repository.

Signing
-------

Every commit to a Flatpak repository should be signed with a GPG signature. If ``flatpak-builder`` is being used to modify or create a repository, a GPG key should therefore be passed to it. This can be done with the ``--gpg-sign`` option, such as::

$ flatpak-builder --gpg-sign=<key-id> --repo=<repository-destination> application.id.json
$ flatpak-builder --gpg-sign=<key> --repo=<repository> <manifest>

The ``--gpg-homedir`` option can also be used to specify the home directory of the key that is being used.
Here, ``<key>`` is the ID of the GPG key that is to be used. The ``--gpg-homedir`` option can also be used to specify the home directory of the key that is being used.

Though it generally isn't recommended, it is possible not to use GPG verification. In this case, the ``--no-gpg-verify`` option should be used when adding the repository. Note that it is necessary to become root in order to update a repository that does not have GPG verification enabled.
5 changes: 3 additions & 2 deletions docs/getting-started.rst
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
Getting Started
===============

First steps with Flatpak, including installation, basic usage, and building your first app.
Information on how to take your first steps with Flatpak, including basic command line usage and how to build a simple application.

To complete this section, Flatpak should have been installed and the Flathub repository have been enabled. The Flatpak website provides `instructions for how to do this with a range of distributions <https://flatpak.org/setup/>`_.

.. toctree::
:maxdepth: 2

setup
using-flatpak
first-build
129 changes: 92 additions & 37 deletions docs/manifests.rst
Original file line number Diff line number Diff line change
@@ -3,46 +3,59 @@ Manifests

The input to ``flatpak-builder`` is a JSON file that describes the parameters for building an application, as well as instructions for each of the modules that are to be built. This file is called the manifest.

The following example is the manifest for the GNOME Dictionary application. It is short, because only one module is built - the application itself::

{
"app-id": "org.gnome.Dictionary",
"runtime": "org.gnome.Platform",
"runtime-version": "3.26",
"sdk": "org.gnome.Sdk",
"command": "gnome-dictionary",
"finish-args": [
"--socket=x11",
"--share=network"
],
"modules": [
{
"name": "gnome-dictionary",
"sources": [
{
"type": "archive",
"url": "https://download.gnome.org/sources/gnome-dictionary/3.26/gnome-dictionary-3.26.0.tar.xz",
"sha256": "387ff8fbb8091448453fd26dcf0b10053601c662e59581097bc0b54ced52e9ef"
}
]
}
]
}
This page provides information and guidance on how to use manifests, including an explanation of the most common parameters that can be specified. It is recommended to have followed the :doc:`first-build` tutorial before reading this section, and to be familiar with :doc:`flatpak-builder`.

Manifest files should be named using the application ID. For example, the manifest file for GNOME Dictionary is named ``org.gnome.Dictionary.json``. This page uses this manifest file, which was introduced in :doc:`first-build`, for all its examples.

A complete list of all the properties that can be specified in manifest files can be found in the :doc:`flatpak-builder-command-reference`, as well as the ``flatpak-builder`` man page.

Basic properties
----------------

Each manifest file should specify basic information about the application that is to be built, including the ``app-id``, ``runtime``, ``runtime-version``, ``sdk`` and ``command`` parameters. These properties are typically specified at the beginning of the file.

For example, the GNOME Dictionary manifest includes:

.. code-block:: json
"app-id": "org.gnome.Dictionary",
"runtime": "org.gnome.Platform",
"runtime-version": "3.26",
"sdk": "org.gnome.Sdk",
"command": "gnome-dictionary",
Specifying a runtime and runtime version allows that the runtime that is needed by your application to be automatically installed on users' systems.

File renaming
-------------

As was described in the :doc:`introduction`, exports are application files that are made available to the host, and include things like the application's ``.desktop`` file and icon.

The names of files that are exported by a Flatpak must prefixed using the application ID, such as ``org.gnome.Dictionary.desktop``. The best way to do this is to rename these files directly in the application's source.

If renaming exported files to use the application ID is not possible, ``flatpak-builder`` allows them to be renamed as part of the build process. This can be done by specifying one of the following properties in the manifest:

As can be seen, this manifest includes basic information about the application before specifying a single .tar file to be downloaded and built. More complex manifests include a sequence of modules. Module sources can be of several types, including ``.tar`` or ``.zip`` archives, Git or Bzr repositories, patch files or shell commands that are run.
- ``rename-icon`` - rename the application icon
- ``rename-desktop-file`` - rename the ``.desktop`` filename
- ``rename-appdata-file`` - rename the AppData file

Each of the properties that can be specified in a manifest file are listed in the :doc:`flatpak-builder-command-reference`, as well as the ``flatpak-builder`` man page.
Each of these properties accepts the name of the source file to be renamed. ``flatpak-builder`` then automatically renames the file to match the application ID. Note that this renaming method can introduce internal naming conflicts, and that renaming files in tree is therefore the most reliable approach.

.. TODO
Finishing
---------

Add a section on the top level options, possibly called "Application metadata". Ideally this should recommend a basic set of options that developers should start with.
Flatpaks have extremely limited access to the host environment by default, but applications require access to resources outside of their sandbox in order to be useful. Finishing is the build stage where the application's sandbox permissions are specified, in order to give access to these resources.

Add a section on specifying the list of modules. There has to be something to say here...
The finishing manifest section uses the ``finish-args`` property, which can be seen in the Dictionary manifest file:

finish-args
-----------
.. code-block:: json
Flatpaks have extremely limited access to the host environment by default. However, most applications require access to resources outside of their sandbox in order to be useful. This can be achieved with the ``finish-args`` manifest section, which allows sandbox permissions to be configured.
"finish-args": [
"--socket=x11",
"--share=network"
],
As was explained in :doc:`first-build`, these two finishing properties give the application access to the X11 display server and to the network.

While there are no restrictions on which sandbox permissions an application can use, as good practice, it is recommended to use the minimum number of as permissions possible. Certain permissions, such as blanket access to the system bus (using the ``--socket=system-bus`` option) are strongly discouraged.

@@ -51,7 +64,7 @@ A list of ``finish-args`` options can be found in :doc:`sandbox-permissions`.
Cleanup
-------

After building has taken place, ``flatpak-builder`` performs a cleanup phase. This can be used to remove headers and development documentation, among other things. Two properties in the manifest file are used for this. First, a list of filename patterns can be included::
The cleanup property can be used to remove files that are produced by the build process but which aren't wanted as part of the application, such as headers or developer documentation. Two properties in the manifest file are used for this. First, a list of filename patterns can be included::

"cleanup": [ "/include", "/bin/foo-*", "*.a" ]

@@ -61,10 +74,52 @@ The second cleanup property is a list of commands that are run during the cleanu

Cleanup properties can be set on a per-module basis, in which case only filenames that were created by that particular module will be matched.

File renaming
-------------
Modules
-------

The module list specifies each of the modules that are to be built as part of the build process. One of these modules is the application itself, and other modules are dependencies and libraries that are bundled as part of the Flatpak. While simple applications may only specify one or two modules, and therefore have short modules sections, some applications can bundle numerous modules and therefore have lengthy modules sections.

GNOME Dictionary's modules section is short, since it just contains the application itself, and looks like:

.. code-block:: json
"modules": [
{
"name": "gnome-dictionary",
"sources": [
{
"type": "archive",
"url": "https://download.gnome.org/sources/gnome-dictionary/3.26/gnome-dictionary-3.26.0.tar.xz",
"sha256": "387ff8fbb8091448453fd26dcf0b10053601c662e59581097bc0b54ced52e9ef"
}
]
}
]
As can be seen, each listed module has a ``name`` (which can be freely assigned) and a list of ``sources``. Each source has a ``type``, and available types include:

- ``archive`` - ``.tar`` or ``.zip`` archive files
- ``git`` - Git repositories
- ``bzr`` - Bazaar repositories
- ``file`` - local file (these are copied into the source directory)
- ``script`` - an array of shell commands (these are put in a shellscript file)
- ``shell`` -an array of shell commands that are run during source extraction
- ``patch`` - a patch (are applied to the source directory)
- ``extra-data`` - data that can be downloaded at install time; this can include archive or package files

Different properties are available for each source type, which are listed in the :doc:`flatpak-builder-command-reference`.

Supported build systems
```````````````````````

Modules can be built with a variety of build systems, including:

- `autotools <https://www.gnu.org/software/automake/manual/html_node/Autotools-Introduction.html>`_
- `cmake <https://cmake.org/>`_
- `cmake-ninja <https://cmake.org/cmake/help/v3.0/generator/Ninja.html>`_, `meson <http://mesonbuild.com/>`_
- the "`Build API <https://github.com/cgwalters/build-api/>`_"

Files that are exported by a flatpak must be prefixed using the application ID. If an application's source files are not named using this convention, flatpak-builder allows them to be renamed as part of the build process. To rename application icons, desktop files and AppData files, use the ``rename-icon``, ``rename-desktop-file`` and ``rename-appdata-file`` properties.
A "simple" build method is also available, which allows a series of commands to be specified.

Example manifests
-----------------
33 changes: 0 additions & 33 deletions docs/runtimes-sdks.rst

This file was deleted.

14 changes: 0 additions & 14 deletions docs/setup.rst

This file was deleted.