diff options
Diffstat (limited to 'poky/documentation/overview-manual')
13 files changed, 3931 insertions, 6795 deletions
diff --git a/poky/documentation/overview-manual/history.rst b/poky/documentation/overview-manual/history.rst new file mode 100644 index 0000000000..5fde8f09dd --- /dev/null +++ b/poky/documentation/overview-manual/history.rst @@ -0,0 +1,40 @@ +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK + +*********************** +Manual Revision History +*********************** + +.. list-table:: + :widths: 10 15 40 + :header-rows: 1 + + * - Revision + - Date + - Note + * - 2.5 + - May 2018 + - The initial document released with the Yocto Project 2.5 Release + * - 2.6 + - November 2018 + - Released with the Yocto Project 2.6 Release. + * - 2.7 + - May 2019 + - Released with the Yocto Project 2.7 Release. + * - 3.0 + - October 2019 + - Released with the Yocto Project 3.0 Release. + * - 3.1 + - April 2020 + - Released with the Yocto Project 3.1 Release. + * - 3.1.1 + - June 2020 + - Released with the Yocto Project 3.1.1 Release. + * - 3.1.2 + - August 2020 + - Released with the Yocto Project 3.1.2 Release. + * - 3.1.3 + - September 2020 + - Released with the Yocto Project 3.1.3 Release. + * - 3.1.4 + - November 2020 + - Released with the Yocto Project 3.1.4 Release. diff --git a/poky/documentation/overview-manual/overview-manual-concepts.rst b/poky/documentation/overview-manual/overview-manual-concepts.rst new file mode 100644 index 0000000000..d9f50e5194 --- /dev/null +++ b/poky/documentation/overview-manual/overview-manual-concepts.rst @@ -0,0 +1,2185 @@ +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK + +********************** +Yocto Project Concepts +********************** + +This chapter provides explanations for Yocto Project concepts that go +beyond the surface of "how-to" information and reference (or look-up) +material. Concepts such as components, the :term:`OpenEmbedded Build System` +workflow, +cross-development toolchains, shared state cache, and so forth are +explained. + +Yocto Project Components +======================== + +The :term:`BitBake` task executor +together with various types of configuration files form the +:term:`OpenEmbedded-Core (OE-Core)`. This section +overviews these components by describing their use and how they +interact. + +BitBake handles the parsing and execution of the data files. The data +itself is of various types: + +- *Recipes:* Provides details about particular pieces of software. + +- *Class Data:* Abstracts common build information (e.g. how to build a + Linux kernel). + +- *Configuration Data:* Defines machine-specific settings, policy + decisions, and so forth. Configuration data acts as the glue to bind + everything together. + +BitBake knows how to combine multiple data sources together and refers +to each data source as a layer. For information on layers, see the +":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`" +section of the Yocto Project Development Tasks Manual. + +Following are some brief details on these core components. For +additional information on how these components interact during a build, +see the +":ref:`overview-manual/overview-manual-concepts:openembedded build system concepts`" +section. + +.. _usingpoky-components-bitbake: + +BitBake +------- + +BitBake is the tool at the heart of the :term:`OpenEmbedded Build System` +and is responsible +for parsing the :term:`Metadata`, generating +a list of tasks from it, and then executing those tasks. + +This section briefly introduces BitBake. If you want more information on +BitBake, see the :doc:`BitBake User Manual <bitbake:index>`. + +To see a list of the options BitBake supports, use either of the +following commands: +:: + + $ bitbake -h + $ bitbake --help + +The most common usage for BitBake is ``bitbake recipename``, where +``recipename`` is the name of the recipe you want to build (referred +to as the "target"). The target often equates to the first part of a +recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``). +So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might +type the following: +:: + + $ bitbake matchbox-desktop + +Several different +versions of ``matchbox-desktop`` might exist. BitBake chooses the one +selected by the distribution configuration. You can get more details +about how BitBake chooses between different target versions and +providers in the +":ref:`Preferences <bitbake:bb-bitbake-preferences>`" section +of the BitBake User Manual. + +BitBake also tries to execute any dependent tasks first. So for example, +before building ``matchbox-desktop``, BitBake would build a cross +compiler and ``glibc`` if they had not already been built. + +A useful BitBake option to consider is the ``-k`` or ``--continue`` +option. This option instructs BitBake to try and continue processing the +job as long as possible even after encountering an error. When an error +occurs, the target that failed and those that depend on it cannot be +remade. However, when you use this option other dependencies can still +be processed. + +.. _overview-components-recipes: + +Recipes +------- + +Files that have the ``.bb`` suffix are "recipes" files. In general, a +recipe contains information about a single piece of software. This +information includes the location from which to download the unaltered +source, any source patches to be applied to that source (if needed), +which special configuration options to apply, how to compile the source +files, and how to package the compiled output. + +The term "package" is sometimes used to refer to recipes. However, since +the word "package" is used for the packaged output from the OpenEmbedded +build system (i.e. ``.ipk`` or ``.deb`` files), this document avoids +using the term "package" when referring to recipes. + +.. _overview-components-classes: + +Classes +------- + +Class files (``.bbclass``) contain information that is useful to share +between recipes files. An example is the +:ref:`autotools <ref-classes-autotools>` class, +which contains common settings for any application that Autotools uses. +The ":ref:`ref-manual/ref-classes:Classes`" chapter in the +Yocto Project Reference Manual provides details about classes and how to +use them. + +.. _overview-components-configurations: + +Configurations +-------------- + +The configuration files (``.conf``) define various configuration +variables that govern the OpenEmbedded build process. These files fall +into several areas that define machine configuration options, +distribution configuration options, compiler tuning options, general +common configuration options, and user configuration options in +``conf/local.conf``, which is found in the :term:`Build Directory`. + + +.. _overview-layers: + +Layers +====== + +Layers are repositories that contain related metadata (i.e. sets of +instructions) that tell the OpenEmbedded build system how to build a +target. Yocto Project's `layer model <#the-yocto-project-layer-model>`__ +facilitates collaboration, sharing, customization, and reuse within the +Yocto Project development environment. Layers logically separate +information for your project. For example, you can use a layer to hold +all the configurations for a particular piece of hardware. Isolating +hardware-specific configurations allows you to share other metadata by +using a different layer where that metadata might be common across +several pieces of hardware. + +Many layers exist that work in the Yocto Project development +environment. The `Yocto Project Curated Layer +Index <https://www.yoctoproject.org/software-overview/layers/>`__ +and `OpenEmbedded Layer +Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__ +both contain layers from which you can use or leverage. + +By convention, layers in the Yocto Project follow a specific form. +Conforming to a known structure allows BitBake to make assumptions +during builds on where to find types of metadata. You can find +procedures and learn about tools (i.e. ``bitbake-layers``) for creating +layers suitable for the Yocto Project in the +":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`" +section of the Yocto Project Development Tasks Manual. + +.. _openembedded-build-system-build-concepts: + +OpenEmbedded Build System Concepts +================================== + +This section takes a more detailed look inside the build process used by +the :term:`OpenEmbedded Build System`, +which is the build +system specific to the Yocto Project. At the heart of the build system +is BitBake, the task executor. + +The following diagram represents the high-level workflow of a build. The +remainder of this section expands on the fundamental input, output, +process, and metadata logical blocks that make up the workflow. + +.. image:: figures/YP-flow-diagram.png + :align: center + +In general, the build's workflow consists of several functional areas: + +- *User Configuration:* metadata you can use to control the build + process. + +- *Metadata Layers:* Various layers that provide software, machine, and + distro metadata. + +- *Source Files:* Upstream releases, local projects, and SCMs. + +- *Build System:* Processes under the control of + :term:`BitBake`. This block expands + on how BitBake fetches source, applies patches, completes + compilation, analyzes output for package generation, creates and + tests packages, generates images, and generates cross-development + tools. + +- *Package Feeds:* Directories containing output packages (RPM, DEB or + IPK), which are subsequently used in the construction of an image or + Software Development Kit (SDK), produced by the build system. These + feeds can also be copied and shared using a web server or other means + to facilitate extending or updating existing images on devices at + runtime if runtime package management is enabled. + +- *Images:* Images produced by the workflow. + +- *Application Development SDK:* Cross-development tools that are + produced along with an image or separately with BitBake. + +User Configuration +------------------ + +User configuration helps define the build. Through user configuration, +you can tell BitBake the target architecture for which you are building +the image, where to store downloaded source, and other build properties. + +The following figure shows an expanded representation of the "User +Configuration" box of the `general workflow +figure <#general-workflow-figure>`__: + +.. image:: figures/user-configuration.png + :align: center + +BitBake needs some basic configuration files in order to complete a +build. These files are ``*.conf`` files. The minimally necessary ones +reside as example files in the ``build/conf`` directory of the +:term:`Source Directory`. For simplicity, +this section refers to the Source Directory as the "Poky Directory." + +When you clone the :term:`Poky` Git repository +or you download and unpack a Yocto Project release, you can set up the +Source Directory to be named anything you want. For this discussion, the +cloned repository uses the default name ``poky``. + +.. note:: + + The Poky repository is primarily an aggregation of existing + repositories. It is not a canonical upstream source. + +The ``meta-poky`` layer inside Poky contains a ``conf`` directory that +has example configuration files. These example files are used as a basis +for creating actual configuration files when you source +:ref:`structure-core-script`, which is the +build environment script. + +Sourcing the build environment script creates a +:term:`Build Directory` if one does not +already exist. BitBake uses the Build Directory for all its work during +builds. The Build Directory has a ``conf`` directory that contains +default versions of your ``local.conf`` and ``bblayers.conf`` +configuration files. These default configuration files are created only +if versions do not already exist in the Build Directory at the time you +source the build environment setup script. + +Because the Poky repository is fundamentally an aggregation of existing +repositories, some users might be familiar with running the +:ref:`structure-core-script` script in the context of separate +:term:`OpenEmbedded-Core (OE-Core)` and BitBake +repositories rather than a single Poky repository. This discussion +assumes the script is executed from within a cloned or unpacked version +of Poky. + +Depending on where the script is sourced, different sub-scripts are +called to set up the Build Directory (Yocto or OpenEmbedded). +Specifically, the script ``scripts/oe-setup-builddir`` inside the poky +directory sets up the Build Directory and seeds the directory (if +necessary) with configuration files appropriate for the Yocto Project +development environment. + +.. note:: + + The + scripts/oe-setup-builddir + script uses the + ``$TEMPLATECONF`` + variable to determine which sample configuration files to locate. + +The ``local.conf`` file provides many basic variables that define a +build environment. Here is a list of a few. To see the default +configurations in a ``local.conf`` file created by the build environment +script, see the +:yocto_git:`local.conf.sample </cgit/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample>` +in the ``meta-poky`` layer: + +- *Target Machine Selection:* Controlled by the + :term:`MACHINE` variable. + +- *Download Directory:* Controlled by the + :term:`DL_DIR` variable. + +- *Shared State Directory:* Controlled by the + :term:`SSTATE_DIR` variable. + +- *Build Output:* Controlled by the + :term:`TMPDIR` variable. + +- *Distribution Policy:* Controlled by the + :term:`DISTRO` variable. + +- *Packaging Format:* Controlled by the + :term:`PACKAGE_CLASSES` + variable. + +- *SDK Target Architecture:* Controlled by the + :term:`SDKMACHINE` variable. + +- *Extra Image Packages:* Controlled by the + :term:`EXTRA_IMAGE_FEATURES` + variable. + +.. note:: + + Configurations set in the + conf/local.conf + file can also be set in the + conf/site.conf + and + conf/auto.conf + configuration files. + +The ``bblayers.conf`` file tells BitBake what layers you want considered +during the build. By default, the layers listed in this file include +layers minimally needed by the build system. However, you must manually +add any custom layers you have created. You can find more information on +working with the ``bblayers.conf`` file in the +":ref:`dev-manual/dev-manual-common-tasks:enabling your layer`" +section in the Yocto Project Development Tasks Manual. + +The files ``site.conf`` and ``auto.conf`` are not created by the +environment initialization script. If you want the ``site.conf`` file, +you need to create that yourself. The ``auto.conf`` file is typically +created by an autobuilder: + +- *site.conf:* You can use the ``conf/site.conf`` configuration + file to configure multiple build directories. For example, suppose + you had several build environments and they shared some common + features. You can set these default build properties here. A good + example is perhaps the packaging format to use through the + :term:`PACKAGE_CLASSES` + variable. + + One useful scenario for using the ``conf/site.conf`` file is to + extend your :term:`BBPATH` variable + to include the path to a ``conf/site.conf``. Then, when BitBake looks + for Metadata using ``BBPATH``, it finds the ``conf/site.conf`` file + and applies your common configurations found in the file. To override + configurations in a particular build directory, alter the similar + configurations within that build directory's ``conf/local.conf`` + file. + +- *auto.conf:* The file is usually created and written to by an + autobuilder. The settings put into the file are typically the same as + you would find in the ``conf/local.conf`` or the ``conf/site.conf`` + files. + +You can edit all configuration files to further define any particular +build environment. This process is represented by the "User +Configuration Edits" box in the figure. + +When you launch your build with the ``bitbake target`` command, BitBake +sorts out the configurations to ultimately define your build +environment. It is important to understand that the +:term:`OpenEmbedded Build System` reads the +configuration files in a specific order: ``site.conf``, ``auto.conf``, +and ``local.conf``. And, the build system applies the normal assignment +statement rules as described in the +":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter +of the BitBake User Manual. Because the files are parsed in a specific +order, variable assignments for the same variable could be affected. For +example, if the ``auto.conf`` file and the ``local.conf`` set variable1 +to different values, because the build system parses ``local.conf`` +after ``auto.conf``, variable1 is assigned the value from the +``local.conf`` file. + +Metadata, Machine Configuration, and Policy Configuration +--------------------------------------------------------- + +The previous section described the user configurations that define +BitBake's global behavior. This section takes a closer look at the +layers the build system uses to further control the build. These layers +provide Metadata for the software, machine, and policies. + +In general, three types of layer input exists. You can see them below +the "User Configuration" box in the `general workflow +figure <#general-workflow-figure>`__: + +- *Metadata (.bb + Patches):* Software layers containing + user-supplied recipe files, patches, and append files. A good example + of a software layer might be the + `meta-qt5 layer <https://github.com/meta-qt5/meta-qt5>`__ from + the `OpenEmbedded Layer + Index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__. + This layer is for version 5.0 of the popular + `Qt <https://wiki.qt.io/About_Qt>`__ cross-platform application + development framework for desktop, embedded and mobile. + +- *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e. + "BSP Layer" in the following figure) providing machine-specific + configurations. This type of information is specific to a particular + target architecture. A good example of a BSP layer from the `Poky + Reference Distribution <#gs-reference-distribution-poky>`__ is the + :yocto_git:`meta-yocto-bsp </cgit/cgit.cgi/poky/tree/meta-yocto-bsp>` + layer. + +- *Policy Configuration:* Distribution Layers (i.e. "Distro Layer" in + the following figure) providing top-level or general policies for the + images or SDKs being built for a particular distribution. For + example, in the Poky Reference Distribution the distro layer is the + :yocto_git:`meta-poky </cgit/cgit.cgi/poky/tree/meta-poky>` + layer. Within the distro layer is a ``conf/distro`` directory that + contains distro configuration files (e.g. + :yocto_git:`poky.conf </cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf>` + that contain many policy configurations for the Poky distribution. + +The following figure shows an expanded representation of these three +layers from the `general workflow figure <#general-workflow-figure>`__: + +.. image:: figures/layer-input.png + :align: center + +In general, all layers have a similar structure. They all contain a +licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed, +a ``README`` file as good practice and especially if the layer is to be +distributed, a configuration directory, and recipe directories. You can +learn about the general structure for layers used with the Yocto Project +in the +":ref:`dev-manual/dev-manual-common-tasks:creating your own layer`" +section in the +Yocto Project Development Tasks Manual. For a general discussion on +layers and the many layers from which you can draw, see the +"`Layers <#overview-layers>`__" and "`The Yocto Project Layer +Model <#the-yocto-project-layer-model>`__" sections both earlier in this +manual. + +If you explored the previous links, you discovered some areas where many +layers that work with the Yocto Project exist. The `Source +Repositories <http://git.yoctoproject.org/>`__ also shows layers +categorized under "Yocto Metadata Layers." + +.. note:: + + Layers exist in the Yocto Project Source Repositories that cannot be + found in the OpenEmbedded Layer Index. These layers are either + deprecated or experimental in nature. + +BitBake uses the ``conf/bblayers.conf`` file, which is part of the user +configuration, to find what layers it should be using as part of the +build. + +Distro Layer +~~~~~~~~~~~~ + +The distribution layer provides policy configurations for your +distribution. Best practices dictate that you isolate these types of +configurations into their own layer. Settings you provide in +``conf/distro/distro.conf`` override similar settings that BitBake finds +in your ``conf/local.conf`` file in the Build Directory. + +The following list provides some explanation and references for what you +typically find in the distribution layer: + +- *classes:* Class files (``.bbclass``) hold common functionality that + can be shared among recipes in the distribution. When your recipes + inherit a class, they take on the settings and functions for that + class. You can read more about class files in the + ":ref:`ref-manual/ref-classes:Classes`" chapter of the Yocto + Reference Manual. + +- *conf:* This area holds configuration files for the layer + (``conf/layer.conf``), the distribution + (``conf/distro/distro.conf``), and any distribution-wide include + files. + +- *recipes-*:* Recipes and append files that affect common + functionality across the distribution. This area could include + recipes and append files to add distribution-specific configuration, + initialization scripts, custom image recipes, and so forth. Examples + of ``recipes-*`` directories are ``recipes-core`` and + ``recipes-extra``. Hierarchy and contents within a ``recipes-*`` + directory can vary. Generally, these directories contain recipe files + (``*.bb``), recipe append files (``*.bbappend``), directories that + are distro-specific for configuration files, and so forth. + +BSP Layer +~~~~~~~~~ + +The BSP Layer provides machine configurations that target specific +hardware. Everything in this layer is specific to the machine for which +you are building the image or the SDK. A common structure or form is +defined for BSP layers. You can learn more about this structure in the +:doc:`../bsp-guide/bsp-guide`. + +.. note:: + + In order for a BSP layer to be considered compliant with the Yocto + Project, it must meet some structural requirements. + +The BSP Layer's configuration directory contains configuration files for +the machine (``conf/machine/machine.conf``) and, of course, the layer +(``conf/layer.conf``). + +The remainder of the layer is dedicated to specific recipes by function: +``recipes-bsp``, ``recipes-core``, ``recipes-graphics``, +``recipes-kernel``, and so forth. Metadata can exist for multiple +formfactors, graphics support systems, and so forth. + +.. note:: + + While the figure shows several + recipes-\* + directories, not all these directories appear in all BSP layers. + +Software Layer +~~~~~~~~~~~~~~ + +The software layer provides the Metadata for additional software +packages used during the build. This layer does not include Metadata +that is specific to the distribution or the machine, which are found in +their respective layers. + +This layer contains any recipes, append files, and patches, that your +project needs. + +.. _sources-dev-environment: + +Sources +------- + +In order for the OpenEmbedded build system to create an image or any +target, it must be able to access source files. The `general workflow +figure <#general-workflow-figure>`__ represents source files using the +"Upstream Project Releases", "Local Projects", and "SCMs (optional)" +boxes. The figure represents mirrors, which also play a role in locating +source files, with the "Source Materials" box. + +The method by which source files are ultimately organized is a function +of the project. For example, for released software, projects tend to use +tarballs or other archived files that can capture the state of a release +guaranteeing that it is statically represented. On the other hand, for a +project that is more dynamic or experimental in nature, a project might +keep source files in a repository controlled by a Source Control Manager +(SCM) such as Git. Pulling source from a repository allows you to +control the point in the repository (the revision) from which you want +to build software. Finally, a combination of the two might exist, which +would give the consumer a choice when deciding where to get source +files. + +BitBake uses the :term:`SRC_URI` +variable to point to source files regardless of their location. Each +recipe must have a ``SRC_URI`` variable that points to the source. + +Another area that plays a significant role in where source files come +from is pointed to by the +:term:`DL_DIR` variable. This area is +a cache that can hold previously downloaded source. You can also +instruct the OpenEmbedded build system to create tarballs from Git +repositories, which is not the default behavior, and store them in the +``DL_DIR`` by using the +:term:`BB_GENERATE_MIRROR_TARBALLS` +variable. + +Judicious use of a ``DL_DIR`` directory can save the build system a trip +across the Internet when looking for files. A good method for using a +download directory is to have ``DL_DIR`` point to an area outside of +your Build Directory. Doing so allows you to safely delete the Build +Directory if needed without fear of removing any downloaded source file. + +The remainder of this section provides a deeper look into the source +files and the mirrors. Here is a more detailed look at the source file +area of the `general workflow figure <#general-workflow-figure>`__: + +.. image:: figures/source-input.png + :align: center + +Upstream Project Releases +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Upstream project releases exist anywhere in the form of an archived file +(e.g. tarball or zip file). These files correspond to individual +recipes. For example, the figure uses specific releases each for +BusyBox, Qt, and Dbus. An archive file can be for any released product +that can be built using a recipe. + +Local Projects +~~~~~~~~~~~~~~ + +Local projects are custom bits of software the user provides. These bits +reside somewhere local to a project - perhaps a directory into which the +user checks in items (e.g. a local directory containing a development +source tree used by the group). + +The canonical method through which to include a local project is to use +the :ref:`externalsrc <ref-classes-externalsrc>` +class to include that local project. You use either the ``local.conf`` +or a recipe's append file to override or set the recipe to point to the +local directory on your disk to pull in the whole source tree. + +.. _scms: + +Source Control Managers (Optional) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Another place from which the build system can get source files is with +:ref:`fetchers <bitbake:bb-fetchers>` employing various Source +Control Managers (SCMs) such as Git or Subversion. In such cases, a +repository is cloned or checked out. The +:ref:`ref-tasks-fetch` task inside +BitBake uses the :term:`SRC_URI` +variable and the argument's prefix to determine the correct fetcher +module. + +.. note:: + + For information on how to have the OpenEmbedded build system generate + tarballs for Git repositories and place them in the + DL_DIR + directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS` + variable in the Yocto Project Reference Manual. + +When fetching a repository, BitBake uses the +:term:`SRCREV` variable to determine +the specific revision from which to build. + +Source Mirror(s) +~~~~~~~~~~~~~~~~ + +Two kinds of mirrors exist: pre-mirrors and regular mirrors. The +:term:`PREMIRRORS` and +:term:`MIRRORS` variables point to +these, respectively. BitBake checks pre-mirrors before looking upstream +for any source files. Pre-mirrors are appropriate when you have a shared +directory that is not a directory defined by the +:term:`DL_DIR` variable. A Pre-mirror +typically points to a shared directory that is local to your +organization. + +Regular mirrors can be any site across the Internet that is used as an +alternative location for source code should the primary site not be +functioning for some reason or another. + +.. _package-feeds-dev-environment: + +Package Feeds +------------- + +When the OpenEmbedded build system generates an image or an SDK, it gets +the packages from a package feed area located in the +:term:`Build Directory`. The `general +workflow figure <#general-workflow-figure>`__ shows this package feeds +area in the upper-right corner. + +This section looks a little closer into the package feeds area used by +the build system. Here is a more detailed look at the area: + +.. image:: figures/package-feeds.png + :align: center + +Package feeds are an intermediary step in the build process. The +OpenEmbedded build system provides classes to generate different package +types, and you specify which classes to enable through the +:term:`PACKAGE_CLASSES` +variable. Before placing the packages into package feeds, the build +process validates them with generated output quality assurance checks +through the :ref:`insane <ref-classes-insane>` +class. + +The package feed area resides in the Build Directory. The directory the +build system uses to temporarily store packages is determined by a +combination of variables and the particular package manager in use. See +the "Package Feeds" box in the illustration and note the information to +the right of that area. In particular, the following defines where +package files are kept: + +- :term:`DEPLOY_DIR`: Defined as + ``tmp/deploy`` in the Build Directory. + +- ``DEPLOY_DIR_*``: Depending on the package manager used, the package + type sub-folder. Given RPM, IPK, or DEB packaging and tarball + creation, the + :term:`DEPLOY_DIR_RPM`, + :term:`DEPLOY_DIR_IPK`, + :term:`DEPLOY_DIR_DEB`, or + :term:`DEPLOY_DIR_TAR`, + variables are used, respectively. + +- :term:`PACKAGE_ARCH`: Defines + architecture-specific sub-folders. For example, packages could exist + for the i586 or qemux86 architectures. + +BitBake uses the +:ref:`do_package_write_* <ref-tasks-package_write_deb>` +tasks to generate packages and place them into the package holding area +(e.g. ``do_package_write_ipk`` for IPK packages). See the +":ref:`ref-tasks-package_write_deb`", +":ref:`ref-tasks-package_write_ipk`", +":ref:`ref-tasks-package_write_rpm`", +and +":ref:`ref-tasks-package_write_tar`" +sections in the Yocto Project Reference Manual for additional +information. As an example, consider a scenario where an IPK packaging +manager is being used and package architecture support for both i586 and +qemux86 exist. Packages for the i586 architecture are placed in +``build/tmp/deploy/ipk/i586``, while packages for the qemux86 +architecture are placed in ``build/tmp/deploy/ipk/qemux86``. + +.. _bitbake-dev-environment: + +BitBake Tool +------------ + +The OpenEmbedded build system uses +:term:`BitBake` to produce images and +Software Development Kits (SDKs). You can see from the `general workflow +figure <#general-workflow-figure>`__, the BitBake area consists of +several functional areas. This section takes a closer look at each of +those areas. + +.. note:: + + Separate documentation exists for the BitBake tool. See the + BitBake User Manual + for reference material on BitBake. + +.. _source-fetching-dev-environment: + +Source Fetching +~~~~~~~~~~~~~~~ + +The first stages of building a recipe are to fetch and unpack the source +code: + +.. image:: figures/source-fetching.png + :align: center + +The :ref:`ref-tasks-fetch` and +:ref:`ref-tasks-unpack` tasks fetch +the source files and unpack them into the +:term:`Build Directory`. + +.. note:: + + For every local file (e.g. + file:// + ) that is part of a recipe's + SRC_URI + statement, the OpenEmbedded build system takes a checksum of the file + for the recipe and inserts the checksum into the signature for the + do_fetch + task. If any local file has been modified, the + do_fetch + task and all tasks that depend on it are re-executed. + +By default, everything is accomplished in the Build Directory, which has +a defined structure. For additional general information on the Build +Directory, see the ":ref:`structure-core-build`" section in +the Yocto Project Reference Manual. + +Each recipe has an area in the Build Directory where the unpacked source +code resides. The :term:`S` variable points +to this area for a recipe's unpacked source code. The name of that +directory for any given recipe is defined from several different +variables. The preceding figure and the following list describe the +Build Directory's hierarchy: + +- :term:`TMPDIR`: The base directory + where the OpenEmbedded build system performs all its work during the + build. The default base directory is the ``tmp`` directory. + +- :term:`PACKAGE_ARCH`: The + architecture of the built package or packages. Depending on the + eventual destination of the package or packages (i.e. machine + architecture, :term:`Build Host`, SDK, or + specific machine), ``PACKAGE_ARCH`` varies. See the variable's + description for details. + +- :term:`TARGET_OS`: The operating + system of the target device. A typical value would be "linux" (e.g. + "qemux86-poky-linux"). + +- :term:`PN`: The name of the recipe used + to build the package. This variable can have multiple meanings. + However, when used in the context of input files, ``PN`` represents + the name of the recipe. + +- :term:`WORKDIR`: The location + where the OpenEmbedded build system builds a recipe (i.e. does the + work to create the package). + + - :term:`PV`: The version of the + recipe used to build the package. + + - :term:`PR`: The revision of the + recipe used to build the package. + +- :term:`S`: Contains the unpacked source + files for a given recipe. + + - :term:`BPN`: The name of the recipe + used to build the package. The ``BPN`` variable is a version of + the ``PN`` variable but with common prefixes and suffixes removed. + + - :term:`PV`: The version of the + recipe used to build the package. + +.. note:: + + In the previous figure, notice that two sample hierarchies exist: one + based on package architecture (i.e. + PACKAGE_ARCH + ) and one based on a machine (i.e. + MACHINE + ). The underlying structures are identical. The differentiator being + what the OpenEmbedded build system is using as a build target (e.g. + general architecture, a build host, an SDK, or a specific machine). + +.. _patching-dev-environment: + +Patching +~~~~~~~~ + +Once source code is fetched and unpacked, BitBake locates patch files +and applies them to the source files: + +.. image:: figures/patching.png + :align: center + +The :ref:`ref-tasks-patch` task uses a +recipe's :term:`SRC_URI` statements +and the :term:`FILESPATH` variable +to locate applicable patch files. + +Default processing for patch files assumes the files have either +``*.patch`` or ``*.diff`` file types. You can use ``SRC_URI`` parameters +to change the way the build system recognizes patch files. See the +:ref:`ref-tasks-patch` task for more +information. + +BitBake finds and applies multiple patches for a single recipe in the +order in which it locates the patches. The ``FILESPATH`` variable +defines the default set of directories that the build system uses to +search for patch files. Once found, patches are applied to the recipe's +source files, which are located in the +:term:`S` directory. + +For more information on how the source directories are created, see the +"`Source Fetching <#source-fetching-dev-environment>`__" section. For +more information on how to create patches and how the build system +processes patches, see the +":ref:`dev-manual/dev-manual-common-tasks:patching code`" +section in the +Yocto Project Development Tasks Manual. You can also see the +":ref:`sdk-manual/sdk-extensible:use \`\`devtool modify\`\` to modify the source of an existing component`" +section in the Yocto Project Application Development and the Extensible +Software Development Kit (SDK) manual and the +":ref:`kernel-dev/kernel-dev-common:using traditional kernel development to patch the kernel`" +section in the Yocto Project Linux Kernel Development Manual. + +.. _configuration-compilation-and-staging-dev-environment: + +Configuration, Compilation, and Staging +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After source code is patched, BitBake executes tasks that configure and +compile the source code. Once compilation occurs, the files are copied +to a holding area (staged) in preparation for packaging: + +.. image:: figures/configuration-compile-autoreconf.png + :align: center + +This step in the build process consists of the following tasks: + +- :ref:`ref-tasks-prepare_recipe_sysroot`: + This task sets up the two sysroots in + ``${``\ :term:`WORKDIR`\ ``}`` + (i.e. ``recipe-sysroot`` and ``recipe-sysroot-native``) so that + during the packaging phase the sysroots can contain the contents of + the + :ref:`ref-tasks-populate_sysroot` + tasks of the recipes on which the recipe containing the tasks + depends. A sysroot exists for both the target and for the native + binaries, which run on the host system. + +- *do_configure*: This task configures the source by enabling and + disabling any build-time and configuration options for the software + being built. Configurations can come from the recipe itself as well + as from an inherited class. Additionally, the software itself might + configure itself depending on the target for which it is being built. + + The configurations handled by the + :ref:`ref-tasks-configure` task + are specific to configurations for the source code being built by the + recipe. + + If you are using the + :ref:`autotools <ref-classes-autotools>` class, + you can add additional configuration options by using the + :term:`EXTRA_OECONF` or + :term:`PACKAGECONFIG_CONFARGS` + variables. For information on how this variable works within that + class, see the + :ref:`autotools <ref-classes-autotools>` class + :yocto_git:`here </cgit/cgit.cgi/poky/tree/meta/classes/autotools.bbclass>`. + +- *do_compile*: Once a configuration task has been satisfied, + BitBake compiles the source using the + :ref:`ref-tasks-compile` task. + Compilation occurs in the directory pointed to by the + :term:`B` variable. Realize that the + ``B`` directory is, by default, the same as the + :term:`S` directory. + +- *do_install*: After compilation completes, BitBake executes the + :ref:`ref-tasks-install` task. + This task copies files from the ``B`` directory and places them in a + holding area pointed to by the :term:`D` + variable. Packaging occurs later using files from this holding + directory. + +.. _package-splitting-dev-environment: + +Package Splitting +~~~~~~~~~~~~~~~~~ + +After source code is configured, compiled, and staged, the build system +analyzes the results and splits the output into packages: + +.. image:: figures/analysis-for-package-splitting.png + :align: center + +The :ref:`ref-tasks-package` and +:ref:`ref-tasks-packagedata` +tasks combine to analyze the files found in the +:term:`D` directory and split them into +subsets based on available packages and files. Analysis involves the +following as well as other items: splitting out debugging symbols, +looking at shared library dependencies between packages, and looking at +package relationships. + +The ``do_packagedata`` task creates package metadata based on the +analysis such that the build system can generate the final packages. The +:ref:`ref-tasks-populate_sysroot` +task stages (copies) a subset of the files installed by the +:ref:`ref-tasks-install` task into +the appropriate sysroot. Working, staged, and intermediate results of +the analysis and package splitting process use several areas: + +- :term:`PKGD`: The destination + directory (i.e. ``package``) for packages before they are split into + individual packages. + +- :term:`PKGDESTWORK`: A + temporary work area (i.e. ``pkgdata``) used by the ``do_package`` + task to save package metadata. + +- :term:`PKGDEST`: The parent + directory (i.e. ``packages-split``) for packages after they have been + split. + +- :term:`PKGDATA_DIR`: A shared, + global-state directory that holds packaging metadata generated during + the packaging process. The packaging process copies metadata from + ``PKGDESTWORK`` to the ``PKGDATA_DIR`` area where it becomes globally + available. + +- :term:`STAGING_DIR_HOST`: + The path for the sysroot for the system on which a component is built + to run (i.e. ``recipe-sysroot``). + +- :term:`STAGING_DIR_NATIVE`: + The path for the sysroot used when building components for the build + host (i.e. ``recipe-sysroot-native``). + +- :term:`STAGING_DIR_TARGET`: + The path for the sysroot used when a component that is built to + execute on a system and it generates code for yet another machine + (e.g. cross-canadian recipes). + +The :term:`FILES` variable defines the +files that go into each package in +:term:`PACKAGES`. If you want +details on how this is accomplished, you can look at +:yocto_git:`package.bbclass </cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass>`. + +Depending on the type of packages being created (RPM, DEB, or IPK), the +:ref:`do_package_write_* <ref-tasks-package_write_deb>` +task creates the actual packages and places them in the Package Feed +area, which is ``${TMPDIR}/deploy``. You can see the "`Package +Feeds <#package-feeds-dev-environment>`__" section for more detail on +that part of the build process. + +.. note:: + + Support for creating feeds directly from the + deploy/\* + directories does not exist. Creating such feeds usually requires some + kind of feed maintenance mechanism that would upload the new packages + into an official package feed (e.g. the Ångström distribution). This + functionality is highly distribution-specific and thus is not + provided out of the box. + +.. _image-generation-dev-environment: + +Image Generation +~~~~~~~~~~~~~~~~ + +Once packages are split and stored in the Package Feeds area, the build +system uses BitBake to generate the root filesystem image: + +.. image:: figures/image-generation.png + :align: center + +The image generation process consists of several stages and depends on +several tasks and variables. The +:ref:`ref-tasks-rootfs` task creates +the root filesystem (file and directory structure) for an image. This +task uses several key variables to help create the list of packages to +actually install: + +- :term:`IMAGE_INSTALL`: Lists + out the base set of packages from which to install from the Package + Feeds area. + +- :term:`PACKAGE_EXCLUDE`: + Specifies packages that should not be installed into the image. + +- :term:`IMAGE_FEATURES`: + Specifies features to include in the image. Most of these features + map to additional packages for installation. + +- :term:`PACKAGE_CLASSES`: + Specifies the package backend (e.g. RPM, DEB, or IPK) to use and + consequently helps determine where to locate packages within the + Package Feeds area. + +- :term:`IMAGE_LINGUAS`: + Determines the language(s) for which additional language support + packages are installed. + +- :term:`PACKAGE_INSTALL`: + The final list of packages passed to the package manager for + installation into the image. + +With :term:`IMAGE_ROOTFS` +pointing to the location of the filesystem under construction and the +``PACKAGE_INSTALL`` variable providing the final list of packages to +install, the root file system is created. + +Package installation is under control of the package manager (e.g. +dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package +management is enabled for the target. At the end of the process, if +package management is not enabled for the target, the package manager's +data files are deleted from the root filesystem. As part of the final +stage of package installation, post installation scripts that are part +of the packages are run. Any scripts that fail to run on the build host +are run on the target when the target system is first booted. If you are +using a +:ref:`read-only root filesystem <dev-manual/dev-manual-common-tasks:creating a read-only root filesystem>`, +all the post installation scripts must succeed on the build host during +the package installation phase since the root filesystem on the target +is read-only. + +The final stages of the ``do_rootfs`` task handle post processing. Post +processing includes creation of a manifest file and optimizations. + +The manifest file (``.manifest``) resides in the same directory as the +root filesystem image. This file lists out, line-by-line, the installed +packages. The manifest file is useful for the +:ref:`testimage <ref-classes-testimage*>` class, +for example, to determine whether or not to run specific tests. See the +:term:`IMAGE_MANIFEST` +variable for additional information. + +Optimizing processes that are run across the image include ``mklibs``, +``prelink``, and any other post-processing commands as defined by the +:term:`ROOTFS_POSTPROCESS_COMMAND` +variable. The ``mklibs`` process optimizes the size of the libraries, +while the ``prelink`` process optimizes the dynamic linking of shared +libraries to reduce start up time of executables. + +After the root filesystem is built, processing begins on the image +through the :ref:`ref-tasks-image` +task. The build system runs any pre-processing commands as defined by +the +:term:`IMAGE_PREPROCESS_COMMAND` +variable. This variable specifies a list of functions to call before the +build system creates the final image output files. + +The build system dynamically creates ``do_image_*`` tasks as needed, +based on the image types specified in the +:term:`IMAGE_FSTYPES` variable. +The process turns everything into an image file or a set of image files +and can compress the root filesystem image to reduce the overall size of +the image. The formats used for the root filesystem depend on the +``IMAGE_FSTYPES`` variable. Compression depends on whether the formats +support compression. + +As an example, a dynamically created task when creating a particular +image type would take the following form: +:: + + do_image_type + +So, if the type +as specified by the ``IMAGE_FSTYPES`` were ``ext4``, the dynamically +generated task would be as follows: +:: + + do_image_ext4 + +The final task involved in image creation is the +:ref:`do_image_complete <ref-tasks-image-complete>` +task. This task completes the image by applying any image post +processing as defined through the +:term:`IMAGE_POSTPROCESS_COMMAND` +variable. The variable specifies a list of functions to call once the +build system has created the final image output files. + +.. note:: + + The entire image generation process is run under + Pseudo. Running under Pseudo ensures that the files in the root filesystem + have correct ownership. + +.. _sdk-generation-dev-environment: + +SDK Generation +~~~~~~~~~~~~~~ + +The OpenEmbedded build system uses BitBake to generate the Software +Development Kit (SDK) installer scripts for both the standard SDK and +the extensible SDK (eSDK): + +.. image:: figures/sdk-generation.png + :align: center + +.. note:: + + For more information on the cross-development toolchain generation, + see the ":ref:`overview-manual/overview-manual-concepts:cross-development toolchain generation`" + section. For information on advantages gained when building a + cross-development toolchain using the do_populate_sdk task, see the + ":ref:`sdk-manual/sdk-appendix-obtain:building an sdk installer`" section in + the Yocto Project Application Development and the Extensible Software + Development Kit (eSDK) manual. + +Like image generation, the SDK script process consists of several stages +and depends on many variables. The +:ref:`ref-tasks-populate_sdk` +and +:ref:`ref-tasks-populate_sdk_ext` +tasks use these key variables to help create the list of packages to +actually install. For information on the variables listed in the figure, +see the "`Application Development SDK <#sdk-dev-environment>`__" +section. + +The ``do_populate_sdk`` task helps create the standard SDK and handles +two parts: a target part and a host part. The target part is the part +built for the target hardware and includes libraries and headers. The +host part is the part of the SDK that runs on the +:term:`SDKMACHINE`. + +The ``do_populate_sdk_ext`` task helps create the extensible SDK and +handles host and target parts differently than its counter part does for +the standard SDK. For the extensible SDK, the task encapsulates the +build system, which includes everything needed (host and target) for the +SDK. + +Regardless of the type of SDK being constructed, the tasks perform some +cleanup after which a cross-development environment setup script and any +needed configuration files are created. The final output is the +Cross-development toolchain installation script (``.sh`` file), which +includes the environment setup script. + +Stamp Files and the Rerunning of Tasks +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For each task that completes successfully, BitBake writes a stamp file +into the :term:`STAMPS_DIR` +directory. The beginning of the stamp file's filename is determined by +the :term:`STAMP` variable, and the end +of the name consists of the task's name and current `input +checksum <#overview-checksums>`__. + +.. note:: + + This naming scheme assumes that + BB_SIGNATURE_HANDLER + is "OEBasicHash", which is almost always the case in current + OpenEmbedded. + +To determine if a task needs to be rerun, BitBake checks if a stamp file +with a matching input checksum exists for the task. If such a stamp file +exists, the task's output is assumed to exist and still be valid. If the +file does not exist, the task is rerun. + +.. note:: + + The stamp mechanism is more general than the shared state (sstate) + cache mechanism described in the "`Setscene Tasks and Shared + State <#setscene-tasks-and-shared-state>`__" section. BitBake avoids + rerunning any task that has a valid stamp file, not just tasks that + can be accelerated through the sstate cache. + + However, you should realize that stamp files only serve as a marker + that some work has been done and that these files do not record task + output. The actual task output would usually be somewhere in + :term:`TMPDIR` (e.g. in some + recipe's :term:`WORKDIR`.) What + the sstate cache mechanism adds is a way to cache task output that + can then be shared between build machines. + +Since ``STAMPS_DIR`` is usually a subdirectory of ``TMPDIR``, removing +``TMPDIR`` will also remove ``STAMPS_DIR``, which means tasks will +properly be rerun to repopulate ``TMPDIR``. + +If you want some task to always be considered "out of date", you can +mark it with the :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>` +varflag. If some other task depends on such a task, then that task will +also always be considered out of date, which might not be what you want. + +For details on how to view information about a task's signature, see the +":ref:`dev-manual/dev-manual-common-tasks:viewing task variable dependencies`" +section in the Yocto Project Development Tasks Manual. + +Setscene Tasks and Shared State +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The description of tasks so far assumes that BitBake needs to build +everything and no available prebuilt objects exist. BitBake does support +skipping tasks if prebuilt objects are available. These objects are +usually made available in the form of a shared state (sstate) cache. + +.. note:: + + For information on variables affecting sstate, see the + :term:`SSTATE_DIR` + and + :term:`SSTATE_MIRRORS` + variables. + +The idea of a setscene task (i.e ``do_``\ taskname\ ``_setscene``) is a +version of the task where instead of building something, BitBake can +skip to the end result and simply place a set of files into specific +locations as needed. In some cases, it makes sense to have a setscene +task variant (e.g. generating package files in the +:ref:`do_package_write_* <ref-tasks-package_write_deb>` +task). In other cases, it does not make sense (e.g. a +:ref:`ref-tasks-patch` task or a +:ref:`ref-tasks-unpack` task) since +the work involved would be equal to or greater than the underlying task. + +In the build system, the common tasks that have setscene variants are +:ref:`ref-tasks-package`, +``do_package_write_*``, +:ref:`ref-tasks-deploy`, +:ref:`ref-tasks-packagedata`, and +:ref:`ref-tasks-populate_sysroot`. +Notice that these tasks represent most of the tasks whose output is an +end result. + +The build system has knowledge of the relationship between these tasks +and other preceding tasks. For example, if BitBake runs +``do_populate_sysroot_setscene`` for something, it does not make sense +to run any of the ``do_fetch``, ``do_unpack``, ``do_patch``, +``do_configure``, ``do_compile``, and ``do_install`` tasks. However, if +``do_package`` needs to be run, BitBake needs to run those other tasks. + +It becomes more complicated if everything can come from an sstate cache +because some objects are simply not required at all. For example, you do +not need a compiler or native tools, such as quilt, if nothing exists to +compile or patch. If the ``do_package_write_*`` packages are available +from sstate, BitBake does not need the ``do_package`` task data. + +To handle all these complexities, BitBake runs in two phases. The first +is the "setscene" stage. During this stage, BitBake first checks the +sstate cache for any targets it is planning to build. BitBake does a +fast check to see if the object exists rather than a complete download. +If nothing exists, the second phase, which is the setscene stage, +completes and the main build proceeds. + +If objects are found in the sstate cache, the build system works +backwards from the end targets specified by the user. For example, if an +image is being built, the build system first looks for the packages +needed for that image and the tools needed to construct an image. If +those are available, the compiler is not needed. Thus, the compiler is +not even downloaded. If something was found to be unavailable, or the +download or setscene task fails, the build system then tries to install +dependencies, such as the compiler, from the cache. + +The availability of objects in the sstate cache is handled by the +function specified by the +:term:`bitbake:BB_HASHCHECK_FUNCTION` +variable and returns a list of available objects. The function specified +by the +:term:`bitbake:BB_SETSCENE_DEPVALID` +variable is the function that determines whether a given dependency +needs to be followed, and whether for any given relationship the +function needs to be passed. The function returns a True or False value. + +.. _images-dev-environment: + +Images +------ + +The images produced by the build system are compressed forms of the root +filesystem and are ready to boot on a target device. You can see from +the `general workflow figure <#general-workflow-figure>`__ that BitBake +output, in part, consists of images. This section takes a closer look at +this output: + +.. image:: figures/images.png + :align: center + +.. note:: + + For a list of example images that the Yocto Project provides, see the + ":doc:`../ref-manual/ref-images`" chapter in the Yocto Project Reference + Manual. + +The build process writes images out to the :term:`Build Directory` +inside the +``tmp/deploy/images/machine/`` folder as shown in the figure. This +folder contains any files expected to be loaded on the target device. +The :term:`DEPLOY_DIR` variable +points to the ``deploy`` directory, while the +:term:`DEPLOY_DIR_IMAGE` +variable points to the appropriate directory containing images for the +current configuration. + +- kernel-image: A kernel binary file. The + :term:`KERNEL_IMAGETYPE` + variable determines the naming scheme for the kernel image file. + Depending on this variable, the file could begin with a variety of + naming strings. The ``deploy/images/``\ machine directory can contain + multiple image files for the machine. + +- root-filesystem-image: Root filesystems for the target device (e.g. + ``*.ext3`` or ``*.bz2`` files). The + :term:`IMAGE_FSTYPES` + variable determines the root filesystem image type. The + ``deploy/images/``\ machine directory can contain multiple root + filesystems for the machine. + +- kernel-modules: Tarballs that contain all the modules built for the + kernel. Kernel module tarballs exist for legacy purposes and can be + suppressed by setting the + :term:`MODULE_TARBALL_DEPLOY` + variable to "0". The ``deploy/images/``\ machine directory can + contain multiple kernel module tarballs for the machine. + +- bootloaders: If applicable to the target machine, bootloaders + supporting the image. The ``deploy/images/``\ machine directory can + contain multiple bootloaders for the machine. + +- symlinks: The ``deploy/images/``\ machine folder contains a symbolic + link that points to the most recently built file for each machine. + These links might be useful for external scripts that need to obtain + the latest version of each file. + +.. _sdk-dev-environment: + +Application Development SDK +--------------------------- + +In the `general workflow figure <#general-workflow-figure>`__, the +output labeled "Application Development SDK" represents an SDK. The SDK +generation process differs depending on whether you build an extensible +SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK +(e.g. ``bitbake -c populate_sdk`` imagename). This section takes a +closer look at this output: + +.. image:: figures/sdk.png + :align: center + +The specific form of this output is a set of files that includes a +self-extracting SDK installer (``*.sh``), host and target manifest +files, and files used for SDK testing. When the SDK installer file is +run, it installs the SDK. The SDK consists of a cross-development +toolchain, a set of libraries and headers, and an SDK environment setup +script. Running this installer essentially sets up your +cross-development environment. You can think of the cross-toolchain as +the "host" part because it runs on the SDK machine. You can think of the +libraries and headers as the "target" part because they are built for +the target hardware. The environment setup script is added so that you +can initialize the environment before using the tools. + +.. note:: + + - The Yocto Project supports several methods by which you can set up + this cross-development environment. These methods include + downloading pre-built SDK installers or building and installing + your own SDK installer. + + - For background information on cross-development toolchains in the + Yocto Project development environment, see the "`Cross-Development + Toolchain Generation <#cross-development-toolchain-generation>`__" + section. + + - For information on setting up a cross-development environment, see + the :doc:`../sdk-manual/sdk-manual` manual. + +All the output files for an SDK are written to the ``deploy/sdk`` folder +inside the :term:`Build Directory` as +shown in the previous figure. Depending on the type of SDK, several +variables exist that help configure these files. The following list +shows the variables associated with an extensible SDK: + +- :term:`DEPLOY_DIR`: Points to + the ``deploy`` directory. + +- :term:`SDK_EXT_TYPE`: + Controls whether or not shared state artifacts are copied into the + extensible SDK. By default, all required shared state artifacts are + copied into the SDK. + +- :term:`SDK_INCLUDE_PKGDATA`: + Specifies whether or not packagedata is included in the extensible + SDK for all recipes in the "world" target. + +- :term:`SDK_INCLUDE_TOOLCHAIN`: + Specifies whether or not the toolchain is included when building the + extensible SDK. + +- :term:`SDK_LOCAL_CONF_WHITELIST`: + A list of variables allowed through from the build system + configuration into the extensible SDK configuration. + +- :term:`SDK_LOCAL_CONF_BLACKLIST`: + A list of variables not allowed through from the build system + configuration into the extensible SDK configuration. + +- :term:`SDK_INHERIT_BLACKLIST`: + A list of classes to remove from the + :term:`INHERIT` value globally + within the extensible SDK configuration. + +This next list, shows the variables associated with a standard SDK: + +- :term:`DEPLOY_DIR`: Points to + the ``deploy`` directory. + +- :term:`SDKMACHINE`: Specifies + the architecture of the machine on which the cross-development tools + are run to create packages for the target hardware. + +- :term:`SDKIMAGE_FEATURES`: + Lists the features to include in the "target" part of the SDK. + +- :term:`TOOLCHAIN_HOST_TASK`: + Lists packages that make up the host part of the SDK (i.e. the part + that runs on the ``SDKMACHINE``). When you use + ``bitbake -c populate_sdk imagename`` to create the SDK, a set of + default packages apply. This variable allows you to add more + packages. + +- :term:`TOOLCHAIN_TARGET_TASK`: + Lists packages that make up the target part of the SDK (i.e. the part + built for the target hardware). + +- :term:`SDKPATH`: Defines the + default SDK installation path offered by the installation script. + +- :term:`SDK_HOST_MANIFEST`: + Lists all the installed packages that make up the host part of the + SDK. This variable also plays a minor role for extensible SDK + development as well. However, it is mainly used for the standard SDK. + +- :term:`SDK_TARGET_MANIFEST`: + Lists all the installed packages that make up the target part of the + SDK. This variable also plays a minor role for extensible SDK + development as well. However, it is mainly used for the standard SDK. + +Cross-Development Toolchain Generation +====================================== + +The Yocto Project does most of the work for you when it comes to +creating :ref:`sdk-manual/sdk-intro:the cross-development toolchain`. This +section provides some technical background on how cross-development +toolchains are created and used. For more information on toolchains, you +can also see the :doc:`../sdk-manual/sdk-manual` manual. + +In the Yocto Project development environment, cross-development +toolchains are used to build images and applications that run on the +target hardware. With just a few commands, the OpenEmbedded build system +creates these necessary toolchains for you. + +The following figure shows a high-level build environment regarding +toolchain construction and use. + +.. image:: figures/cross-development-toolchains.png + :align: center + +Most of the work occurs on the Build Host. This is the machine used to +build images and generally work within the the Yocto Project +environment. When you run +:term:`BitBake` to create an image, the +OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a +cross-compiler named ``gcc-cross``. The ``gcc-cross`` compiler is what +BitBake uses to compile source files when creating the target image. You +can think of ``gcc-cross`` simply as an automatically generated +cross-compiler that is used internally within BitBake only. + +.. note:: + + The extensible SDK does not use + gcc-cross-canadian + since this SDK ships a copy of the OpenEmbedded build system and the + sysroot within it contains + gcc-cross + . + +The chain of events that occurs when ``gcc-cross`` is bootstrapped is as +follows: +:: + + gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime + +- ``gcc``: The build host's GNU Compiler Collection (GCC). + +- ``binutils-cross``: The bare minimum binary utilities needed in order + to run the ``gcc-cross-initial`` phase of the bootstrap operation. + +- ``gcc-cross-initial``: An early stage of the bootstrap process for + creating the cross-compiler. This stage builds enough of the + ``gcc-cross``, the C library, and other pieces needed to finish + building the final cross-compiler in later stages. This tool is a + "native" package (i.e. it is designed to run on the build host). + +- ``linux-libc-headers``: Headers needed for the cross-compiler. + +- ``glibc-initial``: An initial version of the Embedded GNU C Library + (GLIBC) needed to bootstrap ``glibc``. + +- ``glibc``: The GNU C Library. + +- ``gcc-cross``: The final stage of the bootstrap process for the + cross-compiler. This stage results in the actual cross-compiler that + BitBake uses when it builds an image for a targeted device. + + .. note:: + + If you are replacing this cross compiler toolchain with a custom + version, you must replace + gcc-cross + . + + This tool is also a "native" package (i.e. it is designed to run on + the build host). + +- ``gcc-runtime``: Runtime libraries resulting from the toolchain + bootstrapping process. This tool produces a binary that consists of + the runtime libraries need for the targeted device. + +You can use the OpenEmbedded build system to build an installer for the +relocatable SDK used to develop applications. When you run the +installer, it installs the toolchain, which contains the development +tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and +other ``nativesdk-*`` tools), which are tools native to the SDK (i.e. +native to :term:`SDK_ARCH`), you +need to cross-compile and test your software. The figure shows the +commands you use to easily build out this toolchain. This +cross-development toolchain is built to execute on the +:term:`SDKMACHINE`, which might or +might not be the same machine as the Build Host. + +.. note:: + + If your target architecture is supported by the Yocto Project, you + can take advantage of pre-built images that ship with the Yocto + Project and already contain cross-development toolchain installers. + +Here is the bootstrap process for the relocatable toolchain: +:: + + gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian + +- ``gcc``: The build host's GNU Compiler Collection (GCC). + +- ``binutils-crosssdk``: The bare minimum binary utilities needed in + order to run the ``gcc-crosssdk-initial`` phase of the bootstrap + operation. + +- ``gcc-crosssdk-initial``: An early stage of the bootstrap process for + creating the cross-compiler. This stage builds enough of the + ``gcc-crosssdk`` and supporting pieces so that the final stage of the + bootstrap process can produce the finished cross-compiler. This tool + is a "native" binary that runs on the build host. + +- ``linux-libc-headers``: Headers needed for the cross-compiler. + +- ``glibc-initial``: An initial version of the Embedded GLIBC needed to + bootstrap ``nativesdk-glibc``. + +- ``nativesdk-glibc``: The Embedded GLIBC needed to bootstrap the + ``gcc-crosssdk``. + +- ``gcc-crosssdk``: The final stage of the bootstrap process for the + relocatable cross-compiler. The ``gcc-crosssdk`` is a transitory + compiler and never leaves the build host. Its purpose is to help in + the bootstrap process to create the eventual ``gcc-cross-canadian`` + compiler, which is relocatable. This tool is also a "native" package + (i.e. it is designed to run on the build host). + +- ``gcc-cross-canadian``: The final relocatable cross-compiler. When + run on the :term:`SDKMACHINE`, + this tool produces executable code that runs on the target device. + Only one cross-canadian compiler is produced per architecture since + they can be targeted at different processor optimizations using + configurations passed to the compiler through the compile commands. + This circumvents the need for multiple compilers and thus reduces the + size of the toolchains. + +.. note:: + + For information on advantages gained when building a + cross-development toolchain installer, see the + ":ref:`sdk-manual/sdk-appendix-obtain:building an sdk installer`" appendix + in the Yocto Project Application Development and the + Extensible Software Development Kit (eSDK) manual. + +Shared State Cache +================== + +By design, the OpenEmbedded build system builds everything from scratch +unless :term:`BitBake` can determine +that parts do not need to be rebuilt. Fundamentally, building from +scratch is attractive as it means all parts are built fresh and no +possibility of stale data exists that can cause problems. When +developers hit problems, they typically default back to building from +scratch so they have a know state from the start. + +Building an image from scratch is both an advantage and a disadvantage +to the process. As mentioned in the previous paragraph, building from +scratch ensures that everything is current and starts from a known +state. However, building from scratch also takes much longer as it +generally means rebuilding things that do not necessarily need to be +rebuilt. + +The Yocto Project implements shared state code that supports incremental +builds. The implementation of the shared state code answers the +following questions that were fundamental roadblocks within the +OpenEmbedded incremental build support system: + +- What pieces of the system have changed and what pieces have not + changed? + +- How are changed pieces of software removed and replaced? + +- How are pre-built components that do not need to be rebuilt from + scratch used when they are available? + +For the first question, the build system detects changes in the "inputs" +to a given task by creating a checksum (or signature) of the task's +inputs. If the checksum changes, the system assumes the inputs have +changed and the task needs to be rerun. For the second question, the +shared state (sstate) code tracks which tasks add which output to the +build process. This means the output from a given task can be removed, +upgraded or otherwise manipulated. The third question is partly +addressed by the solution for the second question assuming the build +system can fetch the sstate objects from remote locations and install +them if they are deemed to be valid. + +.. note:: + + - The build system does not maintain + :term:`PR` information as part of + the shared state packages. Consequently, considerations exist that + affect maintaining shared state feeds. For information on how the + build system works with packages and can track incrementing ``PR`` + information, see the ":ref:`dev-manual/dev-manual-common-tasks:automatically incrementing a package version number`" + section in the Yocto Project Development Tasks Manual. + + - The code in the build system that supports incremental builds is + not simple code. For techniques that help you work around issues + related to shared state code, see the + ":ref:`dev-manual/dev-manual-common-tasks:viewing metadata used to create the input signature of a shared state task`" + and + ":ref:`dev-manual/dev-manual-common-tasks:invalidating shared state to force a task to run`" + sections both in the Yocto Project Development Tasks Manual. + +The rest of this section goes into detail about the overall incremental +build architecture, the checksums (signatures), and shared state. + +.. _concepts-overall-architecture: + +Overall Architecture +-------------------- + +When determining what parts of the system need to be built, BitBake +works on a per-task basis rather than a per-recipe basis. You might +wonder why using a per-task basis is preferred over a per-recipe basis. +To help explain, consider having the IPK packaging backend enabled and +then switching to DEB. In this case, the +:ref:`ref-tasks-install` and +:ref:`ref-tasks-package` task outputs +are still valid. However, with a per-recipe approach, the build would +not include the ``.deb`` files. Consequently, you would have to +invalidate the whole build and rerun it. Rerunning everything is not the +best solution. Also, in this case, the core must be "taught" much about +specific tasks. This methodology does not scale well and does not allow +users to easily add new tasks in layers or as external recipes without +touching the packaged-staging core. + +.. _overview-checksums: + +Checksums (Signatures) +---------------------- + +The shared state code uses a checksum, which is a unique signature of a +task's inputs, to determine if a task needs to be run again. Because it +is a change in a task's inputs that triggers a rerun, the process needs +to detect all the inputs to a given task. For shell tasks, this turns +out to be fairly easy because the build process generates a "run" shell +script for each task and it is possible to create a checksum that gives +you a good idea of when the task's data changes. + +To complicate the problem, there are things that should not be included +in the checksum. First, there is the actual specific build path of a +given task - the :term:`WORKDIR`. It +does not matter if the work directory changes because it should not +affect the output for target packages. Also, the build process has the +objective of making native or cross packages relocatable. + +.. note:: + + Both native and cross packages run on the + build host. However, cross packages generate output for the target + architecture. + +The checksum therefore needs to exclude ``WORKDIR``. The simplistic +approach for excluding the work directory is to set ``WORKDIR`` to some +fixed value and create the checksum for the "run" script. + +Another problem results from the "run" scripts containing functions that +might or might not get called. The incremental build solution contains +code that figures out dependencies between shell functions. This code is +used to prune the "run" scripts down to the minimum set, thereby +alleviating this problem and making the "run" scripts much more readable +as a bonus. + +So far, solutions for shell scripts exist. What about Python tasks? The +same approach applies even though these tasks are more difficult. The +process needs to figure out what variables a Python function accesses +and what functions it calls. Again, the incremental build solution +contains code that first figures out the variable and function +dependencies, and then creates a checksum for the data used as the input +to the task. + +Like the ``WORKDIR`` case, situations exist where dependencies should be +ignored. For these situations, you can instruct the build process to +ignore a dependency by using a line like the following: +:: + + PACKAGE_ARCHS[vardepsexclude] = "MACHINE" + +This example ensures that the :term:`PACKAGE_ARCHS` variable +does not depend on the value of :term:`MACHINE`, even if it does +reference it. + +Equally, there are cases where you need to add dependencies BitBake is +not able to find. You can accomplish this by using a line like the +following: +:: + + PACKAGE_ARCHS[vardeps] = "MACHINE" + +This example explicitly +adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``. + +As an example, consider a case with in-line Python where BitBake is not +able to figure out dependencies. When running in debug mode (i.e. using +``-DDD``), BitBake produces output when it discovers something for which +it cannot figure out dependencies. The Yocto Project team has currently +not managed to cover those dependencies in detail and is aware of the +need to fix this situation. + +Thus far, this section has limited discussion to the direct inputs into +a task. Information based on direct inputs is referred to as the +"basehash" in the code. However, the question of a task's indirect +inputs still exits - items already built and present in the +:term:`Build Directory`. The checksum (or +signature) for a particular task needs to add the hashes of all the +tasks on which the particular task depends. Choosing which dependencies +to add is a policy decision. However, the effect is to generate a master +checksum that combines the basehash and the hashes of the task's +dependencies. + +At the code level, a variety of ways exist by which both the basehash +and the dependent task hashes can be influenced. Within the BitBake +configuration file, you can give BitBake some extra information to help +it construct the basehash. The following statement effectively results +in a list of global variable dependency excludes (i.e. variables never +included in any checksum): +:: + + BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\ + SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\ + USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\ + PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\ + CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" + +The +previous example excludes +:term:`WORKDIR` since that variable +is actually constructed as a path within +:term:`TMPDIR`, which is on the +whitelist. + +The rules for deciding which hashes of dependent tasks to include +through dependency chains are more complex and are generally +accomplished with a Python function. The code in +``meta/lib/oe/sstatesig.py`` shows two examples of this and also +illustrates how you can insert your own policy into the system if so +desired. This file defines the two basic signature generators +:term:`OpenEmbedded-Core (OE-Core)` uses: "OEBasic" and +"OEBasicHash". By default, a dummy "noop" signature handler is enabled +in BitBake. This means that behavior is unchanged from previous +versions. OE-Core uses the "OEBasicHash" signature handler by default +through this setting in the ``bitbake.conf`` file: +:: + + BB_SIGNATURE_HANDLER ?= "OEBasicHash" + +The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same +as the "OEBasic" version but adds the task hash to the `stamp +files <#stamp-files-and-the-rerunning-of-tasks>`__. This results in any +metadata change that changes the task hash, automatically causing the +task to be run again. This removes the need to bump +:term:`PR` values, and changes to metadata +automatically ripple across the build. + +It is also worth noting that the end result of these signature +generators is to make some dependency and hash information available to +the build. This information includes: + +- ``BB_BASEHASH_task-``\ taskname: The base hashes for each task in the + recipe. + +- ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each + dependent task. + +- ``BBHASHDEPS_``\ filename\ ``:``\ taskname: The task dependencies for + each task. + +- ``BB_TASKHASH``: The hash of the currently running task. + +Shared State +------------ + +Checksums and dependencies, as discussed in the previous section, solve +half the problem of supporting a shared state. The other half of the +problem is being able to use checksum information during the build and +being able to reuse or rebuild specific components. + +The :ref:`sstate <ref-classes-sstate>` class is a +relatively generic implementation of how to "capture" a snapshot of a +given task. The idea is that the build process does not care about the +source of a task's output. Output could be freshly built or it could be +downloaded and unpacked from somewhere. In other words, the build +process does not need to worry about its origin. + +Two types of output exist. One type is just about creating a directory +in :term:`WORKDIR`. A good example is +the output of either +:ref:`ref-tasks-install` or +:ref:`ref-tasks-package`. The other +type of output occurs when a set of data is merged into a shared +directory tree such as the sysroot. + +The Yocto Project team has tried to keep the details of the +implementation hidden in ``sstate`` class. From a user's perspective, +adding shared state wrapping to a task is as simple as this +:ref:`ref-tasks-deploy` example taken +from the :ref:`deploy <ref-classes-deploy>` class: +:: + + DEPLOYDIR = "${WORKDIR}/deploy-${PN}" + SSTATETASKS += "do_deploy" + do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" + do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" + + python do_deploy_setscene () { + sstate_setscene(d) + } + addtask do_deploy_setscene + do_deploy[dirs] = "${DEPLOYDIR} ${B}" + do_deploy[stamp-extra-info] = "${MACHINE_ARCH}" + +The following list explains the previous example: + +- Adding "do_deploy" to ``SSTATETASKS`` adds some required + sstate-related processing, which is implemented in the + :ref:`sstate <ref-classes-sstate>` class, to + before and after the + :ref:`ref-tasks-deploy` task. + +- The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that + ``do_deploy`` places its output in ``${DEPLOYDIR}`` when run normally + (i.e. when not using the sstate cache). This output becomes the input + to the shared state cache. + +- The ``do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"`` line + causes the contents of the shared state cache to be copied to + ``${DEPLOY_DIR_IMAGE}``. + + .. note:: + + If ``do_deploy`` is not already in the shared state cache or if its input + checksum (signature) has changed from when the output was cached, the task + runs to populate the shared state cache, after which the contents of the + shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If + ``do_deploy`` is in the shared state cache and its signature indicates + that the cached output is still valid (i.e. if no relevant task inputs + have changed), then the contents of the shared state cache copies + directly to ${``DEPLOY_DIR_IMAGE``} by the ``do_deploy_setscene`` task + instead, skipping the ``do_deploy`` task. + +- The following task definition is glue logic needed to make the + previous settings effective: + :: + + python do_deploy_setscene () { + sstate_setscene(d) + } + addtask do_deploy_setscene + + ``sstate_setscene()`` takes the flags above as input and accelerates the ``do_deploy`` task + through the shared state cache if possible. If the task was + accelerated, ``sstate_setscene()`` returns True. Otherwise, it + returns False, and the normal ``do_deploy`` task runs. For more + information, see the ":ref:`setscene <bitbake:bitbake-user-manual/bitbake-user-manual-execution:setscene>`" + section in the BitBake User Manual. + +- The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates + ``${DEPLOYDIR}`` and ``${B}`` before the ``do_deploy`` task runs, and + also sets the current working directory of ``do_deploy`` to ``${B}``. + For more information, see the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" + section in the BitBake + User Manual. + + .. note:: + + In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be + the same, you can use ``sstate-plaindirs``. For example, to preserve the + ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the ``do_package`` + task, use the following: + :: + + do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" + + +- The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends + extra metadata to the `stamp + file <#stamp-files-and-the-rerunning-of-tasks>`__. In this case, the + metadata makes the task specific to a machine's architecture. See + ":ref:`bitbake:ref-bitbake-tasklist`" + section in the BitBake User Manual for more information on the + ``stamp-extra-info`` flag. + +- ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with + multiple directories. For example, the following declares + ``PKGDESTWORK`` and ``SHLIBWORK`` as shared state input directories, + which populates the shared state cache, and ``PKGDATA_DIR`` and + ``SHLIBSDIR`` as the corresponding shared state output directories: + :: + + do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" + do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" + +- These methods also include the ability to take a lockfile when + manipulating shared state directory structures, for cases where file + additions or removals are sensitive: + :: + + do_package[sstate-lockfile] = "${PACKAGELOCK}" + +Behind the scenes, the shared state code works by looking in +:term:`SSTATE_DIR` and +:term:`SSTATE_MIRRORS` for +shared state files. Here is an example: +:: + + SSTATE_MIRRORS ?= "\ + file://.\* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ + file://.\* file:///some/local/dir/sstate/PATH" + +.. note:: + + The shared state directory (``SSTATE_DIR``) is organized into two-character + subdirectories, where the subdirectory names are based on the first two + characters of the hash. + If the shared state directory structure for a mirror has the same structure + as ``SSTATE_DIR``, you must specify "PATH" as part of the URI to enable the build + system to map to the appropriate subdirectory. + +The shared state package validity can be detected just by looking at the +filename since the filename contains the task checksum (or signature) as +described earlier in this section. If a valid shared state package is +found, the build process downloads it and uses it to accelerate the +task. + +The build processes use the ``*_setscene`` tasks for the task +acceleration phase. BitBake goes through this phase before the main +execution code and tries to accelerate any tasks for which it can find +shared state packages. If a shared state package for a task is +available, the shared state package is used. This means the task and any +tasks on which it is dependent are not executed. + +As a real world example, the aim is when building an IPK-based image, +only the +:ref:`ref-tasks-package_write_ipk` +tasks would have their shared state packages fetched and extracted. +Since the sysroot is not used, it would never get extracted. This is +another reason why a task-based approach is preferred over a +recipe-based approach, which would have to install the output from every +task. + +Automatically Added Runtime Dependencies +======================================== + +The OpenEmbedded build system automatically adds common types of runtime +dependencies between packages, which means that you do not need to +explicitly declare the packages using +:term:`RDEPENDS`. Three automatic +mechanisms exist (``shlibdeps``, ``pcdeps``, and ``depchains``) that +handle shared libraries, package configuration (pkg-config) modules, and +``-dev`` and ``-dbg`` packages, respectively. For other types of runtime +dependencies, you must manually declare the dependencies. + +- ``shlibdeps``: During the + :ref:`ref-tasks-package` task of + each recipe, all shared libraries installed by the recipe are + located. For each shared library, the package that contains the + shared library is registered as providing the shared library. More + specifically, the package is registered as providing the + `soname <https://en.wikipedia.org/wiki/Soname>`__ of the library. The + resulting shared-library-to-package mapping is saved globally in + :term:`PKGDATA_DIR` by the + :ref:`ref-tasks-packagedata` + task. + + Simultaneously, all executables and shared libraries installed by the + recipe are inspected to see what shared libraries they link against. + For each shared library dependency that is found, ``PKGDATA_DIR`` is + queried to see if some package (likely from a different recipe) + contains the shared library. If such a package is found, a runtime + dependency is added from the package that depends on the shared + library to the package that contains the library. + + The automatically added runtime dependency also includes a version + restriction. This version restriction specifies that at least the + current version of the package that provides the shared library must + be used, as if "package (>= version)" had been added to ``RDEPENDS``. + This forces an upgrade of the package containing the shared library + when installing the package that depends on the library, if needed. + + If you want to avoid a package being registered as providing a + particular shared library (e.g. because the library is for internal + use only), then add the library to + :term:`PRIVATE_LIBS` inside + the package's recipe. + +- ``pcdeps``: During the ``do_package`` task of each recipe, all + pkg-config modules (``*.pc`` files) installed by the recipe are + located. For each module, the package that contains the module is + registered as providing the module. The resulting module-to-package + mapping is saved globally in ``PKGDATA_DIR`` by the + ``do_packagedata`` task. + + Simultaneously, all pkg-config modules installed by the recipe are + inspected to see what other pkg-config modules they depend on. A + module is seen as depending on another module if it contains a + "Requires:" line that specifies the other module. For each module + dependency, ``PKGDATA_DIR`` is queried to see if some package + contains the module. If such a package is found, a runtime dependency + is added from the package that depends on the module to the package + that contains the module. + + .. note:: + + The + pcdeps + mechanism most often infers dependencies between + -dev + packages. + +- ``depchains``: If a package ``foo`` depends on a package ``bar``, + then ``foo-dev`` and ``foo-dbg`` are also made to depend on + ``bar-dev`` and ``bar-dbg``, respectively. Taking the ``-dev`` + packages as an example, the ``bar-dev`` package might provide headers + and shared library symlinks needed by ``foo-dev``, which shows the + need for a dependency between the packages. + + The dependencies added by ``depchains`` are in the form of + :term:`RRECOMMENDS`. + + .. note:: + + By default, ``foo-dev`` also has an ``RDEPENDS``-style dependency on + ``foo``, because the default value of ``RDEPENDS_${PN}-dev`` (set in + bitbake.conf) includes "${PN}". + + To ensure that the dependency chain is never broken, ``-dev`` and + ``-dbg`` packages are always generated by default, even if the + packages turn out to be empty. See the + :term:`ALLOW_EMPTY` variable + for more information. + +The ``do_package`` task depends on the ``do_packagedata`` task of each +recipe in :term:`DEPENDS` through use +of a ``[``\ :ref:`deptask <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]`` +declaration, which guarantees that the required +shared-library/module-to-package mapping information will be available +when needed as long as ``DEPENDS`` has been correctly set. + +Fakeroot and Pseudo +=================== + +Some tasks are easier to implement when allowed to perform certain +operations that are normally reserved for the root user (e.g. +:ref:`ref-tasks-install`, +:ref:`do_package_write* <ref-tasks-package_write_deb>`, +:ref:`ref-tasks-rootfs`, and +:ref:`do_image* <ref-tasks-image>`). For example, +the ``do_install`` task benefits from being able to set the UID and GID +of installed files to arbitrary values. + +One approach to allowing tasks to perform root-only operations would be +to require :term:`BitBake` to run as +root. However, this method is cumbersome and has security issues. The +approach that is actually used is to run tasks that benefit from root +privileges in a "fake" root environment. Within this environment, the +task and its child processes believe that they are running as the root +user, and see an internally consistent view of the filesystem. As long +as generating the final output (e.g. a package or an image) does not +require root privileges, the fact that some earlier steps ran in a fake +root environment does not cause problems. + +The capability to run tasks in a fake root environment is known as +"`fakeroot <http://man.he.net/man1/fakeroot>`__", which is derived from +the BitBake keyword/variable flag that requests a fake root environment +for a task. + +In the :term:`OpenEmbedded Build System`, +the program that +implements fakeroot is known as +`Pseudo <https://www.yoctoproject.org/software-item/pseudo/>`__. Pseudo +overrides system calls by using the environment variable ``LD_PRELOAD``, +which results in the illusion of running as root. To keep track of +"fake" file ownership and permissions resulting from operations that +require root permissions, Pseudo uses an SQLite 3 database. This +database is stored in +``${``\ :term:`WORKDIR`\ ``}/pseudo/files.db`` +for individual recipes. Storing the database in a file as opposed to in +memory gives persistence between tasks and builds, which is not +accomplished using fakeroot. + +.. note:: + + If you add your own task that manipulates the same files or + directories as a fakeroot task, then that task also needs to run + under fakeroot. Otherwise, the task cannot run root-only operations, + and cannot see the fake file ownership and permissions set by the + other task. You need to also add a dependency on + virtual/fakeroot-native:do_populate_sysroot + , giving the following: + :: + + fakeroot do_mytask () { + ... + } + do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot" + + +For more information, see the +:term:`FAKEROOT* <bitbake:FAKEROOT>` variables in the +BitBake User Manual. You can also reference the "`Why Not +Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__" +article for background information on Fakeroot and Pseudo. diff --git a/poky/documentation/overview-manual/overview-manual-concepts.xml b/poky/documentation/overview-manual/overview-manual-concepts.xml deleted file mode 100644 index f085dd710d..0000000000 --- a/poky/documentation/overview-manual/overview-manual-concepts.xml +++ /dev/null @@ -1,3234 +0,0 @@ -<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" -"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" -[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > - -<chapter id=' overview-manual-concepts'> -<title>Yocto Project Concepts</title> - - <para> - This chapter provides explanations for Yocto Project concepts that - go beyond the surface of "how-to" information and reference (or - look-up) material. - Concepts such as components, the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> - workflow, cross-development toolchains, shared state cache, and so - forth are explained. - </para> - - <section id='yocto-project-components'> - <title>Yocto Project Components</title> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> - task executor together with various types of configuration files - form the - <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>. - This section overviews these components by describing their use and - how they interact. - </para> - - <para> - BitBake handles the parsing and execution of the data files. - The data itself is of various types: - <itemizedlist> - <listitem><para> - <emphasis>Recipes:</emphasis> - Provides details about particular pieces of software. - </para></listitem> - <listitem><para> - <emphasis>Class Data:</emphasis> - Abstracts common build information (e.g. how to build a - Linux kernel). - </para></listitem> - <listitem><para> - <emphasis>Configuration Data:</emphasis> - Defines machine-specific settings, policy decisions, and - so forth. - Configuration data acts as the glue to bind everything - together. - </para></listitem> - </itemizedlist> - </para> - - <para> - BitBake knows how to combine multiple data sources together and - refers to each data source as a layer. - For information on layers, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>" - section of the Yocto Project Development Tasks Manual. - </para> - - <para> - Following are some brief details on these core components. - For additional information on how these components interact during - a build, see the - "<link linkend='openembedded-build-system-build-concepts'>OpenEmbedded Build System Concepts</link>" - section. - </para> - - <section id='usingpoky-components-bitbake'> - <title>BitBake</title> - - <para> - BitBake is the tool at the heart of the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> - and is responsible for parsing the - <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>, - generating a list of tasks from it, and then executing those - tasks. - </para> - - <para> - This section briefly introduces BitBake. - If you want more information on BitBake, see the - <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>. - </para> - - <para> - To see a list of the options BitBake supports, use either of - the following commands: - <literallayout class='monospaced'> - $ bitbake -h - $ bitbake --help - </literallayout> - </para> - - <para> - The most common usage for BitBake is - <filename>bitbake <replaceable>packagename</replaceable></filename>, - where <filename>packagename</filename> is the name of the - package you want to build (referred to as the "target"). - The target often equates to the first part of a recipe's - filename (e.g. "foo" for a recipe named - <filename>foo_1.3.0-r0.bb</filename>). - So, to process the - <filename>matchbox-desktop_1.2.3.bb</filename> recipe file, you - might type the following: - <literallayout class='monospaced'> - $ bitbake matchbox-desktop - </literallayout> - Several different versions of - <filename>matchbox-desktop</filename> might exist. - BitBake chooses the one selected by the distribution - configuration. - You can get more details about how BitBake chooses between - different target versions and providers in the - "<ulink url='&YOCTO_DOCS_BB_URL;#bb-bitbake-preferences'>Preferences</ulink>" - section of the BitBake User Manual. - </para> - - <para> - BitBake also tries to execute any dependent tasks first. - So for example, before building - <filename>matchbox-desktop</filename>, BitBake would build a - cross compiler and <filename>glibc</filename> if they had not - already been built. - </para> - - <para> - A useful BitBake option to consider is the - <filename>-k</filename> or <filename>--continue</filename> - option. - This option instructs BitBake to try and continue processing - the job as long as possible even after encountering an error. - When an error occurs, the target that failed and those that - depend on it cannot be remade. - However, when you use this option other dependencies can - still be processed. - </para> - </section> - - <section id='overview-components-recipes'> - <title>Recipes</title> - - <para> - Files that have the <filename>.bb</filename> suffix are - "recipes" files. - In general, a recipe contains information about a single piece - of software. - This information includes the location from which to download - the unaltered source, any source patches to be applied to that - source (if needed), which special configuration options to - apply, how to compile the source files, and how to package the - compiled output. - </para> - - <para> - The term "package" is sometimes used to refer to recipes. - However, since the word "package" is used for the packaged - output from the OpenEmbedded build system (i.e. - <filename>.ipk</filename> or <filename>.deb</filename> files), - this document avoids using the term "package" when referring - to recipes. - </para> - </section> - - <section id='overview-components-classes'> - <title>Classes</title> - - <para> - Class files (<filename>.bbclass</filename>) contain information - that is useful to share between recipes files. - An example is the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> - class, which contains common settings for any application that - Autotools uses. - The - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>" - chapter in the Yocto Project Reference Manual provides - details about classes and how to use them. - </para> - </section> - - <section id='overview-components-configurations'> - <title>Configurations</title> - - <para> - The configuration files (<filename>.conf</filename>) define - various configuration variables that govern the OpenEmbedded - build process. - These files fall into several areas that define machine - configuration options, distribution configuration options, - compiler tuning options, general common configuration options, - and user configuration options in - <filename>conf/local.conf</filename>, which is found in the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. - </para> - </section> - </section> - - <section id='overview-layers'> - <title>Layers</title> - - <para> - Layers are repositories that contain related metadata (i.e. - sets of instructions) that tell the OpenEmbedded build system how - to build a target. - Yocto Project's - <link linkend='the-yocto-project-layer-model'>layer model</link> - facilitates collaboration, sharing, customization, and reuse - within the Yocto Project development environment. - Layers logically separate information for your project. - For example, you can use a layer to hold all the configurations - for a particular piece of hardware. - Isolating hardware-specific configurations allows you to share - other metadata by using a different layer where that metadata - might be common across several pieces of hardware. - </para> - - <para> - Many layers exist that work in the Yocto Project development - environment. - The - <ulink url='https://caffelli-staging.yoctoproject.org/software-overview/layers/'>Yocto Project Curated Layer Index</ulink> - and - <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink> - both contain layers from which you can use or leverage. - </para> - - <para> - By convention, layers in the Yocto Project follow a specific form. - Conforming to a known structure allows BitBake to make assumptions - during builds on where to find types of metadata. - You can find procedures and learn about tools (i.e. - <filename>bitbake-layers</filename>) for creating layers suitable - for the Yocto Project in the - "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>" - section of the Yocto Project Development Tasks Manual. - </para> - </section> - - <section id="openembedded-build-system-build-concepts"> - <title>OpenEmbedded Build System Concepts</title> - - <para> - This section takes a more detailed look inside the build - process used by the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>, - which is the build system specific to the Yocto Project. - At the heart of the build system is BitBake, the task executor. - </para> - - <para> - The following diagram represents the high-level workflow of a - build. - The remainder of this section expands on the fundamental input, - output, process, and metadata logical blocks that make up the - workflow. - </para> - - <para id='general-workflow-figure'> - <imagedata fileref="figures/YP-flow-diagram.png" format="PNG" align='center' width="8in"/> - </para> - - <para> - In general, the build's workflow consists of several functional - areas: - <itemizedlist> - <listitem><para> - <emphasis>User Configuration:</emphasis> - metadata you can use to control the build process. - </para></listitem> - <listitem><para> - <emphasis>Metadata Layers:</emphasis> - Various layers that provide software, machine, and - distro metadata. - </para></listitem> - <listitem><para> - <emphasis>Source Files:</emphasis> - Upstream releases, local projects, and SCMs. - </para></listitem> - <listitem><para> - <emphasis>Build System:</emphasis> - Processes under the control of - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>. - This block expands on how BitBake fetches source, applies - patches, completes compilation, analyzes output for package - generation, creates and tests packages, generates images, - and generates cross-development tools. - </para></listitem> - <listitem><para> - <emphasis>Package Feeds:</emphasis> - Directories containing output packages (RPM, DEB or IPK), - which are subsequently used in the construction of an - image or Software Development Kit (SDK), produced by the - build system. - These feeds can also be copied and shared using a web - server or other means to facilitate extending or updating - existing images on devices at runtime if runtime package - management is enabled. - </para></listitem> - <listitem><para> - <emphasis>Images:</emphasis> - Images produced by the workflow. - </para></listitem> - <listitem><para> - <emphasis>Application Development SDK:</emphasis> - Cross-development tools that are produced along with - an image or separately with BitBake. - </para></listitem> - </itemizedlist> - </para> - - <section id="user-configuration"> - <title>User Configuration</title> - - <para> - User configuration helps define the build. - Through user configuration, you can tell BitBake the - target architecture for which you are building the image, - where to store downloaded source, and other build properties. - </para> - - <para> - The following figure shows an expanded representation of the - "User Configuration" box of the - <link linkend='general-workflow-figure'>general workflow figure</link>: - </para> - - <para> - <imagedata fileref="figures/user-configuration.png" align="center" width="8in" depth="4.5in" /> - </para> - - <para> - BitBake needs some basic configuration files in order to - complete a build. - These files are <filename>*.conf</filename> files. - The minimally necessary ones reside as example files in the - <filename>build/conf</filename> directory of the - <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. - For simplicity, this section refers to the Source Directory as - the "Poky Directory." - </para> - - <para> - When you clone the - <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink> - Git repository or you download and unpack a Yocto Project - release, you can set up the Source Directory to be named - anything you want. - For this discussion, the cloned repository uses the default - name <filename>poky</filename>. - <note> - The Poky repository is primarily an aggregation of existing - repositories. - It is not a canonical upstream source. - </note> - </para> - - <para> - The <filename>meta-poky</filename> layer inside Poky contains - a <filename>conf</filename> directory that has example - configuration files. - These example files are used as a basis for creating actual - configuration files when you source - <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>, - which is the build environment script. - </para> - - <para> - Sourcing the build environment script creates a - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> - if one does not already exist. - BitBake uses the Build Directory for all its work during - builds. - The Build Directory has a <filename>conf</filename> directory - that contains default versions of your - <filename>local.conf</filename> and - <filename>bblayers.conf</filename> configuration files. - These default configuration files are created only if versions - do not already exist in the Build Directory at the time you - source the build environment setup script. - </para> - - <para> - Because the Poky repository is fundamentally an aggregation of - existing repositories, some users might be familiar with - running the <filename>&OE_INIT_FILE;</filename> script - in the context of separate - <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink> - and BitBake repositories rather than a single Poky repository. - This discussion assumes the script is executed from - within a cloned or unpacked version of Poky. - </para> - - <para> - Depending on where the script is sourced, different - sub-scripts are called to set up the Build Directory - (Yocto or OpenEmbedded). - Specifically, the script - <filename>scripts/oe-setup-builddir</filename> inside the - poky directory sets up the Build Directory and seeds the - directory (if necessary) with configuration files appropriate - for the Yocto Project development environment. - <note> - The <filename>scripts/oe-setup-builddir</filename> script - uses the <filename>$TEMPLATECONF</filename> variable to - determine which sample configuration files to locate. - </note> - </para> - - <para> - The <filename>local.conf</filename> file provides many - basic variables that define a build environment. - Here is a list of a few. - To see the default configurations in a - <filename>local.conf</filename> file created by the build - environment script, see the - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/local.conf.sample'><filename>local.conf.sample</filename></ulink> - in the <filename>meta-poky</filename> layer: - <itemizedlist> - <listitem><para> - <emphasis>Target Machine Selection:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Download Directory:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Shared State Directory:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Build Output:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Distribution Policy:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Packaging Format:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>SDK Target Architecture:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink> - variable. - </para></listitem> - <listitem><para> - <emphasis>Extra Image Packages:</emphasis> - Controlled by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink> - variable. - </para></listitem> - </itemizedlist> - <note> - Configurations set in the - <filename>conf/local.conf</filename> file can also be set - in the <filename>conf/site.conf</filename> and - <filename>conf/auto.conf</filename> configuration files. - </note> - </para> - - <para> - The <filename>bblayers.conf</filename> file tells BitBake what - layers you want considered during the build. - By default, the layers listed in this file include layers - minimally needed by the build system. - However, you must manually add any custom layers you have - created. - You can find more information on working with the - <filename>bblayers.conf</filename> file in the - "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>" - section in the Yocto Project Development Tasks Manual. - </para> - - <para> - The files <filename>site.conf</filename> and - <filename>auto.conf</filename> are not created by the - environment initialization script. - If you want the <filename>site.conf</filename> file, you - need to create that yourself. - The <filename>auto.conf</filename> file is typically created by - an autobuilder: - <itemizedlist> - <listitem><para> - <emphasis><filename>site.conf</filename>:</emphasis> - You can use the <filename>conf/site.conf</filename> - configuration file to configure multiple - build directories. - For example, suppose you had several build environments - and they shared some common features. - You can set these default build properties here. - A good example is perhaps the packaging format to use - through the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> - variable.</para> - - <para>One useful scenario for using the - <filename>conf/site.conf</filename> file is to extend - your - <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink> - variable to include the path to a - <filename>conf/site.conf</filename>. - Then, when BitBake looks for Metadata using - <filename>BBPATH</filename>, it finds the - <filename>conf/site.conf</filename> file and applies - your common configurations found in the file. - To override configurations in a particular build - directory, alter the similar configurations within - that build directory's - <filename>conf/local.conf</filename> file. - </para></listitem> - <listitem><para> - <emphasis><filename>auto.conf</filename>:</emphasis> - The file is usually created and written to by - an autobuilder. - The settings put into the file are typically the - same as you would find in the - <filename>conf/local.conf</filename> or the - <filename>conf/site.conf</filename> files. - </para></listitem> - </itemizedlist> - </para> - - <para> - You can edit all configuration files to further define - any particular build environment. - This process is represented by the "User Configuration Edits" - box in the figure. - </para> - - <para> - When you launch your build with the - <filename>bitbake <replaceable>target</replaceable></filename> - command, BitBake sorts out the configurations to ultimately - define your build environment. - It is important to understand that the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> - reads the configuration files in a specific order: - <filename>site.conf</filename>, <filename>auto.conf</filename>, - and <filename>local.conf</filename>. - And, the build system applies the normal assignment statement - rules as described in the - "<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>" - chapter of the BitBake User Manual. - Because the files are parsed in a specific order, variable - assignments for the same variable could be affected. - For example, if the <filename>auto.conf</filename> file and - the <filename>local.conf</filename> set - <replaceable>variable1</replaceable> to different values, - because the build system parses <filename>local.conf</filename> - after <filename>auto.conf</filename>, - <replaceable>variable1</replaceable> is assigned the value from - the <filename>local.conf</filename> file. - </para> - </section> - - <section id="metadata-machine-configuration-and-policy-configuration"> - <title>Metadata, Machine Configuration, and Policy Configuration</title> - - <para> - The previous section described the user configurations that - define BitBake's global behavior. - This section takes a closer look at the layers the build system - uses to further control the build. - These layers provide Metadata for the software, machine, and - policies. - </para> - - <para> - In general, three types of layer input exists. - You can see them below the "User Configuration" box in the - <link linkend='general-workflow-figure'>general workflow figure</link>: - <itemizedlist> - <listitem><para> - <emphasis>Metadata (<filename>.bb</filename> + Patches):</emphasis> - Software layers containing user-supplied recipe files, - patches, and append files. - A good example of a software layer might be the - <ulink url='https://github.com/meta-qt5/meta-qt5'><filename>meta-qt5</filename></ulink> - layer from the - <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded Layer Index</ulink>. - This layer is for version 5.0 of the popular - <ulink url='https://wiki.qt.io/About_Qt'>Qt</ulink> - cross-platform application development framework for - desktop, embedded and mobile. - </para></listitem> - <listitem><para> - <emphasis>Machine BSP Configuration:</emphasis> - Board Support Package (BSP) layers (i.e. "BSP Layer" - in the following figure) providing machine-specific - configurations. - This type of information is specific to a particular - target architecture. - A good example of a BSP layer from the - <link linkend='gs-reference-distribution-poky'>Poky Reference Distribution</link> - is the - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-yocto-bsp'><filename>meta-yocto-bsp</filename></ulink> - layer. - </para></listitem> - <listitem><para> - <emphasis>Policy Configuration:</emphasis> - Distribution Layers (i.e. "Distro Layer" in the - following figure) providing top-level or general - policies for the images or SDKs being built for a - particular distribution. - For example, in the Poky Reference Distribution the - distro layer is the - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky'><filename>meta-poky</filename></ulink> - layer. - Within the distro layer is a - <filename>conf/distro</filename> directory that - contains distro configuration files (e.g. - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-poky/conf/distro/poky.conf'><filename>poky.conf</filename></ulink> - that contain many policy configurations for the - Poky distribution. - </para></listitem> - </itemizedlist> - </para> - - <para> - The following figure shows an expanded representation of - these three layers from the - <link linkend='general-workflow-figure'>general workflow figure</link>: - </para> - - <para> - <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="8in" /> - </para> - - <para> - In general, all layers have a similar structure. - They all contain a licensing file - (e.g. <filename>COPYING.MIT</filename>) if the layer is to be - distributed, a <filename>README</filename> file as good - practice and especially if the layer is to be distributed, a - configuration directory, and recipe directories. - You can learn about the general structure for layers used with - the Yocto Project in the - "<ulink url='&YOCTO_DOCS_DEV_URL;#creating-your-own-layer'>Creating Your Own Layer</ulink>" - section in the Yocto Project Development Tasks Manual. - For a general discussion on layers and the many layers from - which you can draw, see the - "<link linkend='overview-layers'>Layers</link>" and - "<link linkend='the-yocto-project-layer-model'>The Yocto Project Layer Model</link>" - sections both earlier in this manual. - </para> - - <para> - If you explored the previous links, you discovered some - areas where many layers that work with the Yocto Project - exist. - The - <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink> - also shows layers categorized under "Yocto Metadata Layers." - <note> - Layers exist in the Yocto Project Source Repositories that - cannot be found in the OpenEmbedded Layer Index. - These layers are either deprecated or experimental - in nature. - </note> - </para> - - <para> - BitBake uses the <filename>conf/bblayers.conf</filename> file, - which is part of the user configuration, to find what layers it - should be using as part of the build. - </para> - - <section id="distro-layer"> - <title>Distro Layer</title> - - <para> - The distribution layer provides policy configurations for - your distribution. - Best practices dictate that you isolate these types of - configurations into their own layer. - Settings you provide in - <filename>conf/distro/<replaceable>distro</replaceable>.conf</filename> override - similar settings that BitBake finds in your - <filename>conf/local.conf</filename> file in the Build - Directory. - </para> - - <para> - The following list provides some explanation and references - for what you typically find in the distribution layer: - <itemizedlist> - <listitem><para> - <emphasis>classes:</emphasis> - Class files (<filename>.bbclass</filename>) hold - common functionality that can be shared among - recipes in the distribution. - When your recipes inherit a class, they take on the - settings and functions for that class. - You can read more about class files in the - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>" - chapter of the Yocto Reference Manual. - </para></listitem> - <listitem><para> - <emphasis>conf:</emphasis> - This area holds configuration files for the - layer (<filename>conf/layer.conf</filename>), - the distribution - (<filename>conf/distro/<replaceable>distro</replaceable>.conf</filename>), - and any distribution-wide include files. - </para></listitem> - <listitem><para> - <emphasis>recipes-*:</emphasis> - Recipes and append files that affect common - functionality across the distribution. - This area could include recipes and append files - to add distribution-specific configuration, - initialization scripts, custom image recipes, - and so forth. - Examples of <filename>recipes-*</filename> - directories are <filename>recipes-core</filename> - and <filename>recipes-extra</filename>. - Hierarchy and contents within a - <filename>recipes-*</filename> directory can vary. - Generally, these directories contain recipe files - (<filename>*.bb</filename>), recipe append files - (<filename>*.bbappend</filename>), directories - that are distro-specific for configuration files, - and so forth. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id="bsp-layer"> - <title>BSP Layer</title> - - <para> - The BSP Layer provides machine configurations that - target specific hardware. - Everything in this layer is specific to the machine for - which you are building the image or the SDK. - A common structure or form is defined for BSP layers. - You can learn more about this structure in the - <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>. - <note> - In order for a BSP layer to be considered compliant - with the Yocto Project, it must meet some structural - requirements. - </note> - </para> - - <para> - The BSP Layer's configuration directory contains - configuration files for the machine - (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>) - and, of course, the layer - (<filename>conf/layer.conf</filename>). - </para> - - <para> - The remainder of the layer is dedicated to specific recipes - by function: <filename>recipes-bsp</filename>, - <filename>recipes-core</filename>, - <filename>recipes-graphics</filename>, - <filename>recipes-kernel</filename>, and so forth. - Metadata can exist for multiple formfactors, graphics - support systems, and so forth. - <note> - While the figure shows several - <filename>recipes-*</filename> directories, not all - these directories appear in all BSP layers. - </note> - </para> - </section> - - <section id="software-layer"> - <title>Software Layer</title> - - <para> - The software layer provides the Metadata for additional - software packages used during the build. - This layer does not include Metadata that is specific to - the distribution or the machine, which are found in their - respective layers. - </para> - - <para> - This layer contains any recipes, append files, and - patches, that your project needs. - </para> - </section> - </section> - - <section id="sources-dev-environment"> - <title>Sources</title> - - <para> - In order for the OpenEmbedded build system to create an - image or any target, it must be able to access source files. - The - <link linkend='general-workflow-figure'>general workflow figure</link> - represents source files using the "Upstream Project Releases", - "Local Projects", and "SCMs (optional)" boxes. - The figure represents mirrors, which also play a role in - locating source files, with the "Source Materials" box. - </para> - - <para> - The method by which source files are ultimately organized is - a function of the project. - For example, for released software, projects tend to use - tarballs or other archived files that can capture the - state of a release guaranteeing that it is statically - represented. - On the other hand, for a project that is more dynamic or - experimental in nature, a project might keep source files in a - repository controlled by a Source Control Manager (SCM) such as - Git. - Pulling source from a repository allows you to control - the point in the repository (the revision) from which you - want to build software. - Finally, a combination of the two might exist, which would - give the consumer a choice when deciding where to get - source files. - </para> - - <para> - BitBake uses the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> - variable to point to source files regardless of their location. - Each recipe must have a <filename>SRC_URI</filename> variable - that points to the source. - </para> - - <para> - Another area that plays a significant role in where source - files come from is pointed to by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> - variable. - This area is a cache that can hold previously downloaded - source. - You can also instruct the OpenEmbedded build system to create - tarballs from Git repositories, which is not the default - behavior, and store them in the <filename>DL_DIR</filename> - by using the - <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink> - variable. - </para> - - <para> - Judicious use of a <filename>DL_DIR</filename> directory can - save the build system a trip across the Internet when looking - for files. - A good method for using a download directory is to have - <filename>DL_DIR</filename> point to an area outside of your - Build Directory. - Doing so allows you to safely delete the Build Directory - if needed without fear of removing any downloaded source file. - </para> - - <para> - The remainder of this section provides a deeper look into the - source files and the mirrors. - Here is a more detailed look at the source file area of the - <link linkend='general-workflow-figure'>general workflow figure</link>: - </para> - - <para> - <imagedata fileref="figures/source-input.png" width="6in" depth="6in" align="center" /> - </para> - - <section id='upstream-project-releases'> - <title>Upstream Project Releases</title> - - <para> - Upstream project releases exist anywhere in the form of an - archived file (e.g. tarball or zip file). - These files correspond to individual recipes. - For example, the figure uses specific releases each for - BusyBox, Qt, and Dbus. - An archive file can be for any released product that can be - built using a recipe. - </para> - </section> - - <section id='local-projects'> - <title>Local Projects</title> - - <para> - Local projects are custom bits of software the user - provides. - These bits reside somewhere local to a project - perhaps - a directory into which the user checks in items (e.g. - a local directory containing a development source tree - used by the group). - </para> - - <para> - The canonical method through which to include a local - project is to use the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink> - class to include that local project. - You use either the <filename>local.conf</filename> or a - recipe's append file to override or set the - recipe to point to the local directory on your disk to pull - in the whole source tree. - </para> - </section> - - <section id='scms'> - <title>Source Control Managers (Optional)</title> - - <para> - Another place from which the build system can get source - files is with - <ulink url='&YOCTO_DOCS_BB_URL;#bb-fetchers'>fetchers</ulink> - employing various Source Control Managers (SCMs) such as - Git or Subversion. - In such cases, a repository is cloned or checked out. - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink> - task inside BitBake uses - the <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> - variable and the argument's prefix to determine the correct - fetcher module. - <note> - For information on how to have the OpenEmbedded build - system generate tarballs for Git repositories and place - them in the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> - directory, see the - <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink> - variable in the Yocto Project Reference Manual. - </note> - </para> - - <para> - When fetching a repository, BitBake uses the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink> - variable to determine the specific revision from which to - build. - </para> - </section> - - <section id='source-mirrors'> - <title>Source Mirror(s)</title> - - <para> - Two kinds of mirrors exist: pre-mirrors and regular - mirrors. - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-PREMIRRORS'><filename>PREMIRRORS</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#var-MIRRORS'><filename>MIRRORS</filename></ulink> - variables point to these, respectively. - BitBake checks pre-mirrors before looking upstream for any - source files. - Pre-mirrors are appropriate when you have a shared - directory that is not a directory defined by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> - variable. - A Pre-mirror typically points to a shared directory that is - local to your organization. - </para> - - <para> - Regular mirrors can be any site across the Internet - that is used as an alternative location for source - code should the primary site not be functioning for - some reason or another. - </para> - </section> - </section> - - <section id="package-feeds-dev-environment"> - <title>Package Feeds</title> - - <para> - When the OpenEmbedded build system generates an image or an - SDK, it gets the packages from a package feed area located - in the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. - The - <link linkend='general-workflow-figure'>general workflow figure</link> - shows this package feeds area in the upper-right corner. - </para> - - <para> - This section looks a little closer into the package feeds - area used by the build system. - Here is a more detailed look at the area: - <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" /> - </para> - - <para> - Package feeds are an intermediary step in the build process. - The OpenEmbedded build system provides classes to generate - different package types, and you specify which classes to - enable through the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> - variable. - Before placing the packages into package feeds, - the build process validates them with generated output quality - assurance checks through the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink> - class. - </para> - - <para> - The package feed area resides in the Build Directory. - The directory the build system uses to temporarily store - packages is determined by a combination of variables and the - particular package manager in use. - See the "Package Feeds" box in the illustration and note the - information to the right of that area. - In particular, the following defines where package files are - kept: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>: - Defined as <filename>tmp/deploy</filename> in the Build - Directory. - </para></listitem> - <listitem><para> - <filename>DEPLOY_DIR_*</filename>: - Depending on the package manager used, the package type - sub-folder. - Given RPM, IPK, or DEB packaging and tarball creation, - the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_RPM'><filename>DEPLOY_DIR_RPM</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IPK'><filename>DEPLOY_DIR_IPK</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_DEB'><filename>DEPLOY_DIR_DEB</filename></ulink>, - or - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_TAR'><filename>DEPLOY_DIR_TAR</filename></ulink>, - variables are used, respectively. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>: - Defines architecture-specific sub-folders. - For example, packages could exist for the i586 or - qemux86 architectures. - </para></listitem> - </itemizedlist> - </para> - - <para> - BitBake uses the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink> - tasks to generate packages and place them into the package - holding area (e.g. <filename>do_package_write_ipk</filename> - for IPK packages). - See the - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_deb</filename></ulink>", - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>", - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_rpm'><filename>do_package_write_rpm</filename></ulink>", - and - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_tar'><filename>do_package_write_tar</filename></ulink>" - sections in the Yocto Project Reference Manual - for additional information. - As an example, consider a scenario where an IPK packaging - manager is being used and package architecture support for - both i586 and qemux86 exist. - Packages for the i586 architecture are placed in - <filename>build/tmp/deploy/ipk/i586</filename>, while packages - for the qemux86 architecture are placed in - <filename>build/tmp/deploy/ipk/qemux86</filename>. - </para> - </section> - - <section id='bitbake-dev-environment'> - <title>BitBake</title> - - <para> - The OpenEmbedded build system uses - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> - to produce images and Software Development Kits (SDKs). - You can see from the - <link linkend='general-workflow-figure'>general workflow figure</link>, - the BitBake area consists of several functional areas. - This section takes a closer look at each of those areas. - <note> - Separate documentation exists for the BitBake tool. - See the - <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink> - for reference material on BitBake. - </note> - </para> - - <section id='source-fetching-dev-environment'> - <title>Source Fetching</title> - - <para> - The first stages of building a recipe are to fetch and - unpack the source code: - <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" /> - </para> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink> - tasks fetch the source files and unpack them into the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. - <note> - For every local file (e.g. <filename>file://</filename>) - that is part of a recipe's - <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> - statement, the OpenEmbedded build system takes a - checksum of the file for the recipe and inserts the - checksum into the signature for the - <filename>do_fetch</filename> task. - If any local file has been modified, the - <filename>do_fetch</filename> task and all tasks that - depend on it are re-executed. - </note> - By default, everything is accomplished in the Build - Directory, which has a defined structure. - For additional general information on the Build Directory, - see the - "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-build'><filename>build/</filename></ulink>" - section in the Yocto Project Reference Manual. - </para> - - <para> - Each recipe has an area in the Build Directory where the - unpacked source code resides. - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> - variable points to this area for a recipe's unpacked source - code. - The name of that directory for any given recipe is defined - from several different variables. - The preceding figure and the following list describe - the Build Directory's hierarchy: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>: - The base directory where the OpenEmbedded build - system performs all its work during the build. - The default base directory is the - <filename>tmp</filename> directory. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>: - The architecture of the built package or packages. - Depending on the eventual destination of the - package or packages (i.e. machine architecture, - <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>, - SDK, or specific machine), - <filename>PACKAGE_ARCH</filename> varies. - See the variable's description for details. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_OS'><filename>TARGET_OS</filename></ulink>: - The operating system of the target device. - A typical value would be "linux" (e.g. - "qemux86-poky-linux"). - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>: - The name of the recipe used to build the package. - This variable can have multiple meanings. - However, when used in the context of input files, - <filename>PN</filename> represents the the name - of the recipe. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>: - The location where the OpenEmbedded build system - builds a recipe (i.e. does the work to create the - package). - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>: - The version of the recipe used to build the - package. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>: - The revision of the recipe used to build the - package. - </para></listitem> - </itemizedlist> - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>: - Contains the unpacked source files for a given - recipe. - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink>: - The name of the recipe used to build the - package. - The <filename>BPN</filename> variable is - a version of the <filename>PN</filename> - variable but with common prefixes and - suffixes removed. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>: - The version of the recipe used to build the - package. - </para></listitem> - </itemizedlist> - </para></listitem> - </itemizedlist> - <note> - In the previous figure, notice that two sample - hierarchies exist: one based on package architecture (i.e. - <filename>PACKAGE_ARCH</filename>) and one based on a - machine (i.e. <filename>MACHINE</filename>). - The underlying structures are identical. - The differentiator being what the OpenEmbedded build - system is using as a build target (e.g. general - architecture, a build host, an SDK, or a specific - machine). - </note> - </para> - </section> - - <section id='patching-dev-environment'> - <title>Patching</title> - - <para> - Once source code is fetched and unpacked, BitBake locates - patch files and applies them to the source files: - <imagedata fileref="figures/patching.png" align="center" width="7in" depth="6in" /> - </para> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> - task uses a recipe's - <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> - statements and the - <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink> - variable to locate applicable patch files. - </para> - - <para> - Default processing for patch files assumes the files have - either <filename>*.patch</filename> or - <filename>*.diff</filename> file types. - You can use <filename>SRC_URI</filename> parameters to - change the way the build system recognizes patch files. - See the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> - task for more information. - </para> - - <para> - BitBake finds and applies multiple patches for a single - recipe in the order in which it locates the patches. - The <filename>FILESPATH</filename> variable defines the - default set of directories that the build system uses to - search for patch files. - Once found, patches are applied to the recipe's source - files, which are located in the - <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> - directory. - </para> - - <para> - For more information on how the source directories are - created, see the - "<link linkend='source-fetching-dev-environment'>Source Fetching</link>" - section. - For more information on how to create patches and how the - build system processes patches, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#new-recipe-patching-code'>Patching Code</ulink>" - section in the Yocto Project Development Tasks Manual. - You can also see the - "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-devtool-use-devtool-modify-to-modify-the-source-of-an-existing-component'>Use <filename>devtool modify</filename> to Modify the Source of an Existing Component</ulink>" - section in the Yocto Project Application Development and - the Extensible Software Development Kit (SDK) manual and - the - "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#using-traditional-kernel-development-to-patch-the-kernel'>Using Traditional Kernel Development to Patch the Kernel</ulink>" - section in the Yocto Project Linux Kernel Development - Manual. - </para> - </section> - - <section id='configuration-compilation-and-staging-dev-environment'> - <title>Configuration, Compilation, and Staging</title> - - <para> - After source code is patched, BitBake executes tasks that - configure and compile the source code. - Once compilation occurs, the files are copied to a holding - area (staged) in preparation for packaging: - <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" /> - </para> - - <para> - This step in the build process consists of the following - tasks: - <itemizedlist> - <listitem><para> - <emphasis><ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></ulink></emphasis>: - This task sets up the two sysroots in - <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename> - (i.e. <filename>recipe-sysroot</filename> and - <filename>recipe-sysroot-native</filename>) so that - during the packaging phase the sysroots can contain - the contents of the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink> - tasks of the recipes on which the recipe - containing the tasks depends. - A sysroot exists for both the target and for the - native binaries, which run on the host system. - </para></listitem> - <listitem><para> - <emphasis><filename>do_configure</filename></emphasis>: - This task configures the source by enabling and - disabling any build-time and configuration options - for the software being built. - Configurations can come from the recipe itself as - well as from an inherited class. - Additionally, the software itself might configure - itself depending on the target for which it is - being built.</para> - - <para>The configurations handled by the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> - task are specific to configurations for the source - code being built by the recipe.</para> - - <para>If you are using the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> - class, you can add additional configuration options - by using the - <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink> - or - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink> - variables. - For information on how this variable works within - that class, see the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> - class - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/autotools.bbclass'>here</ulink>. - </para></listitem> - <listitem><para> - <emphasis><filename>do_compile</filename></emphasis>: - Once a configuration task has been satisfied, - BitBake compiles the source using the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink> - task. - Compilation occurs in the directory pointed to by - the - <ulink url='&YOCTO_DOCS_REF_URL;#var-B'><filename>B</filename></ulink> - variable. - Realize that the <filename>B</filename> directory - is, by default, the same as the - <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> - directory. - </para></listitem> - <listitem><para> - <emphasis><filename>do_install</filename></emphasis>: - After compilation completes, BitBake executes the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> - task. - This task copies files from the - <filename>B</filename> directory and places them - in a holding area pointed to by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink> - variable. - Packaging occurs later using files from this - holding directory. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='package-splitting-dev-environment'> - <title>Package Splitting</title> - - <para> - After source code is configured, compiled, and staged, the - build system analyzes the results and splits the output - into packages: - <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" /> - </para> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink> - tasks combine to analyze the files found in the - <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink> - directory and split them into subsets based on available - packages and files. - Analysis involves the following as well as other items: - splitting out debugging symbols, looking at shared library - dependencies between packages, and looking at package - relationships. - </para> - - <para> - The <filename>do_packagedata</filename> task creates - package metadata based on the analysis such that the - build system can generate the final packages. - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink> - task stages (copies) a subset of the files installed by - the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> - task into the appropriate sysroot. - Working, staged, and intermediate results of the analysis - and package splitting process use several areas: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGD'><filename>PKGD</filename></ulink>: - The destination directory - (i.e. <filename>package</filename>) for packages - before they are split into individual packages. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDESTWORK'><filename>PKGDESTWORK</filename></ulink>: - A temporary work area (i.e. - <filename>pkgdata</filename>) used by the - <filename>do_package</filename> task to save - package metadata. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDEST'><filename>PKGDEST</filename></ulink>: - The parent directory (i.e. - <filename>packages-split</filename>) for packages - after they have been split. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink>: - A shared, global-state directory that holds - packaging metadata generated during the packaging - process. - The packaging process copies metadata from - <filename>PKGDESTWORK</filename> to the - <filename>PKGDATA_DIR</filename> area where it - becomes globally available. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_HOST'><filename>STAGING_DIR_HOST</filename></ulink>: - The path for the sysroot for the system on which - a component is built to run (i.e. - <filename>recipe-sysroot</filename>). - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_NATIVE'><filename>STAGING_DIR_NATIVE</filename></ulink>: - The path for the sysroot used when building - components for the build host (i.e. - <filename>recipe-sysroot-native</filename>). - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR_TARGET'><filename>STAGING_DIR_TARGET</filename></ulink>: - The path for the sysroot used when a component that - is built to execute on a system and it generates - code for yet another machine (e.g. cross-canadian - recipes). - </para></listitem> - </itemizedlist> - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink> - variable defines the files that go into each package in - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>. - If you want details on how this is accomplished, you can - look at - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass'><filename>package.bbclass</filename></ulink>. - </para> - - <para> - Depending on the type of packages being created (RPM, DEB, - or IPK), the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink> - task creates the actual packages and places them in the - Package Feed area, which is - <filename>${TMPDIR}/deploy</filename>. - You can see the - "<link linkend='package-feeds-dev-environment'>Package Feeds</link>" - section for more detail on that part of the build process. - <note> - Support for creating feeds directly from the - <filename>deploy/*</filename> directories does not - exist. - Creating such feeds usually requires some kind of feed - maintenance mechanism that would upload the new - packages into an official package feed (e.g. the - Ångström distribution). - This functionality is highly distribution-specific - and thus is not provided out of the box. - </note> - </para> - </section> - - <section id='image-generation-dev-environment'> - <title>Image Generation</title> - - <para> - Once packages are split and stored in the Package Feeds - area, the build system uses BitBake to generate the root - filesystem image: - <imagedata fileref="figures/image-generation.png" align="center" width="7.5in" depth="7.5in" /> - </para> - - <para> - The image generation process consists of several stages and - depends on several tasks and variables. - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink> - task creates the root filesystem (file and directory - structure) for an image. - This task uses several key variables to help create the - list of packages to actually install: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>: - Lists out the base set of packages from which to - install from the Package Feeds area. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></ulink>: - Specifies packages that should not be installed - into the image. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>: - Specifies features to include in the image. - Most of these features map to additional packages - for installation. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>: - Specifies the package backend (e.g. RPM, DEB, or - IPK) to use and consequently helps determine where - to locate packages within the Package Feeds area. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></ulink>: - Determines the language(s) for which additional - language support packages are installed. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink>: - The final list of packages passed to the package - manager for installation into the image. - </para></listitem> - </itemizedlist> - </para> - - <para> - With - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_ROOTFS'><filename>IMAGE_ROOTFS</filename></ulink> - pointing to the location of the filesystem under - construction and the <filename>PACKAGE_INSTALL</filename> - variable providing the final list of packages to install, - the root file system is created. - </para> - - <para> - Package installation is under control of the package - manager (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of - whether or not package management is enabled for the - target. - At the end of the process, if package management is not - enabled for the target, the package manager's data files - are deleted from the root filesystem. - As part of the final stage of package installation, - post installation scripts that are part of the packages - are run. - Any scripts that fail to run on the build host are run on - the target when the target system is first booted. - If you are using a - <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>, - all the post installation scripts must succeed on the - build host during the package installation phase since the - root filesystem on the target is read-only. - </para> - - <para> - The final stages of the <filename>do_rootfs</filename> task - handle post processing. - Post processing includes creation of a manifest file and - optimizations. - </para> - - <para> - The manifest file (<filename>.manifest</filename>) resides - in the same directory as the root filesystem image. - This file lists out, line-by-line, the installed packages. - The manifest file is useful for the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink> - class, for example, to determine whether or not to run - specific tests. - See the - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></ulink> - variable for additional information. - </para> - - <para> - Optimizing processes that are run across the image include - <filename>mklibs</filename>, <filename>prelink</filename>, - and any other post-processing commands as defined by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-ROOTFS_POSTPROCESS_COMMAND'><filename>ROOTFS_POSTPROCESS_COMMAND</filename></ulink> - variable. - The <filename>mklibs</filename> process optimizes the size - of the libraries, while the <filename>prelink</filename> - process optimizes the dynamic linking of shared libraries - to reduce start up time of executables. - </para> - - <para> - After the root filesystem is built, processing begins on - the image through the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image</filename></ulink> - task. - The build system runs any pre-processing commands as - defined by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_PREPROCESS_COMMAND'><filename>IMAGE_PREPROCESS_COMMAND</filename></ulink> - variable. - This variable specifies a list of functions to call before - the build system creates the final image output files. - </para> - - <para> - The build system dynamically creates - <filename>do_image_*</filename> tasks as needed, based - on the image types specified in the - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink> - variable. - The process turns everything into an image file or a set of - image files and can compress the root filesystem image to - reduce the overall size of the image. - The formats used for the root filesystem depend on the - <filename>IMAGE_FSTYPES</filename> variable. - Compression depends on whether the formats support - compression. - </para> - - <para> - As an example, a dynamically created task when creating a - particular image <replaceable>type</replaceable> would - take the following form: - <literallayout class='monospaced'> - do_image_<replaceable>type</replaceable> - </literallayout> - So, if the <replaceable>type</replaceable> as specified by - the <filename>IMAGE_FSTYPES</filename> were - <filename>ext4</filename>, the dynamically generated task - would be as follows: - <literallayout class='monospaced'> - do_image_ext4 - </literallayout> - </para> - - <para> - The final task involved in image creation is the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image-complete'><filename>do_image_complete</filename></ulink> - task. - This task completes the image by applying any image - post processing as defined through the - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_POSTPROCESS_COMMAND'><filename>IMAGE_POSTPROCESS_COMMAND</filename></ulink> - variable. - The variable specifies a list of functions to call once the - build system has created the final image output files. - <note> - The entire image generation process is run under - <link linkend='fakeroot-and-pseudo'>Pseudo</link>. - Running under Pseudo ensures that the files in the - root filesystem have correct ownership. - </note> - </para> - </section> - - <section id='sdk-generation-dev-environment'> - <title>SDK Generation</title> - - <para> - The OpenEmbedded build system uses BitBake to generate the - Software Development Kit (SDK) installer scripts for both - the standard SDK and the extensible SDK (eSDK): - </para> - - <para> - <imagedata fileref="figures/sdk-generation.png" width="9in" align="center" /> - <note> - For more information on the cross-development toolchain - generation, see the - "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>" - section. - For information on advantages gained when building a - cross-development toolchain using the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink> - task, see the - "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>" - section in the Yocto Project Application Development - and the Extensible Software Development Kit (eSDK) - manual. - </note> - </para> - - <para> - Like image generation, the SDK script process consists of - several stages and depends on many variables. - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk_ext'><filename>do_populate_sdk_ext</filename></ulink> - tasks use these key variables to help create the list of - packages to actually install. - For information on the variables listed in the figure, - see the - "<link linkend='sdk-dev-environment'>Application Development SDK</link>" - section. - </para> - - <para> - The <filename>do_populate_sdk</filename> task helps create - the standard SDK and handles two parts: a target part and a - host part. - The target part is the part built for the target hardware - and includes libraries and headers. - The host part is the part of the SDK that runs on the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>. - </para> - - <para> - The <filename>do_populate_sdk_ext</filename> task helps - create the extensible SDK and handles host and target parts - differently than its counter part does for the standard SDK. - For the extensible SDK, the task encapsulates the build - system, which includes everything needed (host and target) - for the SDK. - </para> - - <para> - Regardless of the type of SDK being constructed, the - tasks perform some cleanup after which a cross-development - environment setup script and any needed configuration files - are created. - The final output is the Cross-development - toolchain installation script (<filename>.sh</filename> - file), which includes the environment setup script. - </para> - </section> - - <section id='stamp-files-and-the-rerunning-of-tasks'> - <title>Stamp Files and the Rerunning of Tasks</title> - - <para> - For each task that completes successfully, BitBake writes a - stamp file into the - <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR'><filename>STAMPS_DIR</filename></ulink> - directory. - The beginning of the stamp file's filename is determined - by the - <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink> - variable, and the end of the name consists of the task's - name and current - <link linkend='overview-checksums'>input checksum</link>. - <note> - This naming scheme assumes that - <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink> - is "OEBasicHash", which is almost always the case in - current OpenEmbedded. - </note> - To determine if a task needs to be rerun, BitBake checks - if a stamp file with a matching input checksum exists - for the task. - If such a stamp file exists, the task's output is - assumed to exist and still be valid. - If the file does not exist, the task is rerun. - <note> - <para>The stamp mechanism is more general than the - shared state (sstate) cache mechanism described in the - "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>" - section. - BitBake avoids rerunning any task that has a valid - stamp file, not just tasks that can be accelerated - through the sstate cache.</para> - - <para>However, you should realize that stamp files only - serve as a marker that some work has been done and that - these files do not record task output. - The actual task output would usually be somewhere in - <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> - (e.g. in some recipe's - <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.) - What the sstate cache mechanism adds is a way to cache - task output that can then be shared between build - machines.</para> - </note> - Since <filename>STAMPS_DIR</filename> is usually a - subdirectory of <filename>TMPDIR</filename>, removing - <filename>TMPDIR</filename> will also remove - <filename>STAMPS_DIR</filename>, which means tasks will - properly be rerun to repopulate - <filename>TMPDIR</filename>. - </para> - - <para> - If you want some task to always be considered "out of - date", you can mark it with the - <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink> - varflag. - If some other task depends on such a task, then that - task will also always be considered out of date, which - might not be what you want. - </para> - - <para> - For details on how to view information about a task's - signature, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>" - section in the Yocto Project Development Tasks Manual. - </para> - </section> - - <section id='setscene-tasks-and-shared-state'> - <title>Setscene Tasks and Shared State</title> - - <para> - The description of tasks so far assumes that BitBake needs - to build everything and no available prebuilt objects - exist. - BitBake does support skipping tasks if prebuilt objects are - available. - These objects are usually made available in the form of a - shared state (sstate) cache. - <note> - For information on variables affecting sstate, see the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink> - variables. - </note> - </para> - - <para> - The idea of a setscene task (i.e - <filename>do_</filename><replaceable>taskname</replaceable><filename>_setscene</filename>) - is a version of the task where - instead of building something, BitBake can skip to the end - result and simply place a set of files into specific - locations as needed. - In some cases, it makes sense to have a setscene task - variant (e.g. generating package files in the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_*</filename></ulink> - task). - In other cases, it does not make sense (e.g. a - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> - task or a - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink> - task) since the work involved would be equal to or greater - than the underlying task. - </para> - - <para> - In the build system, the common tasks that have setscene - variants are - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>, - <filename>do_package_write_*</filename>, - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>, - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>. - Notice that these tasks represent most of the tasks whose - output is an end result. - </para> - - <para> - The build system has knowledge of the relationship between - these tasks and other preceding tasks. - For example, if BitBake runs - <filename>do_populate_sysroot_setscene</filename> for - something, it does not make sense to run any of the - <filename>do_fetch</filename>, - <filename>do_unpack</filename>, - <filename>do_patch</filename>, - <filename>do_configure</filename>, - <filename>do_compile</filename>, and - <filename>do_install</filename> tasks. - However, if <filename>do_package</filename> needs to be - run, BitBake needs to run those other tasks. - </para> - - <para> - It becomes more complicated if everything can come - from an sstate cache because some objects are simply - not required at all. - For example, you do not need a compiler or native tools, - such as quilt, if nothing exists to compile or patch. - If the <filename>do_package_write_*</filename> packages - are available from sstate, BitBake does not need the - <filename>do_package</filename> task data. - </para> - - <para> - To handle all these complexities, BitBake runs in two - phases. - The first is the "setscene" stage. - During this stage, BitBake first checks the sstate cache - for any targets it is planning to build. - BitBake does a fast check to see if the object exists - rather than a complete download. - If nothing exists, the second phase, which is the setscene - stage, completes and the main build proceeds. - </para> - - <para> - If objects are found in the sstate cache, the build system - works backwards from the end targets specified by the user. - For example, if an image is being built, the build system - first looks for the packages needed for that image and the - tools needed to construct an image. - If those are available, the compiler is not needed. - Thus, the compiler is not even downloaded. - If something was found to be unavailable, or the - download or setscene task fails, the build system then - tries to install dependencies, such as the compiler, from - the cache. - </para> - - <para> - The availability of objects in the sstate cache is - handled by the function specified by the - <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></ulink> - variable and returns a list of available objects. - The function specified by the - <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></ulink> - variable is the function that determines whether a given - dependency needs to be followed, and whether for any given - relationship the function needs to be passed. - The function returns a True or False value. - </para> - </section> - </section> - - <section id='images-dev-environment'> - <title>Images</title> - - <para> - The images produced by the build system are compressed forms - of the root filesystem and are ready to boot on a target - device. - You can see from the - <link linkend='general-workflow-figure'>general workflow figure</link> - that BitBake output, in part, consists of images. - This section takes a closer look at this output: - <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" /> - </para> - - <note> - For a list of example images that the Yocto Project provides, - see the - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" - chapter in the Yocto Project Reference Manual. - </note> - - <para> - The build process writes images out to the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> - inside the - <filename>tmp/deploy/images/<replaceable>machine</replaceable>/</filename> - folder as shown in the figure. - This folder contains any files expected to be loaded on the - target device. - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink> - variable points to the <filename>deploy</filename> directory, - while the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></ulink> - variable points to the appropriate directory containing images - for the current configuration. - <itemizedlist> - <listitem><para> - <replaceable>kernel-image</replaceable>: - A kernel binary file. - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></ulink> - variable determines the naming scheme for the - kernel image file. - Depending on this variable, the file could begin with - a variety of naming strings. - The - <filename>deploy/images/</filename><replaceable>machine</replaceable> - directory can contain multiple image files for the - machine. - </para></listitem> - <listitem><para> - <replaceable>root-filesystem-image</replaceable>: - Root filesystems for the target device (e.g. - <filename>*.ext3</filename> or - <filename>*.bz2</filename> files). - The - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink> - variable determines the root filesystem image type. - The - <filename>deploy/images/</filename><replaceable>machine</replaceable> - directory can contain multiple root filesystems for the - machine. - </para></listitem> - <listitem><para> - <replaceable>kernel-modules</replaceable>: - Tarballs that contain all the modules built for the - kernel. - Kernel module tarballs exist for legacy purposes and - can be suppressed by setting the - <ulink url='&YOCTO_DOCS_REF_URL;#var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></ulink> - variable to "0". - The - <filename>deploy/images/</filename><replaceable>machine</replaceable> - directory can contain multiple kernel module tarballs - for the machine. - </para></listitem> - <listitem><para> - <replaceable>bootloaders</replaceable>: - If applicable to the target machine, bootloaders - supporting the image. - The <filename>deploy/images/</filename><replaceable>machine</replaceable> - directory can contain multiple bootloaders for the - machine. - </para></listitem> - <listitem><para> - <replaceable>symlinks</replaceable>: - The - <filename>deploy/images/</filename><replaceable>machine</replaceable> - folder contains a symbolic link that points to the - most recently built file for each machine. - These links might be useful for external scripts that - need to obtain the latest version of each file. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='sdk-dev-environment'> - <title>Application Development SDK</title> - - <para> - In the - <link linkend='general-workflow-figure'>general workflow figure</link>, - the output labeled "Application Development SDK" represents an - SDK. - The SDK generation process differs depending on whether you - build an extensible SDK (e.g. - <filename>bitbake -c populate_sdk_ext</filename> <replaceable>imagename</replaceable>) - or a standard SDK (e.g. - <filename>bitbake -c populate_sdk</filename> <replaceable>imagename</replaceable>). - This section takes a closer look at this output: - <imagedata fileref="figures/sdk.png" align="center" width="9in" depth="7.25in" /> - </para> - - <para> - The specific form of this output is a set of files that - includes a self-extracting SDK installer - (<filename>*.sh</filename>), host and target manifest files, - and files used for SDK testing. - When the SDK installer file is run, it installs the SDK. - The SDK consists of a cross-development toolchain, a set of - libraries and headers, and an SDK environment setup script. - Running this installer essentially sets up your - cross-development environment. - You can think of the cross-toolchain as the "host" - part because it runs on the SDK machine. - You can think of the libraries and headers as the "target" - part because they are built for the target hardware. - The environment setup script is added so that you can - initialize the environment before using the tools. - </para> - - <note><title>Notes</title> - <itemizedlist> - <listitem><para> - The Yocto Project supports several methods by which - you can set up this cross-development environment. - These methods include downloading pre-built SDK - installers or building and installing your own SDK - installer. - </para></listitem> - <listitem><para> - For background information on cross-development - toolchains in the Yocto Project development - environment, see the - "<link linkend='cross-development-toolchain-generation'>Cross-Development Toolchain Generation</link>" - section. - </para></listitem> - <listitem><para> - For information on setting up a cross-development - environment, see the - <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> - manual. - </para></listitem> - </itemizedlist> - </note> - - <para> - All the output files for an SDK are written to the - <filename>deploy/sdk</filename> folder inside the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> - as shown in the previous figure. - Depending on the type of SDK, several variables exist that help - configure these files. - The following list shows the variables associated with an - extensible SDK: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>: - Points to the <filename>deploy</filename> directory. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></ulink>: - Controls whether or not shared state artifacts are - copied into the extensible SDK. - By default, all required shared state artifacts are - copied into the SDK. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_PKGDATA'><filename>SDK_INCLUDE_PKGDATA</filename></ulink>: - Specifies whether or not packagedata is included in the - extensible SDK for all recipes in the "world" target. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></ulink>: - Specifies whether or not the toolchain is included - when building the extensible SDK. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></ulink>: - A list of variables allowed through from the build - system configuration into the extensible SDK - configuration. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_BLACKLIST'><filename>SDK_LOCAL_CONF_BLACKLIST</filename></ulink>: - A list of variables not allowed through from the build - system configuration into the extensible SDK - configuration. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INHERIT_BLACKLIST'><filename>SDK_INHERIT_BLACKLIST</filename></ulink>: - A list of classes to remove from the - <ulink url='&YOCTO_DOCS_REF_URL;#var-INHERIT'><filename>INHERIT</filename></ulink> - value globally within the extensible SDK configuration. - </para></listitem> - </itemizedlist> - This next list, shows the variables associated with a standard - SDK: - <itemizedlist> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>: - Points to the <filename>deploy</filename> directory. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>: - Specifies the architecture of the machine on which the - cross-development tools are run to create packages for - the target hardware. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></ulink>: - Lists the features to include in the "target" part - of the SDK. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></ulink>: - Lists packages that make up the host part of the SDK - (i.e. the part that runs on the - <filename>SDKMACHINE</filename>). - When you use - <filename>bitbake -c populate_sdk <replaceable>imagename</replaceable></filename> - to create the SDK, a set of default packages apply. - This variable allows you to add more packages. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></ulink>: - Lists packages that make up the target part of the SDK - (i.e. the part built for the target hardware). - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKPATH'><filename>SDKPATH</filename></ulink>: - Defines the default SDK installation path offered by - the installation script. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_HOST_MANIFEST'><filename>SDK_HOST_MANIFEST</filename></ulink>: - Lists all the installed packages that make up the host - part of the SDK. - This variable also plays a minor role for extensible - SDK development as well. - However, it is mainly used for the standard SDK. - </para></listitem> - <listitem><para> - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_TARGET_MANIFEST'><filename>SDK_TARGET_MANIFEST</filename></ulink>: - Lists all the installed packages that make up the - target part of the SDK. - This variable also plays a minor role for extensible - SDK development as well. - However, it is mainly used for the standard SDK. - </para></listitem> - </itemizedlist> - </para> - </section> - </section> - - <section id="cross-development-toolchain-generation"> - <title>Cross-Development Toolchain Generation</title> - - <para> - The Yocto Project does most of the work for you when it comes to - creating - <ulink url='&YOCTO_DOCS_REF_URL;#cross-development-toolchain'>cross-development toolchains</ulink>. - This section provides some technical background on how - cross-development toolchains are created and used. - For more information on toolchains, you can also see the - <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> - manual. - </para> - - <para> - In the Yocto Project development environment, cross-development - toolchains are used to build images and applications that run - on the target hardware. - With just a few commands, the OpenEmbedded build system creates - these necessary toolchains for you. - </para> - - <para> - The following figure shows a high-level build environment regarding - toolchain construction and use. - </para> - - <para> - <imagedata fileref="figures/cross-development-toolchains.png" width="8in" depth="6in" align="center" /> - </para> - - <para> - Most of the work occurs on the Build Host. - This is the machine used to build images and generally work within - the the Yocto Project environment. - When you run - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> - to create an image, the OpenEmbedded build system - uses the host <filename>gcc</filename> compiler to bootstrap a - cross-compiler named <filename>gcc-cross</filename>. - The <filename>gcc-cross</filename> compiler is what BitBake uses to - compile source files when creating the target image. - You can think of <filename>gcc-cross</filename> simply as an - automatically generated cross-compiler that is used internally - within BitBake only. - <note> - The extensible SDK does not use - <filename>gcc-cross-canadian</filename> since this SDK - ships a copy of the OpenEmbedded build system and the sysroot - within it contains <filename>gcc-cross</filename>. - </note> - </para> - - <para> - The chain of events that occurs when <filename>gcc-cross</filename> is - bootstrapped is as follows: - <literallayout class='monospaced'> - gcc -> binutils-cross -> gcc-cross-initial -> linux-libc-headers -> glibc-initial -> glibc -> gcc-cross -> gcc-runtime - </literallayout> - <itemizedlist> - <listitem><para> - <filename>gcc</filename>: - The build host's GNU Compiler Collection (GCC). - </para></listitem> - <listitem><para> - <filename>binutils-cross</filename>: - The bare minimum binary utilities needed in order to run - the <filename>gcc-cross-initial</filename> phase of the - bootstrap operation. - </para></listitem> - <listitem><para> - <filename>gcc-cross-initial</filename>: - An early stage of the bootstrap process for creating - the cross-compiler. - This stage builds enough of the <filename>gcc-cross</filename>, - the C library, and other pieces needed to finish building the - final cross-compiler in later stages. - This tool is a "native" package (i.e. it is designed to run on - the build host). - </para></listitem> - <listitem><para> - <filename>linux-libc-headers</filename>: - Headers needed for the cross-compiler. - </para></listitem> - <listitem><para> - <filename>glibc-initial</filename>: - An initial version of the Embedded GNU C Library - (GLIBC) needed to bootstrap <filename>glibc</filename>. - </para></listitem> - <listitem><para> - <filename>glibc</filename>: - The GNU C Library. - </para></listitem> - <listitem><para> - <filename>gcc-cross</filename>: - The final stage of the bootstrap process for the - cross-compiler. - This stage results in the actual cross-compiler that - BitBake uses when it builds an image for a targeted - device. - <note> - If you are replacing this cross compiler toolchain - with a custom version, you must replace - <filename>gcc-cross</filename>. - </note> - This tool is also a "native" package (i.e. it is - designed to run on the build host). - </para></listitem> - <listitem><para> - <filename>gcc-runtime</filename>: - Runtime libraries resulting from the toolchain bootstrapping - process. - This tool produces a binary that consists of the - runtime libraries need for the targeted device. - </para></listitem> - </itemizedlist> - </para> - - <para> - You can use the OpenEmbedded build system to build an installer for - the relocatable SDK used to develop applications. - When you run the installer, it installs the toolchain, which - contains the development tools (e.g., - <filename>gcc-cross-canadian</filename>, - <filename>binutils-cross-canadian</filename>, and other - <filename>nativesdk-*</filename> tools), - which are tools native to the SDK (i.e. native to - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_ARCH'><filename>SDK_ARCH</filename></ulink>), - you need to cross-compile and test your software. - The figure shows the commands you use to easily build out this - toolchain. - This cross-development toolchain is built to execute on the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>, - which might or might not be the same - machine as the Build Host. - <note> - If your target architecture is supported by the Yocto Project, - you can take advantage of pre-built images that ship with the - Yocto Project and already contain cross-development toolchain - installers. - </note> - </para> - - <para> - Here is the bootstrap process for the relocatable toolchain: - <literallayout class='monospaced'> - gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> - glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian - </literallayout> - <itemizedlist> - <listitem><para> - <filename>gcc</filename>: - The build host's GNU Compiler Collection (GCC). - </para></listitem> - <listitem><para> - <filename>binutils-crosssdk</filename>: - The bare minimum binary utilities needed in order to run - the <filename>gcc-crosssdk-initial</filename> phase of the - bootstrap operation. - </para></listitem> - <listitem><para> - <filename>gcc-crosssdk-initial</filename>: - An early stage of the bootstrap process for creating - the cross-compiler. - This stage builds enough of the - <filename>gcc-crosssdk</filename> and supporting pieces so that - the final stage of the bootstrap process can produce the - finished cross-compiler. - This tool is a "native" binary that runs on the build host. - </para></listitem> - <listitem><para> - <filename>linux-libc-headers</filename>: - Headers needed for the cross-compiler. - </para></listitem> - <listitem><para> - <filename>glibc-initial</filename>: - An initial version of the Embedded GLIBC needed to bootstrap - <filename>nativesdk-glibc</filename>. - </para></listitem> - <listitem><para> - <filename>nativesdk-glibc</filename>: - The Embedded GLIBC needed to bootstrap the - <filename>gcc-crosssdk</filename>. - </para></listitem> - <listitem><para> - <filename>gcc-crosssdk</filename>: - The final stage of the bootstrap process for the - relocatable cross-compiler. - The <filename>gcc-crosssdk</filename> is a transitory - compiler and never leaves the build host. - Its purpose is to help in the bootstrap process to create - the eventual <filename>gcc-cross-canadian</filename> - compiler, which is relocatable. - This tool is also a "native" package (i.e. it is - designed to run on the build host). - </para></listitem> - <listitem><para> - <filename>gcc-cross-canadian</filename>: - The final relocatable cross-compiler. - When run on the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>, - this tool - produces executable code that runs on the target device. - Only one cross-canadian compiler is produced per architecture - since they can be targeted at different processor optimizations - using configurations passed to the compiler through the - compile commands. - This circumvents the need for multiple compilers and thus - reduces the size of the toolchains. - </para></listitem> - </itemizedlist> - </para> - - <note> - For information on advantages gained when building a - cross-development toolchain installer, see the - "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>" - appendix in the Yocto Project Application Development and the - Extensible Software Development Kit (eSDK) manual. - </note> - </section> - - <section id="shared-state-cache"> - <title>Shared State Cache</title> - - <para> - By design, the OpenEmbedded build system builds everything from - scratch unless - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> - can determine that parts do not need to be rebuilt. - Fundamentally, building from scratch is attractive as it means all - parts are built fresh and no possibility of stale data exists that - can cause problems. - When developers hit problems, they typically default back to - building from scratch so they have a know state from the - start. - </para> - - <para> - Building an image from scratch is both an advantage and a - disadvantage to the process. - As mentioned in the previous paragraph, building from scratch - ensures that everything is current and starts from a known state. - However, building from scratch also takes much longer as it - generally means rebuilding things that do not necessarily need - to be rebuilt. - </para> - - <para> - The Yocto Project implements shared state code that supports - incremental builds. - The implementation of the shared state code answers the following - questions that were fundamental roadblocks within the OpenEmbedded - incremental build support system: - <itemizedlist> - <listitem><para> - What pieces of the system have changed and what pieces have - not changed? - </para></listitem> - <listitem><para> - How are changed pieces of software removed and replaced? - </para></listitem> - <listitem><para> - How are pre-built components that do not need to be rebuilt - from scratch used when they are available? - </para></listitem> - </itemizedlist> - </para> - - <para> - For the first question, the build system detects changes in the - "inputs" to a given task by creating a checksum (or signature) of - the task's inputs. - If the checksum changes, the system assumes the inputs have changed - and the task needs to be rerun. - For the second question, the shared state (sstate) code tracks - which tasks add which output to the build process. - This means the output from a given task can be removed, upgraded - or otherwise manipulated. - The third question is partly addressed by the solution for the - second question assuming the build system can fetch the sstate - objects from remote locations and install them if they are deemed - to be valid. - <note><title>Notes</title> - <itemizedlist> - <listitem><para> - The build system does not maintain - <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> - information as part of the shared state packages. - Consequently, considerations exist that affect - maintaining shared state feeds. - For information on how the build system works with - packages and can track incrementing - <filename>PR</filename> information, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>" - section in the Yocto Project Development Tasks Manual. - </para></listitem> - <listitem><para> - The code in the build system that supports incremental - builds is not simple code. - For techniques that help you work around issues related - to shared state code, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-viewing-metadata-used-to-create-the-input-signature-of-a-shared-state-task'>Viewing Metadata Used to Create the Input Signature of a Shared State Task</ulink>" - and - "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-invalidating-shared-state-to-force-a-task-to-run'>Invalidating Shared State to Force a Task to Run</ulink>" - sections both in the Yocto Project Development Tasks - Manual. - </para></listitem> - </itemizedlist> - </note> - </para> - - <para> - The rest of this section goes into detail about the overall - incremental build architecture, the checksums (signatures), and - shared state. - </para> - - <section id='concepts-overall-architecture'> - <title>Overall Architecture</title> - - <para> - When determining what parts of the system need to be built, - BitBake works on a per-task basis rather than a per-recipe - basis. - You might wonder why using a per-task basis is preferred over - a per-recipe basis. - To help explain, consider having the IPK packaging backend - enabled and then switching to DEB. - In this case, the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> - task outputs are still valid. - However, with a per-recipe approach, the build would not - include the <filename>.deb</filename> files. - Consequently, you would have to invalidate the whole build and - rerun it. - Rerunning everything is not the best solution. - Also, in this case, the core must be "taught" much about - specific tasks. - This methodology does not scale well and does not allow users - to easily add new tasks in layers or as external recipes - without touching the packaged-staging core. - </para> - </section> - - <section id='overview-checksums'> - <title>Checksums (Signatures)</title> - - <para> - The shared state code uses a checksum, which is a unique - signature of a task's inputs, to determine if a task needs to - be run again. - Because it is a change in a task's inputs that triggers a - rerun, the process needs to detect all the inputs to a given - task. - For shell tasks, this turns out to be fairly easy because - the build process generates a "run" shell script for each task - and it is possible to create a checksum that gives you a good - idea of when the task's data changes. - </para> - - <para> - To complicate the problem, there are things that should not be - included in the checksum. - First, there is the actual specific build path of a given - task - the - <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>. - It does not matter if the work directory changes because it - should not affect the output for target packages. - Also, the build process has the objective of making native - or cross packages relocatable. - <note> - Both native and cross packages run on the - <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink>. - However, cross packages generate output for the target - architecture. - </note> - The checksum therefore needs to exclude - <filename>WORKDIR</filename>. - The simplistic approach for excluding the work directory is to - set <filename>WORKDIR</filename> to some fixed value and - create the checksum for the "run" script. - </para> - - <para> - Another problem results from the "run" scripts containing - functions that might or might not get called. - The incremental build solution contains code that figures out - dependencies between shell functions. - This code is used to prune the "run" scripts down to the - minimum set, thereby alleviating this problem and making the - "run" scripts much more readable as a bonus. - </para> - - <para> - So far, solutions for shell scripts exist. - What about Python tasks? - The same approach applies even though these tasks are more - difficult. - The process needs to figure out what variables a Python - function accesses and what functions it calls. - Again, the incremental build solution contains code that first - figures out the variable and function dependencies, and then - creates a checksum for the data used as the input to the task. - </para> - - <para> - Like the <filename>WORKDIR</filename> case, situations exist - where dependencies should be ignored. - For these situations, you can instruct the build process to - ignore a dependency by using a line like the following: - <literallayout class='monospaced'> - PACKAGE_ARCHS[vardepsexclude] = "MACHINE" - </literallayout> - This example ensures that the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink> - variable does not depend on the value of - <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>, - even if it does reference it. - </para> - - <para> - Equally, there are cases where you need to add dependencies - BitBake is not able to find. - You can accomplish this by using a line like the following: - <literallayout class='monospaced'> - PACKAGE_ARCHS[vardeps] = "MACHINE" - </literallayout> - This example explicitly adds the <filename>MACHINE</filename> - variable as a dependency for - <filename>PACKAGE_ARCHS</filename>. - </para> - - <para> - As an example, consider a case with in-line Python where - BitBake is not able to figure out dependencies. - When running in debug mode (i.e. using - <filename>-DDD</filename>), BitBake produces output when it - discovers something for which it cannot figure out dependencies. - The Yocto Project team has currently not managed to cover - those dependencies in detail and is aware of the need to fix - this situation. - </para> - - <para> - Thus far, this section has limited discussion to the direct - inputs into a task. - Information based on direct inputs is referred to as the - "basehash" in the code. - However, the question of a task's indirect inputs still - exits - items already built and present in the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. - The checksum (or signature) for a particular task needs to add - the hashes of all the tasks on which the particular task - depends. - Choosing which dependencies to add is a policy decision. - However, the effect is to generate a master checksum that - combines the basehash and the hashes of the task's - dependencies. - </para> - - <para> - At the code level, a variety of ways exist by which both the - basehash and the dependent task hashes can be influenced. - Within the BitBake configuration file, you can give BitBake - some extra information to help it construct the basehash. - The following statement effectively results in a list of - global variable dependency excludes (i.e. variables never - included in any checksum): - <literallayout class='monospaced'> - BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ - SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ - USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ - PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ - CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" - </literallayout> - The previous example excludes - <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink> - since that variable is actually constructed as a path within - <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>, - which is on the whitelist. - </para> - - <para> - The rules for deciding which hashes of dependent tasks to - include through dependency chains are more complex and are - generally accomplished with a Python function. - The code in <filename>meta/lib/oe/sstatesig.py</filename> shows - two examples of this and also illustrates how you can insert - your own policy into the system if so desired. - This file defines the two basic signature generators - <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink> - uses: "OEBasic" and "OEBasicHash". - By default, a dummy "noop" signature handler is enabled - in BitBake. - This means that behavior is unchanged from previous versions. - OE-Core uses the "OEBasicHash" signature handler by default - through this setting in the <filename>bitbake.conf</filename> - file: - <literallayout class='monospaced'> - BB_SIGNATURE_HANDLER ?= "OEBasicHash" - </literallayout> - The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> - is the same as the "OEBasic" version but adds the task hash to - the - <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp files</link>. - This results in any metadata change that changes the task hash, - automatically causing the task to be run again. - This removes the need to bump - <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> - values, and changes to metadata automatically ripple across - the build. - </para> - - <para> - It is also worth noting that the end result of these - signature generators is to make some dependency and hash - information available to the build. - This information includes: - <itemizedlist> - <listitem><para> - <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>: - The base hashes for each task in the recipe. - </para></listitem> - <listitem><para> - <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: - The base hashes for each dependent task. - </para></listitem> - <listitem><para> - <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: - The task dependencies for each task. - </para></listitem> - <listitem><para> - <filename>BB_TASKHASH</filename>: - The hash of the currently running task. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='shared-state'> - <title>Shared State</title> - - <para> - Checksums and dependencies, as discussed in the previous - section, solve half the problem of supporting a shared state. - The other half of the problem is being able to use checksum - information during the build and being able to reuse or rebuild - specific components. - </para> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink> - class is a relatively generic implementation of how to - "capture" a snapshot of a given task. - The idea is that the build process does not care about the - source of a task's output. - Output could be freshly built or it could be downloaded and - unpacked from somewhere. - In other words, the build process does not need to worry about - its origin. - </para> - - <para> - Two types of output exist. - One type is just about creating a directory in - <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>. - A good example is the output of either - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> - or - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>. - The other type of output occurs when a set of data is merged - into a shared directory tree such as the sysroot. - </para> - - <para> - The Yocto Project team has tried to keep the details of the - implementation hidden in <filename>sstate</filename> class. - From a user's perspective, adding shared state wrapping to a - task is as simple as this - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink> - example taken from the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink> - class: - <literallayout class='monospaced'> - DEPLOYDIR = "${WORKDIR}/deploy-${PN}" - SSTATETASKS += "do_deploy" - do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" - do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" - - python do_deploy_setscene () { - sstate_setscene(d) - } - addtask do_deploy_setscene - do_deploy[dirs] = "${DEPLOYDIR} ${B}" - do_deploy[stamp-extra-info] = "${MACHINE_ARCH}" - </literallayout> - The following list explains the previous example: - <itemizedlist> - <listitem><para> - Adding "do_deploy" to <filename>SSTATETASKS</filename> - adds some required sstate-related processing, which is - implemented in the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink> - class, to before and after the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink> - task. - </para></listitem> - <listitem><para> - The - <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename> - declares that <filename>do_deploy</filename> places its - output in <filename>${DEPLOYDIR}</filename> when run - normally (i.e. when not using the sstate cache). - This output becomes the input to the shared state cache. - </para></listitem> - <listitem><para> - The - <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename> - line causes the contents of the shared state cache to be - copied to <filename>${DEPLOY_DIR_IMAGE}</filename>. - <note> - If <filename>do_deploy</filename> is not already in - the shared state cache or if its input checksum - (signature) has changed from when the output was - cached, the task runs to populate the shared - state cache, after which the contents of the shared - state cache is copied to - <filename>${DEPLOY_DIR_IMAGE}</filename>. - If <filename>do_deploy</filename> is in the shared - state cache and its signature indicates that the - cached output is still valid (i.e. if no - relevant task inputs have changed), then the - contents of the shared state cache copies - directly to - <filename>${DEPLOY_DIR_IMAGE}</filename> by the - <filename>do_deploy_setscene</filename> task - instead, skipping the - <filename>do_deploy</filename> task. - </note> - </para></listitem> - <listitem><para> - The following task definition is glue logic needed to - make the previous settings effective: - <literallayout class='monospaced'> - python do_deploy_setscene () { - sstate_setscene(d) - } - addtask do_deploy_setscene - </literallayout> - <filename>sstate_setscene()</filename> takes the flags - above as input and accelerates the - <filename>do_deploy</filename> task through the - shared state cache if possible. - If the task was accelerated, - <filename>sstate_setscene()</filename> returns True. - Otherwise, it returns False, and the normal - <filename>do_deploy</filename> task runs. - For more information, see the - "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>" - section in the BitBake User Manual. - </para></listitem> - <listitem><para> - The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename> - line creates <filename>${DEPLOYDIR}</filename> and - <filename>${B}</filename> before the - <filename>do_deploy</filename> task runs, and also sets - the current working directory of - <filename>do_deploy</filename> to - <filename>${B}</filename>. - For more information, see the - "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>" - section in the BitBake User Manual. - <note> - In cases where - <filename>sstate-inputdirs</filename> and - <filename>sstate-outputdirs</filename> would be the - same, you can use - <filename>sstate-plaindirs</filename>. - For example, to preserve the - <filename>${PKGD}</filename> and - <filename>${PKGDEST}</filename> output from the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> - task, use the following: - <literallayout class='monospaced'> - do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" - </literallayout> - </note> - </para></listitem> - <listitem><para> - The <filename>do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"</filename> - line appends extra metadata to the - <link linkend='stamp-files-and-the-rerunning-of-tasks'>stamp file</link>. - In this case, the metadata makes the task specific - to a machine's architecture. - See - "<ulink url='&YOCTO_DOCS_BB_URL;#ref-bitbake-tasklist'>The Task List</ulink>" - section in the BitBake User Manual for more - information on the <filename>stamp-extra-info</filename> - flag. - </para></listitem> - <listitem><para> - <filename>sstate-inputdirs</filename> and - <filename>sstate-outputdirs</filename> can also be used - with multiple directories. - For example, the following declares - <filename>PKGDESTWORK</filename> and - <filename>SHLIBWORK</filename> as shared state - input directories, which populates the shared state - cache, and <filename>PKGDATA_DIR</filename> and - <filename>SHLIBSDIR</filename> as the corresponding - shared state output directories: - <literallayout class='monospaced'> - do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" - do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" - </literallayout> - </para></listitem> - <listitem><para> - These methods also include the ability to take a - lockfile when manipulating shared state directory - structures, for cases where file additions or removals - are sensitive: - <literallayout class='monospaced'> - do_package[sstate-lockfile] = "${PACKAGELOCK}" - </literallayout> - </para></listitem> - </itemizedlist> - </para> - - <para> - Behind the scenes, the shared state code works by looking in - <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink> - for shared state files. - Here is an example: - <literallayout class='monospaced'> - SSTATE_MIRRORS ?= "\ - file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ - file://.* file:///some/local/dir/sstate/PATH" - </literallayout> - <note> - The shared state directory - (<filename>SSTATE_DIR</filename>) is organized into - two-character subdirectories, where the subdirectory - names are based on the first two characters of the hash. - If the shared state directory structure for a mirror has the - same structure as <filename>SSTATE_DIR</filename>, you must - specify "PATH" as part of the URI to enable the build system - to map to the appropriate subdirectory. - </note> - </para> - - <para> - The shared state package validity can be detected just by - looking at the filename since the filename contains the task - checksum (or signature) as described earlier in this section. - If a valid shared state package is found, the build process - downloads it and uses it to accelerate the task. - </para> - - <para> - The build processes use the <filename>*_setscene</filename> - tasks for the task acceleration phase. - BitBake goes through this phase before the main execution - code and tries to accelerate any tasks for which it can find - shared state packages. - If a shared state package for a task is available, the - shared state package is used. - This means the task and any tasks on which it is dependent - are not executed. - </para> - - <para> - As a real world example, the aim is when building an IPK-based - image, only the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink> - tasks would have their shared state packages fetched and - extracted. - Since the sysroot is not used, it would never get extracted. - This is another reason why a task-based approach is preferred - over a recipe-based approach, which would have to install the - output from every task. - </para> - </section> - </section> - - <section id='automatically-added-runtime-dependencies'> - <title>Automatically Added Runtime Dependencies</title> - - <para> - The OpenEmbedded build system automatically adds common types of - runtime dependencies between packages, which means that you do not - need to explicitly declare the packages using - <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>. - Three automatic mechanisms exist (<filename>shlibdeps</filename>, - <filename>pcdeps</filename>, and <filename>depchains</filename>) - that handle shared libraries, package configuration (pkg-config) - modules, and <filename>-dev</filename> and - <filename>-dbg</filename> packages, respectively. - For other types of runtime dependencies, you must manually declare - the dependencies. - <itemizedlist> - <listitem><para> - <filename>shlibdeps</filename>: - During the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> - task of each recipe, all shared libraries installed by the - recipe are located. - For each shared library, the package that contains the - shared library is registered as providing the shared - library. - More specifically, the package is registered as providing - the - <ulink url='https://en.wikipedia.org/wiki/Soname'>soname</ulink> - of the library. - The resulting shared-library-to-package mapping - is saved globally in - <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink> - by the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink> - task.</para> - - <para>Simultaneously, all executables and shared libraries - installed by the recipe are inspected to see what shared - libraries they link against. - For each shared library dependency that is found, - <filename>PKGDATA_DIR</filename> is queried to - see if some package (likely from a different recipe) - contains the shared library. - If such a package is found, a runtime dependency is added - from the package that depends on the shared library to the - package that contains the library.</para> - - <para>The automatically added runtime dependency also - includes a version restriction. - This version restriction specifies that at least the - current version of the package that provides the shared - library must be used, as if - "<replaceable>package</replaceable> (>= <replaceable>version</replaceable>)" - had been added to <filename>RDEPENDS</filename>. - This forces an upgrade of the package containing the shared - library when installing the package that depends on the - library, if needed.</para> - - <para>If you want to avoid a package being registered as - providing a particular shared library (e.g. because the library - is for internal use only), then add the library to - <ulink url='&YOCTO_DOCS_REF_URL;#var-PRIVATE_LIBS'><filename>PRIVATE_LIBS</filename></ulink> - inside the package's recipe. - </para></listitem> - <listitem><para> - <filename>pcdeps</filename>: - During the <filename>do_package</filename> task of each - recipe, all pkg-config modules - (<filename>*.pc</filename> files) installed by the recipe - are located. - For each module, the package that contains the module is - registered as providing the module. - The resulting module-to-package mapping is saved globally in - <filename>PKGDATA_DIR</filename> by the - <filename>do_packagedata</filename> task.</para> - - <para>Simultaneously, all pkg-config modules installed by - the recipe are inspected to see what other pkg-config - modules they depend on. - A module is seen as depending on another module if it - contains a "Requires:" line that specifies the other module. - For each module dependency, - <filename>PKGDATA_DIR</filename> is queried to see if some - package contains the module. - If such a package is found, a runtime dependency is added - from the package that depends on the module to the package - that contains the module. - <note> - The <filename>pcdeps</filename> mechanism most often - infers dependencies between <filename>-dev</filename> - packages. - </note> - </para></listitem> - <listitem><para> - <filename>depchains</filename>: - If a package <filename>foo</filename> depends on a package - <filename>bar</filename>, then <filename>foo-dev</filename> - and <filename>foo-dbg</filename> are also made to depend on - <filename>bar-dev</filename> and - <filename>bar-dbg</filename>, respectively. - Taking the <filename>-dev</filename> packages as an - example, the <filename>bar-dev</filename> package might - provide headers and shared library symlinks needed by - <filename>foo-dev</filename>, which shows the need - for a dependency between the packages.</para> - - <para>The dependencies added by - <filename>depchains</filename> are in the form of - <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>. - <note> - By default, <filename>foo-dev</filename> also has an - <filename>RDEPENDS</filename>-style dependency on - <filename>foo</filename>, because the default value of - <filename>RDEPENDS_${PN}-dev</filename> (set in - <filename>bitbake.conf</filename>) includes - "${PN}". - </note></para> - - <para>To ensure that the dependency chain is never broken, - <filename>-dev</filename> and <filename>-dbg</filename> - packages are always generated by default, even if the - packages turn out to be empty. - See the - <ulink url='&YOCTO_DOCS_REF_URL;#var-ALLOW_EMPTY'><filename>ALLOW_EMPTY</filename></ulink> - variable for more information. - </para></listitem> - </itemizedlist> - </para> - - <para> - The <filename>do_package</filename> task depends on the - <filename>do_packagedata</filename> task of each recipe in - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> - through use of a - <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>deptask</filename></ulink><filename>]</filename> - declaration, which guarantees that the required - shared-library/module-to-package mapping information will be available - when needed as long as <filename>DEPENDS</filename> has been - correctly set. - </para> - </section> - - <section id='fakeroot-and-pseudo'> - <title>Fakeroot and Pseudo</title> - - <para> - Some tasks are easier to implement when allowed to perform certain - operations that are normally reserved for the root user (e.g. - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write*</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>, - and - <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image*</filename></ulink>). - For example, the <filename>do_install</filename> task benefits - from being able to set the UID and GID of installed files to - arbitrary values. - </para> - - <para> - One approach to allowing tasks to perform root-only operations - would be to require - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> - to run as root. - However, this method is cumbersome and has security issues. - The approach that is actually used is to run tasks that benefit - from root privileges in a "fake" root environment. - Within this environment, the task and its child processes believe - that they are running as the root user, and see an internally - consistent view of the filesystem. - As long as generating the final output (e.g. a package or an image) - does not require root privileges, the fact that some earlier - steps ran in a fake root environment does not cause problems. - </para> - - <para> - The capability to run tasks in a fake root environment is known as - "<ulink url='http://man.he.net/man1/fakeroot'>fakeroot</ulink>", - which is derived from the BitBake keyword/variable - flag that requests a fake root environment for a task. - </para> - - <para> - In the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>, - the program that implements fakeroot is known as - <ulink url='https://www.yoctoproject.org/software-item/pseudo/'>Pseudo</ulink>. - Pseudo overrides system calls by using the environment variable - <filename>LD_PRELOAD</filename>, which results in the illusion - of running as root. - To keep track of "fake" file ownership and permissions resulting - from operations that require root permissions, Pseudo uses - an SQLite 3 database. - This database is stored in - <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/pseudo/files.db</filename> - for individual recipes. - Storing the database in a file as opposed to in memory - gives persistence between tasks and builds, which is not - accomplished using fakeroot. - <note><title>Caution</title> - If you add your own task that manipulates the same files or - directories as a fakeroot task, then that task also needs to - run under fakeroot. - Otherwise, the task cannot run root-only operations, and - cannot see the fake file ownership and permissions set by the - other task. - You need to also add a dependency on - <filename>virtual/fakeroot-native:do_populate_sysroot</filename>, - giving the following: - <literallayout class='monospaced'> - fakeroot do_mytask () { - ... - } - do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot" - </literallayout> - </note> - For more information, see the - <ulink url='&YOCTO_DOCS_BB_URL;#var-FAKEROOT'><filename>FAKEROOT*</filename></ulink> - variables in the BitBake User Manual. - You can also reference the - "<ulink url='https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot'>Why Not Fakeroot?</ulink>" - article for background information on Fakeroot and Pseudo. - </para> - </section> -</chapter> -<!-- -vim: expandtab tw=80 ts=4 ---> diff --git a/poky/documentation/overview-manual/overview-manual-customization.xsl b/poky/documentation/overview-manual/overview-manual-customization.xsl deleted file mode 100644 index 22360e7bab..0000000000 --- a/poky/documentation/overview-manual/overview-manual-customization.xsl +++ /dev/null @@ -1,27 +0,0 @@ -<?xml version='1.0'?> -<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" xmlns:fo="http://www.w3.org/1999/XSL/Format" version="1.0"> - - <xsl:import href="http://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-xsl-1.76.1/xhtml/docbook.xsl" /> - -<!-- - - <xsl:import href="../template/1.76.1/docbook-xsl-1.76.1/xhtml/docbook.xsl" /> - - <xsl:import href="http://docbook.sourceforge.net/release/xsl/1.76.1/xhtml/docbook.xsl" /> - ---> - - <xsl:include href="../template/permalinks.xsl"/> - <xsl:include href="../template/section.title.xsl"/> - <xsl:include href="../template/component.title.xsl"/> - <xsl:include href="../template/division.title.xsl"/> - <xsl:include href="../template/formal.object.heading.xsl"/> - - <xsl:param name="html.stylesheet" select="'overview-manual-style.css'" /> - <xsl:param name="chapter.autolabel" select="1" /> - <xsl:param name="appendix.autolabel" select="A" /> - <xsl:param name="section.autolabel" select="1" /> - <xsl:param name="section.label.includes.component.label" select="1" /> - <xsl:param name="generate.id.attributes" select="1" /> - -</xsl:stylesheet> diff --git a/poky/documentation/overview-manual/overview-manual-development-environment.rst b/poky/documentation/overview-manual/overview-manual-development-environment.rst new file mode 100644 index 0000000000..4bedd6df67 --- /dev/null +++ b/poky/documentation/overview-manual/overview-manual-development-environment.rst @@ -0,0 +1,672 @@ +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK + +***************************************** +The Yocto Project Development Environment +***************************************** + +This chapter takes a look at the Yocto Project development environment. +The chapter provides Yocto Project Development environment concepts that +help you understand how work is accomplished in an open source +environment, which is very different as compared to work accomplished in +a closed, proprietary environment. + +Specifically, this chapter addresses open source philosophy, source +repositories, workflows, Git, and licensing. + +Open Source Philosophy +====================== + +Open source philosophy is characterized by software development directed +by peer production and collaboration through an active community of +developers. Contrast this to the more standard centralized development +models used by commercial software companies where a finite set of +developers produces a product for sale using a defined set of procedures +that ultimately result in an end product whose architecture and source +material are closed to the public. + +Open source projects conceptually have differing concurrent agendas, +approaches, and production. These facets of the development process can +come from anyone in the public (community) who has a stake in the +software project. The open source environment contains new copyright, +licensing, domain, and consumer issues that differ from the more +traditional development environment. In an open source environment, the +end product, source material, and documentation are all available to the +public at no cost. + +A benchmark example of an open source project is the Linux kernel, which +was initially conceived and created by Finnish computer science student +Linus Torvalds in 1991. Conversely, a good example of a non-open source +project is the Windows family of operating systems developed by +Microsoft Corporation. + +Wikipedia has a good historical description of the Open Source +Philosophy `here <http://en.wikipedia.org/wiki/Open_source>`__. You can +also find helpful information on how to participate in the Linux +Community +`here <https://www.kernel.org/doc/html/latest/process/index.html>`__. + +.. _gs-the-development-host: + +The Development Host +==================== + +A development host or :term:`Build Host` is key to +using the Yocto Project. Because the goal of the Yocto Project is to +develop images or applications that run on embedded hardware, +development of those images and applications generally takes place on a +system not intended to run the software - the development host. + +You need to set up a development host in order to use it with the Yocto +Project. Most find that it is best to have a native Linux machine +function as the development host. However, it is possible to use a +system that does not run Linux as its operating system as your +development host. When you have a Mac or Windows-based system, you can +set it up as the development host by using +`CROPS <https://github.com/crops/poky-container>`__, which leverages +`Docker Containers <https://www.docker.com/>`__. Once you take the steps +to set up a CROPS machine, you effectively have access to a shell +environment that is similar to what you see when using a Linux-based +development host. For the steps needed to set up a system using CROPS, +see the +":ref:`dev-manual/dev-manual-start:setting up to use cross platforms (crops)`" +section in +the Yocto Project Development Tasks Manual. + +If your development host is going to be a system that runs a Linux +distribution, steps still exist that you must take to prepare the system +for use with the Yocto Project. You need to be sure that the Linux +distribution on the system is one that supports the Yocto Project. You +also need to be sure that the correct set of host packages are installed +that allow development using the Yocto Project. For the steps needed to +set up a development host that runs Linux, see the +":ref:`dev-manual/dev-manual-start:setting up a native linux host`" +section in the Yocto Project Development Tasks Manual. + +Once your development host is set up to use the Yocto Project, several +methods exist for you to do work in the Yocto Project environment: + +- *Command Lines, BitBake, and Shells:* Traditional development in the + Yocto Project involves using the :term:`OpenEmbedded Build System`, + which uses + BitBake, in a command-line environment from a shell on your + development host. You can accomplish this from a host that is a + native Linux machine or from a host that has been set up with CROPS. + Either way, you create, modify, and build images and applications all + within a shell-based environment using components and tools available + through your Linux distribution and the Yocto Project. + + For a general flow of the build procedures, see the + ":ref:`dev-manual/dev-manual-common-tasks:building a simple image`" + section in the Yocto Project Development Tasks Manual. + +- *Board Support Package (BSP) Development:* Development of BSPs + involves using the Yocto Project to create and test layers that allow + easy development of images and applications targeted for specific + hardware. To development BSPs, you need to take some additional steps + beyond what was described in setting up a development host. + + The :doc:`../bsp-guide/bsp-guide` provides BSP-related development + information. For specifics on development host preparation, see the + ":ref:`bsp-guide/bsp:preparing your build host to work with bsp layers`" + section in the Yocto Project Board Support Package (BSP) Developer's + Guide. + +- *Kernel Development:* If you are going to be developing kernels using + the Yocto Project you likely will be using ``devtool``. A workflow + using ``devtool`` makes kernel development quicker by reducing + iteration cycle times. + + The :doc:`../kernel-dev/kernel-dev` provides kernel-related + development information. For specifics on development host + preparation, see the + ":ref:`kernel-dev/kernel-dev-common:preparing the build host to work on the kernel`" + section in the Yocto Project Linux Kernel Development Manual. + +- *Using Toaster:* The other Yocto Project development method that + involves an interface that effectively puts the Yocto Project into + the background is Toaster. Toaster provides an interface to the + OpenEmbedded build system. The interface enables you to configure and + run your builds. Information about builds is collected and stored in + a database. You can use Toaster to configure and start builds on + multiple remote build servers. + + For steps that show you how to set up your development host to use + Toaster and on how to use Toaster in general, see the + :doc:`../toaster-manual/toaster-manual`. + +.. _yocto-project-repositories: + +Yocto Project Source Repositories +================================= + +The Yocto Project team maintains complete source repositories for all +Yocto Project files at :yocto_git:`/`. This web-based source +code browser is organized into categories by function such as IDE +Plugins, Matchbox, Poky, Yocto Linux Kernel, and so forth. From the +interface, you can click on any particular item in the "Name" column and +see the URL at the bottom of the page that you need to clone a Git +repository for that particular item. Having a local Git repository of +the :term:`Source Directory`, which +is usually named "poky", allows you to make changes, contribute to the +history, and ultimately enhance the Yocto Project's tools, Board Support +Packages, and so forth. + +For any supported release of Yocto Project, you can also go to the +:yocto_home:`Yocto Project Website <>` and select the "DOWNLOADS" +item from the "SOFTWARE" menu and get a released tarball of the ``poky`` +repository, any supported BSP tarball, or Yocto Project tools. Unpacking +these tarballs gives you a snapshot of the released files. + +.. note:: + + - The recommended method for setting up the Yocto Project + :term:`Source Directory` and the files + for supported BSPs (e.g., ``meta-intel``) is to use `Git <#git>`__ + to create a local copy of the upstream repositories. + + - Be sure to always work in matching branches for both the selected + BSP repository and the Source Directory (i.e. ``poky``) + repository. For example, if you have checked out the "master" + branch of ``poky`` and you are going to use ``meta-intel``, be + sure to checkout the "master" branch of ``meta-intel``. + +In summary, here is where you can get the project files needed for +development: + +- :yocto_git:`Source Repositories: <>` This area contains IDE + Plugins, Matchbox, Poky, Poky Support, Tools, Yocto Linux Kernel, and + Yocto Metadata Layers. You can create local copies of Git + repositories for each of these areas. + + .. image:: figures/source-repos.png + :align: center + + For steps on how to view and access these upstream Git repositories, + see the ":ref:`dev-manual/dev-manual-start:accessing source repositories`" + Section in the Yocto Project Development Tasks Manual. + +- :yocto_dl:`Index of /releases: </releases>` This is an index + of releases such as Poky, Pseudo, installers for cross-development + toolchains, miscellaneous support and all released versions of Yocto + Project in the form of images or tarballs. Downloading and extracting + these files does not produce a local copy of the Git repository but + rather a snapshot of a particular release or image. + + .. image:: figures/index-downloads.png + :align: center + + For steps on how to view and access these files, see the + ":ref:`dev-manual/dev-manual-start:accessing index of releases`" + section in the Yocto Project Development Tasks Manual. + +- *"DOWNLOADS" page for the* :yocto_home:`Yocto Project Website <>` *:* + + The Yocto Project website includes a "DOWNLOADS" page accessible + through the "SOFTWARE" menu that allows you to download any Yocto + Project release, tool, and Board Support Package (BSP) in tarball + form. The tarballs are similar to those found in the + :yocto_dl:`Index of /releases: </releases>` area. + + .. image:: figures/yp-download.png + :align: center + + For steps on how to use the "DOWNLOADS" page, see the + ":ref:`dev-manual/dev-manual-start:using the downloads page`" + section in the Yocto Project Development Tasks Manual. + +.. _gs-git-workflows-and-the-yocto-project: + +Git Workflows and the Yocto Project +=================================== + +Developing using the Yocto Project likely requires the use of +`Git <#git>`__. Git is a free, open source distributed version control +system used as part of many collaborative design environments. This +section provides workflow concepts using the Yocto Project and Git. In +particular, the information covers basic practices that describe roles +and actions in a collaborative development environment. + +.. note:: + + If you are familiar with this type of development environment, you + might not want to read this section. + +The Yocto Project files are maintained using Git in "branches" whose Git +histories track every change and whose structures provide branches for +all diverging functionality. Although there is no need to use Git, many +open source projects do so. + +For the Yocto Project, a key individual called the "maintainer" is +responsible for the integrity of the "master" branch of a given Git +repository. The "master" branch is the "upstream" repository from which +final or most recent builds of a project occur. The maintainer is +responsible for accepting changes from other developers and for +organizing the underlying branch structure to reflect release strategies +and so forth. + +.. note:: + + For information on finding out who is responsible for (maintains) a + particular area of code in the Yocto Project, see the + ":ref:`dev-manual/dev-manual-common-tasks:submitting a change to the yocto project`" + section of the Yocto Project Development Tasks Manual. + +The Yocto Project ``poky`` Git repository also has an upstream +contribution Git repository named ``poky-contrib``. You can see all the +branches in this repository using the web interface of the +:yocto_git:`Source Repositories <>` organized within the "Poky Support" +area. These branches hold changes (commits) to the project that have +been submitted or committed by the Yocto Project development team and by +community members who contribute to the project. The maintainer +determines if the changes are qualified to be moved from the "contrib" +branches into the "master" branch of the Git repository. + +Developers (including contributing community members) create and +maintain cloned repositories of upstream branches. The cloned +repositories are local to their development platforms and are used to +develop changes. When a developer is satisfied with a particular feature +or change, they "push" the change to the appropriate "contrib" +repository. + +Developers are responsible for keeping their local repository up-to-date +with whatever upstream branch they are working against. They are also +responsible for straightening out any conflicts that might arise within +files that are being worked on simultaneously by more than one person. +All this work is done locally on the development host before anything is +pushed to a "contrib" area and examined at the maintainer's level. + +A somewhat formal method exists by which developers commit changes and +push them into the "contrib" area and subsequently request that the +maintainer include them into an upstream branch. This process is called +"submitting a patch" or "submitting a change." For information on +submitting patches and changes, see the +":ref:`dev-manual/dev-manual-common-tasks:submitting a change to the yocto project`" +section in the Yocto Project Development Tasks Manual. + +In summary, a single point of entry exists for changes into a "master" +or development branch of the Git repository, which is controlled by the +project's maintainer. And, a set of developers exist who independently +develop, test, and submit changes to "contrib" areas for the maintainer +to examine. The maintainer then chooses which changes are going to +become a permanent part of the project. + +.. image:: figures/git-workflow.png + :align: center + +While each development environment is unique, there are some best +practices or methods that help development run smoothly. The following +list describes some of these practices. For more information about Git +workflows, see the workflow topics in the `Git Community +Book <http://book.git-scm.com>`__. + +- *Make Small Changes:* It is best to keep the changes you commit small + as compared to bundling many disparate changes into a single commit. + This practice not only keeps things manageable but also allows the + maintainer to more easily include or refuse changes. + +- *Make Complete Changes:* It is also good practice to leave the + repository in a state that allows you to still successfully build + your project. In other words, do not commit half of a feature, then + add the other half as a separate, later commit. Each commit should + take you from one buildable project state to another buildable state. + +- *Use Branches Liberally:* It is very easy to create, use, and delete + local branches in your working Git repository on the development + host. You can name these branches anything you like. It is helpful to + give them names associated with the particular feature or change on + which you are working. Once you are done with a feature or change and + have merged it into your local master branch, simply discard the + temporary branch. + +- *Merge Changes:* The ``git merge`` command allows you to take the + changes from one branch and fold them into another branch. This + process is especially helpful when more than a single developer might + be working on different parts of the same feature. Merging changes + also automatically identifies any collisions or "conflicts" that + might happen as a result of the same lines of code being altered by + two different developers. + +- *Manage Branches:* Because branches are easy to use, you should use a + system where branches indicate varying levels of code readiness. For + example, you can have a "work" branch to develop in, a "test" branch + where the code or change is tested, a "stage" branch where changes + are ready to be committed, and so forth. As your project develops, + you can merge code across the branches to reflect ever-increasing + stable states of the development. + +- *Use Push and Pull:* The push-pull workflow is based on the concept + of developers "pushing" local commits to a remote repository, which + is usually a contribution repository. This workflow is also based on + developers "pulling" known states of the project down into their + local development repositories. The workflow easily allows you to + pull changes submitted by other developers from the upstream + repository into your work area ensuring that you have the most recent + software on which to develop. The Yocto Project has two scripts named + ``create-pull-request`` and ``send-pull-request`` that ship with the + release to facilitate this workflow. You can find these scripts in + the ``scripts`` folder of the + :term:`Source Directory`. For information + on how to use these scripts, see the + ":ref:`dev-manual/dev-manual-common-tasks:using scripts to push a change upstream and request a pull`" + section in the Yocto Project Development Tasks Manual. + +- *Patch Workflow:* This workflow allows you to notify the maintainer + through an email that you have a change (or patch) you would like + considered for the "master" branch of the Git repository. To send + this type of change, you format the patch and then send the email + using the Git commands ``git format-patch`` and ``git send-email``. + For information on how to use these scripts, see the + ":ref:`dev-manual/dev-manual-common-tasks:submitting a change to the yocto project`" + section in the Yocto Project Development Tasks Manual. + +Git +=== + +The Yocto Project makes extensive use of Git, which is a free, open +source distributed version control system. Git supports distributed +development, non-linear development, and can handle large projects. It +is best that you have some fundamental understanding of how Git tracks +projects and how to work with Git if you are going to use the Yocto +Project for development. This section provides a quick overview of how +Git works and provides you with a summary of some essential Git +commands. + +.. note:: + + - For more information on Git, see + http://git-scm.com/documentation. + + - If you need to download Git, it is recommended that you add Git to + your system through your distribution's "software store" (e.g. for + Ubuntu, use the Ubuntu Software feature). For the Git download + page, see http://git-scm.com/download. + + - For information beyond the introductory nature in this section, + see the ":ref:`dev-manual/dev-manual-start:locating yocto project source files`" + section in the Yocto Project Development Tasks Manual. + +Repositories, Tags, and Branches +-------------------------------- + +As mentioned briefly in the previous section and also in the "`Git +Workflows and the Yocto +Project <#gs-git-workflows-and-the-yocto-project>`__" section, the Yocto +Project maintains source repositories at :yocto_git:`/`. If you +look at this web-interface of the repositories, each item is a separate +Git repository. + +Git repositories use branching techniques that track content change (not +files) within a project (e.g. a new feature or updated documentation). +Creating a tree-like structure based on project divergence allows for +excellent historical information over the life of a project. This +methodology also allows for an environment from which you can do lots of +local experimentation on projects as you develop changes or new +features. + +A Git repository represents all development efforts for a given project. +For example, the Git repository ``poky`` contains all changes and +developments for that repository over the course of its entire life. +That means that all changes that make up all releases are captured. The +repository maintains a complete history of changes. + +You can create a local copy of any repository by "cloning" it with the +``git clone`` command. When you clone a Git repository, you end up with +an identical copy of the repository on your development system. Once you +have a local copy of a repository, you can take steps to develop +locally. For examples on how to clone Git repositories, see the +":ref:`dev-manual/dev-manual-start:locating yocto project source files`" +section in the Yocto Project Development Tasks Manual. + +It is important to understand that Git tracks content change and not +files. Git uses "branches" to organize different development efforts. +For example, the ``poky`` repository has several branches that include +the current "&DISTRO_NAME_NO_CAP;" branch, the "master" branch, and many +branches for past Yocto Project releases. You can see all the branches +by going to https://git.yoctoproject.org/cgit.cgi/poky/ and clicking on the +``[...]`` link beneath the "Branch" heading. + +Each of these branches represents a specific area of development. The +"master" branch represents the current or most recent development. All +other branches represent offshoots of the "master" branch. + +When you create a local copy of a Git repository, the copy has the same +set of branches as the original. This means you can use Git to create a +local working area (also called a branch) that tracks a specific +development branch from the upstream source Git repository. in other +words, you can define your local Git environment to work on any +development branch in the repository. To help illustrate, consider the +following example Git commands: +:: + + $ cd ~ + $ git clone git://git.yoctoproject.org/poky + $ cd poky + $ git checkout -b &DISTRO_NAME_NO_CAP; origin/&DISTRO_NAME_NO_CAP; + +In the previous example +after moving to the home directory, the ``git clone`` command creates a +local copy of the upstream ``poky`` Git repository. By default, Git +checks out the "master" branch for your work. After changing the working +directory to the new local repository (i.e. ``poky``), the +``git checkout`` command creates and checks out a local branch named +"&DISTRO_NAME_NO_CAP;", which tracks the upstream +"origin/&DISTRO_NAME_NO_CAP;" branch. Changes you make while in this +branch would ultimately affect the upstream "&DISTRO_NAME_NO_CAP;" branch +of the ``poky`` repository. + +It is important to understand that when you create and checkout a local +working branch based on a branch name, your local environment matches +the "tip" of that particular development branch at the time you created +your local branch, which could be different from the files in the +"master" branch of the upstream repository. In other words, creating and +checking out a local branch based on the "&DISTRO_NAME_NO_CAP;" branch +name is not the same as checking out the "master" branch in the +repository. Keep reading to see how you create a local snapshot of a +Yocto Project Release. + +Git uses "tags" to mark specific changes in a repository branch +structure. Typically, a tag is used to mark a special point such as the +final change (or commit) before a project is released. You can see the +tags used with the ``poky`` Git repository by going to +https://git.yoctoproject.org/cgit.cgi/poky/ and clicking on the ``[...]`` link +beneath the "Tag" heading. + +Some key tags for the ``poky`` repository are ``jethro-14.0.3``, +``morty-16.0.1``, ``pyro-17.0.0``, and +``&DISTRO_NAME_NO_CAP;-&POKYVERSION;``. These tags represent Yocto Project +releases. + +When you create a local copy of the Git repository, you also have access +to all the tags in the upstream repository. Similar to branches, you can +create and checkout a local working Git branch based on a tag name. When +you do this, you get a snapshot of the Git repository that reflects the +state of the files when the change was made associated with that tag. +The most common use is to checkout a working branch that matches a +specific Yocto Project release. Here is an example: +:: + + $ cd ~ + $ git clone git://git.yoctoproject.org/poky + $ cd poky + $ git fetch --tags + $ git checkout tags/rocko-18.0.0 -b my_rocko-18.0.0 + +In this example, the name +of the top-level directory of your local Yocto Project repository is +``poky``. After moving to the ``poky`` directory, the ``git fetch`` +command makes all the upstream tags available locally in your +repository. Finally, the ``git checkout`` command creates and checks out +a branch named "my-rocko-18.0.0" that is based on the upstream branch +whose "HEAD" matches the commit in the repository associated with the +"rocko-18.0.0" tag. The files in your repository now exactly match that +particular Yocto Project release as it is tagged in the upstream Git +repository. It is important to understand that when you create and +checkout a local working branch based on a tag, your environment matches +a specific point in time and not the entire development branch (i.e. +from the "tip" of the branch backwards). + +Basic Commands +-------------- + +Git has an extensive set of commands that lets you manage changes and +perform collaboration over the life of a project. Conveniently though, +you can manage with a small set of basic operations and workflows once +you understand the basic philosophy behind Git. You do not have to be an +expert in Git to be functional. A good place to look for instruction on +a minimal set of Git commands is +`here <http://git-scm.com/documentation>`__. + +The following list of Git commands briefly describes some basic Git +operations as a way to get started. As with any set of commands, this +list (in most cases) simply shows the base command and omits the many +arguments it supports. See the Git documentation for complete +descriptions and strategies on how to use these commands: + +- *git init:* Initializes an empty Git repository. You cannot use + Git commands unless you have a ``.git`` repository. + +- *git clone:* Creates a local clone of a Git repository that is on + equal footing with a fellow developer's Git repository or an upstream + repository. + +- *git add:* Locally stages updated file contents to the index that + Git uses to track changes. You must stage all files that have changed + before you can commit them. + +- *git commit:* Creates a local "commit" that documents the changes + you made. Only changes that have been staged can be committed. + Commits are used for historical purposes, for determining if a + maintainer of a project will allow the change, and for ultimately + pushing the change from your local Git repository into the project's + upstream repository. + +- *git status:* Reports any modified files that possibly need to be + staged and gives you a status of where you stand regarding local + commits as compared to the upstream repository. + +- *git checkout branch-name:* Changes your local working branch and + in this form assumes the local branch already exists. This command is + analogous to "cd". + +- *git checkout –b working-branch upstream-branch:* Creates and + checks out a working branch on your local machine. The local branch + tracks the upstream branch. You can use your local branch to isolate + your work. It is a good idea to use local branches when adding + specific features or changes. Using isolated branches facilitates + easy removal of changes if they do not work out. + +- *git branch:* Displays the existing local branches associated + with your local repository. The branch that you have currently + checked out is noted with an asterisk character. + +- *git branch -D branch-name:* Deletes an existing local branch. + You need to be in a local branch other than the one you are deleting + in order to delete branch-name. + +- *git pull --rebase:* Retrieves information from an upstream Git + repository and places it in your local Git repository. You use this + command to make sure you are synchronized with the repository from + which you are basing changes (.e.g. the "master" branch). The + "--rebase" option ensures that any local commits you have in your + branch are preserved at the top of your local branch. + +- *git push repo-name local-branch:upstream-branch:* Sends + all your committed local changes to the upstream Git repository that + your local repository is tracking (e.g. a contribution repository). + The maintainer of the project draws from these repositories to merge + changes (commits) into the appropriate branch of project's upstream + repository. + +- *git merge:* Combines or adds changes from one local branch of + your repository with another branch. When you create a local Git + repository, the default branch is named "master". A typical workflow + is to create a temporary branch that is based off "master" that you + would use for isolated work. You would make your changes in that + isolated branch, stage and commit them locally, switch to the + "master" branch, and then use the ``git merge`` command to apply the + changes from your isolated branch into the currently checked out + branch (e.g. "master"). After the merge is complete and if you are + done with working in that isolated branch, you can safely delete the + isolated branch. + +- *git cherry-pick commits:* Choose and apply specific commits from + one branch into another branch. There are times when you might not be + able to merge all the changes in one branch with another but need to + pick out certain ones. + +- *gitk:* Provides a GUI view of the branches and changes in your + local Git repository. This command is a good way to graphically see + where things have diverged in your local repository. + + .. note:: + + You need to install the + gitk + package on your development system to use this command. + +- *git log:* Reports a history of your commits to the repository. + This report lists all commits regardless of whether you have pushed + them upstream or not. + +- *git diff:* Displays line-by-line differences between a local + working file and the same file as understood by Git. This command is + useful to see what you have changed in any given file. + +Licensing +========= + +Because open source projects are open to the public, they have different +licensing structures in place. License evolution for both Open Source +and Free Software has an interesting history. If you are interested in +this history, you can find basic information here: + +- `Open source license + history <http://en.wikipedia.org/wiki/Open-source_license>`__ + +- `Free software license + history <http://en.wikipedia.org/wiki/Free_software_license>`__ + +In general, the Yocto Project is broadly licensed under the +Massachusetts Institute of Technology (MIT) License. MIT licensing +permits the reuse of software within proprietary software as long as the +license is distributed with that software. MIT is also compatible with +the GNU General Public License (GPL). Patches to the Yocto Project +follow the upstream licensing scheme. You can find information on the +MIT license +`here <http://www.opensource.org/licenses/mit-license.php>`__. You can +find information on the GNU GPL +`here <http://www.opensource.org/licenses/LGPL-3.0>`__. + +When you build an image using the Yocto Project, the build process uses +a known list of licenses to ensure compliance. You can find this list in +the :term:`Source Directory` at +``meta/files/common-licenses``. Once the build completes, the list of +all licenses found and used during that build are kept in the +:term:`Build Directory` at +``tmp/deploy/licenses``. + +If a module requires a license that is not in the base list, the build +process generates a warning during the build. These tools make it easier +for a developer to be certain of the licenses with which their shipped +products must comply. However, even with these tools it is still up to +the developer to resolve potential licensing issues. + +The base list of licenses used by the build process is a combination of +the Software Package Data Exchange (SPDX) list and the Open Source +Initiative (OSI) projects. `SPDX Group <http://spdx.org>`__ is a working +group of the Linux Foundation that maintains a specification for a +standard format for communicating the components, licenses, and +copyrights associated with a software package. +`OSI <http://opensource.org>`__ is a corporation dedicated to the Open +Source Definition and the effort for reviewing and approving licenses +that conform to the Open Source Definition (OSD). + +You can find a list of the combined SPDX and OSI licenses that the Yocto +Project uses in the ``meta/files/common-licenses`` directory in your +:term:`Source Directory`. + +For information that can help you maintain compliance with various open +source licensing during the lifecycle of a product created using the +Yocto Project, see the +":ref:`dev-manual/dev-manual-common-tasks:maintaining open source license compliance during your product's lifecycle`" +section in the Yocto Project Development Tasks Manual. diff --git a/poky/documentation/overview-manual/overview-manual-development-environment.xml b/poky/documentation/overview-manual/overview-manual-development-environment.xml deleted file mode 100644 index 36ebf8a321..0000000000 --- a/poky/documentation/overview-manual/overview-manual-development-environment.xml +++ /dev/null @@ -1,953 +0,0 @@ -<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" -"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" -[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > - -<chapter id='overview-development-environment'> -<title>The Yocto Project Development Environment</title> - -<para> - This chapter takes a look at the Yocto Project development - environment. - The chapter provides Yocto Project Development environment concepts that - help you understand how work is accomplished in an open source environment, - which is very different as compared to work accomplished in a closed, - proprietary environment. -</para> - -<para> - Specifically, this chapter addresses open source philosophy, source - repositories, workflows, Git, and licensing. -</para> - -<section id='open-source-philosophy'> - <title>Open Source Philosophy</title> - - <para> - Open source philosophy is characterized by software development - directed by peer production and collaboration through an active - community of developers. - Contrast this to the more standard centralized development models - used by commercial software companies where a finite set of developers - produces a product for sale using a defined set of procedures that - ultimately result in an end product whose architecture and source - material are closed to the public. - </para> - - <para> - Open source projects conceptually have differing concurrent agendas, - approaches, and production. - These facets of the development process can come from anyone in the - public (community) who has a stake in the software project. - The open source environment contains new copyright, licensing, domain, - and consumer issues that differ from the more traditional development - environment. - In an open source environment, the end product, source material, - and documentation are all available to the public at no cost. - </para> - - <para> - A benchmark example of an open source project is the Linux kernel, - which was initially conceived and created by Finnish computer science - student Linus Torvalds in 1991. - Conversely, a good example of a non-open source project is the - <trademark class='registered'>Windows</trademark> family of operating - systems developed by - <trademark class='registered'>Microsoft</trademark> Corporation. - </para> - - <para> - Wikipedia has a good historical description of the Open Source - Philosophy - <ulink url='http://en.wikipedia.org/wiki/Open_source'>here</ulink>. - You can also find helpful information on how to participate in the - Linux Community - <ulink url='http://ldn.linuxfoundation.org/book/how-participate-linux-community'>here</ulink>. - </para> -</section> - -<section id='gs-the-development-host'> - <title>The Development Host</title> - - <para> - A development host or - <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>build host</ulink> - is key to using the Yocto Project. - Because the goal of the Yocto Project is to develop images or - applications that run on embedded hardware, development of those - images and applications generally takes place on a system not - intended to run the software - the development host. - </para> - - <para> - You need to set up a development host in order to use it with the - Yocto Project. - Most find that it is best to have a native Linux machine function as - the development host. - However, it is possible to use a system that does not run Linux - as its operating system as your development host. - When you have a Mac or Windows-based system, you can set it up - as the development host by using - <ulink url='https://github.com/crops/poky-container'>CROPS</ulink>, - which leverages - <ulink url='https://www.docker.com/'>Docker Containers</ulink>. - Once you take the steps to set up a CROPS machine, you effectively - have access to a shell environment that is similar to what you see - when using a Linux-based development host. - For the steps needed to set up a system using CROPS, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-to-use-crops'>Setting Up to Use CROss PlatformS (CROPS)</ulink>" - section in the Yocto Project Development Tasks Manual. - </para> - - <para> - If your development host is going to be a system that runs a Linux - distribution, steps still exist that you must take to prepare the - system for use with the Yocto Project. - You need to be sure that the Linux distribution on the system is - one that supports the Yocto Project. - You also need to be sure that the correct set of host packages are - installed that allow development using the Yocto Project. - For the steps needed to set up a development host that runs Linux, - see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-a-native-linux-host'>Setting Up a Native Linux Host</ulink>" - section in the Yocto Project Development Tasks Manual. - </para> - - <para> - Once your development host is set up to use the Yocto Project, - several methods exist for you to do work in the Yocto Project - environment: - <itemizedlist> - <listitem><para> - <emphasis>Command Lines, BitBake, and Shells:</emphasis> - Traditional development in the Yocto Project involves using the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>, - which uses BitBake, in a command-line environment from a shell - on your development host. - You can accomplish this from a host that is a native Linux - machine or from a host that has been set up with CROPS. - Either way, you create, modify, and build images and - applications all within a shell-based environment using - components and tools available through your Linux distribution - and the Yocto Project.</para> - - <para>For a general flow of the build procedures, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-building-a-simple-image'>Building a Simple Image</ulink>" - section in the Yocto Project Development Tasks Manual. - </para></listitem> - <listitem><para> - <emphasis>Board Support Package (BSP) Development:</emphasis> - Development of BSPs involves using the Yocto Project to - create and test layers that allow easy development of - images and applications targeted for specific hardware. - To development BSPs, you need to take some additional steps - beyond what was described in setting up a development host. - </para> - - <para>The - <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink> - provides BSP-related development information. - For specifics on development host preparation, see the - "<ulink url='&YOCTO_DOCS_BSP_URL;#preparing-your-build-host-to-work-with-bsp-layers'>Preparing Your Build Host to Work With BSP Layers</ulink>" - section in the Yocto Project Board Support Package (BSP) - Developer's Guide. - </para></listitem> - <listitem><para> - <emphasis>Kernel Development:</emphasis> - If you are going to be developing kernels using the Yocto - Project you likely will be using <filename>devtool</filename>. - A workflow using <filename>devtool</filename> makes kernel - development quicker by reducing iteration cycle times.</para> - - <para>The - <ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;'>Yocto Project Linux Kernel Development Manual</ulink> - provides kernel-related development information. - For specifics on development host preparation, see the - "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#preparing-the-build-host-to-work-on-the-kernel'>Preparing the Build Host to Work on the Kernel</ulink>" - section in the Yocto Project Linux Kernel Development Manual. - </para></listitem> - <listitem><para> - <emphasis>Using Toaster:</emphasis> - The other Yocto Project development method that involves an - interface that effectively puts the Yocto Project into the - background is Toaster. - Toaster provides an interface to the OpenEmbedded build system. - The interface enables you to configure and run your builds. - Information about builds is collected and stored in a database. - You can use Toaster to configure and start builds on multiple - remote build servers.</para> - - <para>For steps that show you how to set up your development - host to use Toaster and on how to use Toaster in general, - see the - <ulink url='&YOCTO_DOCS_TOAST_URL;'>Toaster User Manual</ulink>. - </para></listitem> - </itemizedlist> - </para> -</section> - -<section id='yocto-project-repositories'> - <title>Yocto Project Source Repositories</title> - - <para> - The Yocto Project team maintains complete source repositories for all - Yocto Project files at - <ulink url='&YOCTO_GIT_URL;'></ulink>. - This web-based source code browser is organized into categories by - function such as IDE Plugins, Matchbox, Poky, Yocto Linux Kernel, and - so forth. - From the interface, you can click on any particular item in the "Name" - column and see the URL at the bottom of the page that you need to clone - a Git repository for that particular item. - Having a local Git repository of the - <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>, - which is usually named "poky", allows - you to make changes, contribute to the history, and ultimately enhance - the Yocto Project's tools, Board Support Packages, and so forth. - </para> - - <para> - For any supported release of Yocto Project, you can also go to the - <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink> and - select the "DOWNLOADS" item from the "SOFTWARE" menu and get a - released tarball of the <filename>poky</filename> repository, any - supported BSP tarball, or Yocto Project tools. - Unpacking these tarballs gives you a snapshot of the released - files. - <note><title>Notes</title> - <itemizedlist> - <listitem><para> - The recommended method for setting up the Yocto Project - <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> - and the files for supported BSPs - (e.g., <filename>meta-intel</filename>) is to use - <link linkend='git'>Git</link> to create a local copy of - the upstream repositories. - </para></listitem> - <listitem><para> - Be sure to always work in matching branches for both - the selected BSP repository and the Source Directory - (i.e. <filename>poky</filename>) repository. - For example, if you have checked out the "master" branch - of <filename>poky</filename> and you are going to use - <filename>meta-intel</filename>, be sure to checkout the - "master" branch of <filename>meta-intel</filename>. - </para></listitem> - </itemizedlist> - </note> - </para> - - <para> - In summary, here is where you can get the project files needed for - development: - <itemizedlist> - <listitem><para id='source-repositories'> - <emphasis> - <ulink url='&YOCTO_GIT_URL;'>Source Repositories:</ulink> - </emphasis> - This area contains IDE Plugins, Matchbox, Poky, Poky Support, - Tools, Yocto Linux Kernel, and Yocto Metadata Layers. - You can create local copies of Git repositories for each of - these areas.</para> - - <para> - <imagedata fileref="figures/source-repos.png" align="center" width="6in" depth="4in" /> - For steps on how to view and access these upstream Git - repositories, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#accessing-source-repositories'>Accessing Source Repositories</ulink>" - Section in the Yocto Project Development Tasks Manual. - </para></listitem> - <listitem><para><anchor id='index-downloads' /> - <emphasis> - <ulink url='&YOCTO_DL_URL;/releases/'>Index of /releases:</ulink> - </emphasis> - This is an index of releases such as Poky, Pseudo, installers - for cross-development toolchains, miscellaneous support - and all released versions of Yocto Project in the form of - images or tarballs. - Downloading and extracting these files does not produce a local - copy of the Git repository but rather a snapshot of a - particular release or image.</para> - - <para> - <imagedata fileref="figures/index-downloads.png" align="center" width="6in" depth="3.5in" /> - For steps on how to view and access these files, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#accessing-index-of-releases'>Accessing Index of Releases</ulink>" - section in the Yocto Project Development Tasks Manual. - </para></listitem> - <listitem><para id='downloads-page'> - <emphasis>"DOWNLOADS" page for the - <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink>: - </emphasis></para> - - <para>The Yocto Project website includes a "DOWNLOADS" page - accessible through the "SOFTWARE" menu that allows you to - download any Yocto Project release, tool, and Board Support - Package (BSP) in tarball form. - The tarballs are similar to those found in the - <ulink url='&YOCTO_DL_URL;/releases/'>Index of /releases:</ulink> - area.</para> - - <para> - <imagedata fileref="figures/yp-download.png" align="center" width="6in" depth="4in" /> - For steps on how to use the "DOWNLOADS" page, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#using-the-downloads-page'>Using the Downloads Page</ulink>" - section in the Yocto Project Development Tasks Manual. - </para></listitem> - </itemizedlist> - </para> -</section> - -<section id='gs-git-workflows-and-the-yocto-project'> - <title>Git Workflows and the Yocto Project</title> - - <para> - Developing using the Yocto Project likely requires the use of - <link linkend='git'>Git</link>. - Git is a free, open source distributed version control system - used as part of many collaborative design environments. - This section provides workflow concepts using the Yocto Project and - Git. - In particular, the information covers basic practices that describe - roles and actions in a collaborative development environment. - <note> - If you are familiar with this type of development environment, you - might not want to read this section. - </note> - </para> - - <para> - The Yocto Project files are maintained using Git in "branches" - whose Git histories track every change and whose structures - provide branches for all diverging functionality. - Although there is no need to use Git, many open source projects do so. - <para> - - </para> - For the Yocto Project, a key individual called the "maintainer" is - responsible for the integrity of the "master" branch of a given Git - repository. - The "master" branch is the “upstream” repository from which final or - most recent builds of a project occur. - The maintainer is responsible for accepting changes from other - developers and for organizing the underlying branch structure to - reflect release strategies and so forth. - <note> - For information on finding out who is responsible for (maintains) - a particular area of code in the Yocto Project, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>" - section of the Yocto Project Development Tasks Manual. - </note> - </para> - - <para> - The Yocto Project <filename>poky</filename> Git repository also has an - upstream contribution Git repository named - <filename>poky-contrib</filename>. - You can see all the branches in this repository using the web interface - of the - <ulink url='&YOCTO_GIT_URL;'>Source Repositories</ulink> organized - within the "Poky Support" area. - These branches hold changes (commits) to the project that have been - submitted or committed by the Yocto Project development team and by - community members who contribute to the project. - The maintainer determines if the changes are qualified to be moved - from the "contrib" branches into the "master" branch of the Git - repository. - </para> - - <para> - Developers (including contributing community members) create and - maintain cloned repositories of upstream branches. - The cloned repositories are local to their development platforms and - are used to develop changes. - When a developer is satisfied with a particular feature or change, - they "push" the change to the appropriate "contrib" repository. - </para> - - <para> - Developers are responsible for keeping their local repository - up-to-date with whatever upstream branch they are working against. - They are also responsible for straightening out any conflicts that - might arise within files that are being worked on simultaneously by - more than one person. - All this work is done locally on the development host before - anything is pushed to a "contrib" area and examined at the maintainer’s - level. - </para> - - <para> - A somewhat formal method exists by which developers commit changes - and push them into the "contrib" area and subsequently request that - the maintainer include them into an upstream branch. - This process is called “submitting a patch” or "submitting a change." - For information on submitting patches and changes, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>" - section in the Yocto Project Development Tasks Manual. - </para> - - <para> - In summary, a single point of entry - exists for changes into a "master" or development branch of the - Git repository, which is controlled by the project’s maintainer. - And, a set of developers exist who independently develop, test, and - submit changes to "contrib" areas for the maintainer to examine. - The maintainer then chooses which changes are going to become a - permanent part of the project. - </para> - - <para> - <imagedata fileref="figures/git-workflow.png" width="6in" depth="3in" align="left" scalefit="1" /> - </para> - - <para> - While each development environment is unique, there are some best - practices or methods that help development run smoothly. - The following list describes some of these practices. - For more information about Git workflows, see the workflow topics in - the - <ulink url='http://book.git-scm.com'>Git Community Book</ulink>. - <itemizedlist> - <listitem><para> - <emphasis>Make Small Changes:</emphasis> - It is best to keep the changes you commit small as compared to - bundling many disparate changes into a single commit. - This practice not only keeps things manageable but also allows - the maintainer to more easily include or refuse changes. - </para></listitem> - <listitem><para> - <emphasis>Make Complete Changes:</emphasis> - It is also good practice to leave the repository in a - state that allows you to still successfully build your project. - In other words, do not commit half of a feature, - then add the other half as a separate, later commit. - Each commit should take you from one buildable project state - to another buildable state. - </para></listitem> - <listitem><para> - <emphasis>Use Branches Liberally:</emphasis> - It is very easy to create, use, and delete local branches in - your working Git repository on the development host. - You can name these branches anything you like. - It is helpful to give them names associated with the particular - feature or change on which you are working. - Once you are done with a feature or change and have merged it - into your local master branch, simply discard the temporary - branch. - </para></listitem> - <listitem><para> - <emphasis>Merge Changes:</emphasis> - The <filename>git merge</filename> command allows you to take - the changes from one branch and fold them into another branch. - This process is especially helpful when more than a single - developer might be working on different parts of the same - feature. - Merging changes also automatically identifies any collisions - or "conflicts" that might happen as a result of the same lines - of code being altered by two different developers. - </para></listitem> - <listitem><para> - <emphasis>Manage Branches:</emphasis> - Because branches are easy to use, you should use a system - where branches indicate varying levels of code readiness. - For example, you can have a "work" branch to develop in, a - "test" branch where the code or change is tested, a "stage" - branch where changes are ready to be committed, and so forth. - As your project develops, you can merge code across the - branches to reflect ever-increasing stable states of the - development. - </para></listitem> - <listitem><para> - <emphasis>Use Push and Pull:</emphasis> - The push-pull workflow is based on the concept of developers - "pushing" local commits to a remote repository, which is - usually a contribution repository. - This workflow is also based on developers "pulling" known - states of the project down into their local development - repositories. - The workflow easily allows you to pull changes submitted by - other developers from the upstream repository into your - work area ensuring that you have the most recent software - on which to develop. - The Yocto Project has two scripts named - <filename>create-pull-request</filename> and - <filename>send-pull-request</filename> that ship with the - release to facilitate this workflow. - You can find these scripts in the <filename>scripts</filename> - folder of the - <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. - For information on how to use these scripts, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#pushing-a-change-upstream'>Using Scripts to Push a Change Upstream and Request a Pull</ulink>" - section in the Yocto Project Development Tasks Manual. - </para></listitem> - <listitem><para> - <emphasis>Patch Workflow:</emphasis> - This workflow allows you to notify the maintainer through an - email that you have a change (or patch) you would like - considered for the "master" branch of the Git repository. - To send this type of change, you format the patch and then - send the email using the Git commands - <filename>git format-patch</filename> and - <filename>git send-email</filename>. - For information on how to use these scripts, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>" - section in the Yocto Project Development Tasks Manual. - </para></listitem> - </itemizedlist> - </para> -</section> - -<section id='git'> - <title>Git</title> - - <para> - The Yocto Project makes extensive use of Git, which is a - free, open source distributed version control system. - Git supports distributed development, non-linear development, - and can handle large projects. - It is best that you have some fundamental understanding - of how Git tracks projects and how to work with Git if - you are going to use the Yocto Project for development. - This section provides a quick overview of how Git works and - provides you with a summary of some essential Git commands. - <note><title>Notes</title> - <itemizedlist> - <listitem><para> - For more information on Git, see - <ulink url='http://git-scm.com/documentation'></ulink>. - </para></listitem> - <listitem><para> - If you need to download Git, it is recommended that you add - Git to your system through your distribution's "software - store" (e.g. for Ubuntu, use the Ubuntu Software feature). - For the Git download page, see - <ulink url='http://git-scm.com/download'></ulink>. - </para></listitem> - <listitem><para> - For information beyond the introductory nature in this - section, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#locating-yocto-project-source-files'>Locating Yocto Project Source Files</ulink>" - section in the Yocto Project Development Tasks Manual. - </para></listitem> - </itemizedlist> - </note> - </para> - - <section id='repositories-tags-and-branches'> - <title>Repositories, Tags, and Branches</title> - - <para> - As mentioned briefly in the previous section and also in the - "<link linkend='gs-git-workflows-and-the-yocto-project'>Git Workflows and the Yocto Project</link>" - section, the Yocto Project maintains source repositories at - <ulink url='&YOCTO_GIT_URL;'></ulink>. - If you look at this web-interface of the repositories, each item - is a separate Git repository. - </para> - - <para> - Git repositories use branching techniques that track content - change (not files) within a project (e.g. a new feature or updated - documentation). - Creating a tree-like structure based on project divergence allows - for excellent historical information over the life of a project. - This methodology also allows for an environment from which you can - do lots of local experimentation on projects as you develop - changes or new features. - </para> - - <para> - A Git repository represents all development efforts for a given - project. - For example, the Git repository <filename>poky</filename> contains - all changes and developments for that repository over the course - of its entire life. - That means that all changes that make up all releases are captured. - The repository maintains a complete history of changes. - </para> - - <para> - You can create a local copy of any repository by "cloning" it - with the <filename>git clone</filename> command. - When you clone a Git repository, you end up with an identical - copy of the repository on your development system. - Once you have a local copy of a repository, you can take steps to - develop locally. - For examples on how to clone Git repositories, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#locating-yocto-project-source-files'>Locating Yocto Project Source Files</ulink>" - section in the Yocto Project Development Tasks Manual. - </para> - - <para> - It is important to understand that Git tracks content change and - not files. - Git uses "branches" to organize different development efforts. - For example, the <filename>poky</filename> repository has - several branches that include the current "&DISTRO_NAME_NO_CAP;" - branch, the "master" branch, and many branches for past - Yocto Project releases. - You can see all the branches by going to - <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/'></ulink> and - clicking on the - <filename><ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/refs/heads'>[...]</ulink></filename> - link beneath the "Branch" heading. - </para> - - <para> - Each of these branches represents a specific area of development. - The "master" branch represents the current or most recent - development. - All other branches represent offshoots of the "master" branch. - </para> - - <para> - When you create a local copy of a Git repository, the copy has - the same set of branches as the original. - This means you can use Git to create a local working area - (also called a branch) that tracks a specific development branch - from the upstream source Git repository. - in other words, you can define your local Git environment to - work on any development branch in the repository. - To help illustrate, consider the following example Git commands: - <literallayout class='monospaced'> - $ cd ~ - $ git clone git://git.yoctoproject.org/poky - $ cd poky - $ git checkout -b &DISTRO_NAME_NO_CAP; origin/&DISTRO_NAME_NO_CAP; - </literallayout> - In the previous example after moving to the home directory, the - <filename>git clone</filename> command creates a - local copy of the upstream <filename>poky</filename> Git repository. - By default, Git checks out the "master" branch for your work. - After changing the working directory to the new local repository - (i.e. <filename>poky</filename>), the - <filename>git checkout</filename> command creates - and checks out a local branch named "&DISTRO_NAME_NO_CAP;", which - tracks the upstream "origin/&DISTRO_NAME_NO_CAP;" branch. - Changes you make while in this branch would ultimately affect - the upstream "&DISTRO_NAME_NO_CAP;" branch of the - <filename>poky</filename> repository. - </para> - - <para> - It is important to understand that when you create and checkout a - local working branch based on a branch name, - your local environment matches the "tip" of that particular - development branch at the time you created your local branch, - which could be different from the files in the "master" branch - of the upstream repository. - In other words, creating and checking out a local branch based on - the "&DISTRO_NAME_NO_CAP;" branch name is not the same as - checking out the "master" branch in the repository. - Keep reading to see how you create a local snapshot of a Yocto - Project Release. - </para> - - <para> - Git uses "tags" to mark specific changes in a repository branch - structure. - Typically, a tag is used to mark a special point such as the final - change (or commit) before a project is released. - You can see the tags used with the <filename>poky</filename> Git - repository by going to - <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/'></ulink> and - clicking on the - <filename><ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/refs/tags'>[...]</ulink></filename> - link beneath the "Tag" heading. - </para> - - <para> - Some key tags for the <filename>poky</filename> repository are - <filename>jethro-14.0.3</filename>, - <filename>morty-16.0.1</filename>, - <filename>pyro-17.0.0</filename>, and - <filename>&DISTRO_NAME_NO_CAP;-&POKYVERSION;</filename>. - These tags represent Yocto Project releases. - </para> - - <para> - When you create a local copy of the Git repository, you also - have access to all the tags in the upstream repository. - Similar to branches, you can create and checkout a local working - Git branch based on a tag name. - When you do this, you get a snapshot of the Git repository that - reflects the state of the files when the change was made associated - with that tag. - The most common use is to checkout a working branch that matches - a specific Yocto Project release. - Here is an example: - <literallayout class='monospaced'> - $ cd ~ - $ git clone git://git.yoctoproject.org/poky - $ cd poky - $ git fetch --tags - $ git checkout tags/rocko-18.0.0 -b my_rocko-18.0.0 - </literallayout> - In this example, the name of the top-level directory of your - local Yocto Project repository is <filename>poky</filename>. - After moving to the <filename>poky</filename> directory, the - <filename>git fetch</filename> command makes all the upstream - tags available locally in your repository. - Finally, the <filename>git checkout</filename> command - creates and checks out a branch named "my-rocko-18.0.0" that is - based on the upstream branch whose "HEAD" matches the - commit in the repository associated with the "rocko-18.0.0" tag. - The files in your repository now exactly match that particular - Yocto Project release as it is tagged in the upstream Git - repository. - It is important to understand that when you create and - checkout a local working branch based on a tag, your environment - matches a specific point in time and not the entire development - branch (i.e. from the "tip" of the branch backwards). - </para> - </section> - - <section id='basic-commands'> - <title>Basic Commands</title> - - <para> - Git has an extensive set of commands that lets you manage changes - and perform collaboration over the life of a project. - Conveniently though, you can manage with a small set of basic - operations and workflows once you understand the basic - philosophy behind Git. - You do not have to be an expert in Git to be functional. - A good place to look for instruction on a minimal set of Git - commands is - <ulink url='http://git-scm.com/documentation'>here</ulink>. - </para> - - <para> - The following list of Git commands briefly describes some basic - Git operations as a way to get started. - As with any set of commands, this list (in most cases) simply shows - the base command and omits the many arguments it supports. - See the Git documentation for complete descriptions and strategies - on how to use these commands: - <itemizedlist> - <listitem><para> - <emphasis><filename>git init</filename>:</emphasis> - Initializes an empty Git repository. - You cannot use Git commands unless you have a - <filename>.git</filename> repository. - </para></listitem> - <listitem><para id='git-commands-clone'> - <emphasis><filename>git clone</filename>:</emphasis> - Creates a local clone of a Git repository that is on - equal footing with a fellow developer’s Git repository - or an upstream repository. - </para></listitem> - <listitem><para> - <emphasis><filename>git add</filename>:</emphasis> - Locally stages updated file contents to the index that - Git uses to track changes. - You must stage all files that have changed before you - can commit them. - </para></listitem> - <listitem><para> - <emphasis><filename>git commit</filename>:</emphasis> - Creates a local "commit" that documents the changes you - made. - Only changes that have been staged can be committed. - Commits are used for historical purposes, for determining - if a maintainer of a project will allow the change, - and for ultimately pushing the change from your local - Git repository into the project’s upstream repository. - </para></listitem> - <listitem><para> - <emphasis><filename>git status</filename>:</emphasis> - Reports any modified files that possibly need to be - staged and gives you a status of where you stand regarding - local commits as compared to the upstream repository. - </para></listitem> - <listitem><para> - <emphasis><filename>git checkout</filename> <replaceable>branch-name</replaceable>:</emphasis> - Changes your local working branch and in this form - assumes the local branch already exists. - This command is analogous to "cd". - </para></listitem> - <listitem><para> - <emphasis><filename>git checkout –b</filename> <replaceable>working-branch</replaceable> <replaceable>upstream-branch</replaceable>:</emphasis> - Creates and checks out a working branch on your local - machine. - The local branch tracks the upstream branch. - You can use your local branch to isolate your work. - It is a good idea to use local branches when adding - specific features or changes. - Using isolated branches facilitates easy removal of - changes if they do not work out. - </para></listitem> - <listitem><para><emphasis><filename>git branch</filename>:</emphasis> - Displays the existing local branches associated with your - local repository. - The branch that you have currently checked out is noted - with an asterisk character. - </para></listitem> - <listitem><para> - <emphasis><filename>git branch -D</filename> <replaceable>branch-name</replaceable>:</emphasis> - Deletes an existing local branch. - You need to be in a local branch other than the one you - are deleting in order to delete - <replaceable>branch-name</replaceable>. - </para></listitem> - <listitem><para> - <emphasis><filename>git pull --rebase</filename>:</emphasis> - Retrieves information from an upstream Git repository - and places it in your local Git repository. - You use this command to make sure you are synchronized with - the repository from which you are basing changes - (.e.g. the "master" branch). - The "--rebase" option ensures that any local commits you - have in your branch are preserved at the top of your - local branch. - </para></listitem> - <listitem><para> - <emphasis><filename>git push</filename> <replaceable>repo-name</replaceable> <replaceable>local-branch</replaceable><filename>:</filename><replaceable>upstream-branch</replaceable>:</emphasis> - Sends all your committed local changes to the upstream Git - repository that your local repository is tracking - (e.g. a contribution repository). - The maintainer of the project draws from these repositories - to merge changes (commits) into the appropriate branch - of project's upstream repository. - </para></listitem> - <listitem><para> - <emphasis><filename>git merge</filename>:</emphasis> - Combines or adds changes from one - local branch of your repository with another branch. - When you create a local Git repository, the default branch - is named "master". - A typical workflow is to create a temporary branch that is - based off "master" that you would use for isolated work. - You would make your changes in that isolated branch, - stage and commit them locally, switch to the "master" - branch, and then use the <filename>git merge</filename> - command to apply the changes from your isolated branch - into the currently checked out branch (e.g. "master"). - After the merge is complete and if you are done with - working in that isolated branch, you can safely delete - the isolated branch. - </para></listitem> - <listitem><para> - <emphasis><filename>git cherry-pick</filename> <replaceable>commits</replaceable>:</emphasis> - Choose and apply specific commits from one branch - into another branch. - There are times when you might not be able to merge - all the changes in one branch with - another but need to pick out certain ones. - </para></listitem> - <listitem><para> - <emphasis><filename>gitk</filename>:</emphasis> - Provides a GUI view of the branches and changes in your - local Git repository. - This command is a good way to graphically see where things - have diverged in your local repository. - <note> - You need to install the <filename>gitk</filename> - package on your development system to use this - command. - </note> - </para></listitem> - <listitem><para> - <emphasis><filename>git log</filename>:</emphasis> - Reports a history of your commits to the repository. - This report lists all commits regardless of whether you - have pushed them upstream or not. - </para></listitem> - <listitem><para> - <emphasis><filename>git diff</filename>:</emphasis> - Displays line-by-line differences between a local - working file and the same file as understood by Git. - This command is useful to see what you have changed - in any given file. - </para></listitem> - </itemizedlist> - </para> - </section> -</section> - -<section id='licensing'> - <title>Licensing</title> - - <para> - Because open source projects are open to the public, they have - different licensing structures in place. - License evolution for both Open Source and Free Software has an - interesting history. - If you are interested in this history, you can find basic information - here: - <itemizedlist> - <listitem><para> - <ulink url='http://en.wikipedia.org/wiki/Open-source_license'>Open source license history</ulink> - </para></listitem> - <listitem><para> - <ulink url='http://en.wikipedia.org/wiki/Free_software_license'>Free software license history</ulink> - </para></listitem> - </itemizedlist> - </para> - - <para> - In general, the Yocto Project is broadly licensed under the - Massachusetts Institute of Technology (MIT) License. - MIT licensing permits the reuse of software within proprietary - software as long as the license is distributed with that software. - MIT is also compatible with the GNU General Public License (GPL). - Patches to the Yocto Project follow the upstream licensing scheme. - You can find information on the MIT license - <ulink url='http://www.opensource.org/licenses/mit-license.php'>here</ulink>. - You can find information on the GNU GPL - <ulink url='http://www.opensource.org/licenses/LGPL-3.0'>here</ulink>. - </para> - - <para> - When you build an image using the Yocto Project, the build process - uses a known list of licenses to ensure compliance. - You can find this list in the - <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> - at <filename>meta/files/common-licenses</filename>. - Once the build completes, the list of all licenses found and used - during that build are kept in the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> - at <filename>tmp/deploy/licenses</filename>. - </para> - - <para> - If a module requires a license that is not in the base list, the - build process generates a warning during the build. - These tools make it easier for a developer to be certain of the - licenses with which their shipped products must comply. - However, even with these tools it is still up to the developer to - resolve potential licensing issues. - </para> - - <para> - The base list of licenses used by the build process is a combination - of the Software Package Data Exchange (SPDX) list and the Open - Source Initiative (OSI) projects. - <ulink url='http://spdx.org'>SPDX Group</ulink> is a working group of - the Linux Foundation that maintains a specification for a standard - format for communicating the components, licenses, and copyrights - associated with a software package. - <ulink url='http://opensource.org'>OSI</ulink> is a corporation - dedicated to the Open Source Definition and the effort for reviewing - and approving licenses that conform to the Open Source Definition - (OSD). - </para> - - <para> - You can find a list of the combined SPDX and OSI licenses that the - Yocto Project uses in the - <filename>meta/files/common-licenses</filename> directory in your - <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. - </para> - - <para> - For information that can help you maintain compliance with various - open source licensing during the lifecycle of a product created using - the Yocto Project, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Product's Lifecycle</ulink>" - section in the Yocto Project Development Tasks Manual. - </para> -</section> -</chapter> -<!-- -vim: expandtab tw=80 ts=4 ---> diff --git a/poky/documentation/overview-manual/overview-manual-intro.rst b/poky/documentation/overview-manual/overview-manual-intro.rst new file mode 100644 index 0000000000..8885eb89ff --- /dev/null +++ b/poky/documentation/overview-manual/overview-manual-intro.rst @@ -0,0 +1,74 @@ +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK + +********************************************** +The Yocto Project Overview and Concepts Manual +********************************************** + +.. _overview-manual-welcome: + +Welcome +======= + +Welcome to the Yocto Project Overview and Concepts Manual! This manual +introduces the Yocto Project by providing concepts, software overviews, +best-known-methods (BKMs), and any other high-level introductory +information suitable for a new Yocto Project user. + +The following list describes what you can get from this manual: + +- `Introducing the Yocto Project <#overview-yp>`__\ *:* This chapter + provides an introduction to the Yocto Project. You will learn about + features and challenges of the Yocto Project, the layer model, + components and tools, development methods, the + :term:`Poky` reference distribution, the + OpenEmbedded build system workflow, and some basic Yocto terms. + +- `The Yocto Project Development + Environment <#overview-development-environment>`__\ *:* This chapter + helps you get started understanding the Yocto Project development + environment. You will learn about open source, development hosts, + Yocto Project source repositories, workflows using Git and the Yocto + Project, a Git primer, and information about licensing. + +- :doc:`overview-manual-concepts` *:* This + chapter presents various concepts regarding the Yocto Project. You + can find conceptual information about components, development, + cross-toolchains, and so forth. + +This manual does not give you the following: + +- *Step-by-step Instructions for Development Tasks:* Instructional + procedures reside in other manuals within the Yocto Project + documentation set. For example, the :doc:`../dev-manual/dev-manual` + provides examples on how to perform + various development tasks. As another example, the + :doc:`../sdk-manual/sdk-manual` manual contains detailed + instructions on how to install an SDK, which is used to develop + applications for target hardware. + +- *Reference Material:* This type of material resides in an appropriate + reference manual. For example, system variables are documented in the + :doc:`../ref-manual/ref-manual`. As another + example, the :doc:`../bsp-guide/bsp-guide` contains reference information on + BSPs. + +- *Detailed Public Information Not Specific to the Yocto Project:* For + example, exhaustive information on how to use the Source Control + Manager Git is better covered with Internet searches and official Git + Documentation than through the Yocto Project documentation. + +.. _overview-manual-other-information: + +Other Information +================= + +Because this manual presents information for many different topics, +supplemental information is recommended for full comprehension. For +additional introductory information on the Yocto Project, see the +:yocto_home:`Yocto Project Website <>`. If you want to build an image +with no knowledge of Yocto Project as a way of quickly testing it out, +see the :doc:`../brief-yoctoprojectqs/brief-yoctoprojectqs` document. +For a comprehensive list of links and other documentation, see the +":ref:`Links and Related +Documentation <resources-links-and-related-documentation>`" +section in the Yocto Project Reference Manual. diff --git a/poky/documentation/overview-manual/overview-manual-intro.xml b/poky/documentation/overview-manual/overview-manual-intro.xml deleted file mode 100644 index 39433aa41b..0000000000 --- a/poky/documentation/overview-manual/overview-manual-intro.xml +++ /dev/null @@ -1,112 +0,0 @@ -<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" -"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" -[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > - -<chapter id='overview-manual-intro'> - -<title>The Yocto Project Overview and Concepts Manual</title> - <section id='overview-manual-welcome'> - <title>Welcome</title> - - <para> - Welcome to the Yocto Project Overview and Concepts Manual! - This manual introduces the Yocto Project by providing concepts, - software overviews, best-known-methods (BKMs), and any other - high-level introductory information suitable for a new Yocto - Project user. - </para> - - <para> - The following list describes what you can get from this manual: - <itemizedlist> - <listitem><para> - <emphasis><link linkend='overview-yp'>Introducing the Yocto Project</link>:</emphasis> - This chapter provides an introduction to the Yocto - Project. - You will learn about features and challenges of the - Yocto Project, the layer model, components and tools, - development methods, the - <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink> - reference distribution, the OpenEmbedded build system - workflow, and some basic Yocto terms. - </para></listitem> - <listitem><para> - <emphasis><link linkend='overview-development-environment'>The Yocto Project Development Environment</link>:</emphasis> - This chapter helps you get started understanding the - Yocto Project development environment. - You will learn about open source, development hosts, - Yocto Project source repositories, workflows using Git - and the Yocto Project, a Git primer, and information - about licensing. - </para></listitem> - <listitem><para> - <emphasis><link linkend='overview-manual-concepts'>Yocto Project Concepts</link>:</emphasis> - This chapter presents various concepts regarding the - Yocto Project. - You can find conceptual information about components, - development, cross-toolchains, and so forth. - </para></listitem> - </itemizedlist> - </para> - - <para> - This manual does not give you the following: - <itemizedlist> - <listitem><para> - <emphasis>Step-by-step Instructions for Development Tasks:</emphasis> - Instructional procedures reside in other manuals within - the Yocto Project documentation set. - For example, the - <ulink url='&YOCTO_DOCS_DEV_URL;'>Yocto Project Development Tasks Manual</ulink> - provides examples on how to perform various development - tasks. - As another example, the - <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> - manual contains detailed instructions on how to install an - SDK, which is used to develop applications for target - hardware. - </para></listitem> - <listitem><para> - <emphasis>Reference Material:</emphasis> - This type of material resides in an appropriate reference - manual. - For example, system variables are documented in the - <ulink url='&YOCTO_DOCS_REF_URL;'>Yocto Project Reference Manual</ulink>. - As another example, the - <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink> - contains reference information on BSPs. - </para></listitem> - <listitem><para> - <emphasis>Detailed Public Information Not Specific to the - Yocto Project:</emphasis> - For example, exhaustive information on how to use the - Source Control Manager Git is better covered with Internet - searches and official Git Documentation than through the - Yocto Project documentation. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='overview-manual-other-information'> - <title>Other Information</title> - - <para> - Because this manual presents information for many different - topics, supplemental information is recommended for full - comprehension. - For additional introductory information on the Yocto Project, see - the <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink>. - If you want to build an image with no knowledge of Yocto Project - as a way of quickly testing it out, see the - <ulink url='&YOCTO_DOCS_BRIEF_URL;'>Yocto Project Quick Build</ulink> - document. - For a comprehensive list of links and other documentation, see the - "<ulink url='&YOCTO_DOCS_REF_URL;#resources-links-and-related-documentation'>Links and Related Documentation</ulink>" - section in the Yocto Project Reference Manual. - </para> - </section> -</chapter> -<!-- -vim: expandtab tw=80 ts=4 ---> diff --git a/poky/documentation/overview-manual/overview-manual-style.css b/poky/documentation/overview-manual/overview-manual-style.css deleted file mode 100644 index 97a364b125..0000000000 --- a/poky/documentation/overview-manual/overview-manual-style.css +++ /dev/null @@ -1,988 +0,0 @@ -/* - Generic XHTML / DocBook XHTML CSS Stylesheet. - - Browser wrangling and typographic design by - Oyvind Kolas / pippin@gimp.org - - Customised for Poky by - Matthew Allum / mallum@o-hand.com - - Thanks to: - Liam R. E. Quin - William Skaggs - Jakub Steiner - - Structure - --------- - - The stylesheet is divided into the following sections: - - Positioning - Margins, paddings, width, font-size, clearing. - Decorations - Borders, style - Colors - Colors - Graphics - Graphical backgrounds - Nasty IE tweaks - Workarounds needed to make it work in internet explorer, - currently makes the stylesheet non validating, but up until - this point it is validating. - Mozilla extensions - Transparency for footer - Rounded corners on boxes - -*/ - - - /*************** / - / Positioning / -/ ***************/ - -body { - font-family: Verdana, Sans, sans-serif; - - min-width: 640px; - width: 80%; - margin: 0em auto; - padding: 2em 5em 5em 5em; - color: #333; -} - -h1,h2,h3,h4,h5,h6,h7 { - font-family: Arial, Sans; - color: #00557D; - clear: both; -} - -h1 { - font-size: 2em; - text-align: left; - padding: 0em 0em 0em 0em; - margin: 2em 0em 0em 0em; -} - -h2.subtitle { - margin: 0.10em 0em 3.0em 0em; - padding: 0em 0em 0em 0em; - font-size: 1.8em; - padding-left: 20%; - font-weight: normal; - font-style: italic; -} - -h2 { - margin: 2em 0em 0.66em 0em; - padding: 0.5em 0em 0em 0em; - font-size: 1.5em; - font-weight: bold; -} - -h3.subtitle { - margin: 0em 0em 1em 0em; - padding: 0em 0em 0em 0em; - font-size: 142.14%; - text-align: right; -} - -h3 { - margin: 1em 0em 0.5em 0em; - padding: 1em 0em 0em 0em; - font-size: 140%; - font-weight: bold; -} - -h4 { - margin: 1em 0em 0.5em 0em; - padding: 1em 0em 0em 0em; - font-size: 120%; - font-weight: bold; -} - -h5 { - margin: 1em 0em 0.5em 0em; - padding: 1em 0em 0em 0em; - font-size: 110%; - font-weight: bold; -} - -h6 { - margin: 1em 0em 0em 0em; - padding: 1em 0em 0em 0em; - font-size: 110%; - font-weight: bold; -} - -.authorgroup { - background-color: transparent; - background-repeat: no-repeat; - padding-top: 256px; - background-image: url("figures/overview-manual-title.png"); - background-position: left top; - margin-top: -256px; - padding-right: 50px; - margin-left: 0px; - text-align: right; - width: 740px; -} - -h3.author { - margin: 0em 0me 0em 0em; - padding: 0em 0em 0em 0em; - font-weight: normal; - font-size: 100%; - color: #333; - clear: both; -} - -.author tt.email { - font-size: 66%; -} - -.titlepage hr { - width: 0em; - clear: both; -} - -.revhistory { - padding-top: 2em; - clear: both; -} - -.toc, -.list-of-tables, -.list-of-examples, -.list-of-figures { - padding: 1.33em 0em 2.5em 0em; - color: #00557D; -} - -.toc p, -.list-of-tables p, -.list-of-figures p, -.list-of-examples p { - padding: 0em 0em 0em 0em; - padding: 0em 0em 0.3em; - margin: 1.5em 0em 0em 0em; -} - -.toc p b, -.list-of-tables p b, -.list-of-figures p b, -.list-of-examples p b{ - font-size: 100.0%; - font-weight: bold; -} - -.toc dl, -.list-of-tables dl, -.list-of-figures dl, -.list-of-examples dl { - margin: 0em 0em 0.5em 0em; - padding: 0em 0em 0em 0em; -} - -.toc dt { - margin: 0em 0em 0em 0em; - padding: 0em 0em 0em 0em; -} - -.toc dd { - margin: 0em 0em 0em 2.6em; - padding: 0em 0em 0em 0em; -} - -div.glossary dl, -div.variablelist dl { -} - -.glossary dl dt, -.variablelist dl dt, -.variablelist dl dt span.term { - font-weight: normal; - width: 20em; - text-align: right; -} - -.variablelist dl dt { - margin-top: 0.5em; -} - -.glossary dl dd, -.variablelist dl dd { - margin-top: -1em; - margin-left: 25.5em; -} - -.glossary dd p, -.variablelist dd p { - margin-top: 0em; - margin-bottom: 1em; -} - - -div.calloutlist table td { - padding: 0em 0em 0em 0em; - margin: 0em 0em 0em 0em; -} - -div.calloutlist table td p { - margin-top: 0em; - margin-bottom: 1em; -} - -div p.copyright { - text-align: left; -} - -div.legalnotice p.legalnotice-title { - margin-bottom: 0em; -} - -p { - line-height: 1.5em; - margin-top: 0em; - -} - -dl { - padding-top: 0em; -} - -hr { - border: solid 1px; -} - - -.mediaobject, -.mediaobjectco { - text-align: center; -} - -img { - border: none; -} - -ul { - padding: 0em 0em 0em 1.5em; -} - -ul li { - padding: 0em 0em 0em 0em; -} - -ul li p { - text-align: left; -} - -table { - width :100%; -} - -th { - padding: 0.25em; - text-align: left; - font-weight: normal; - vertical-align: top; -} - -td { - padding: 0.25em; - vertical-align: top; -} - -p a[id] { - margin: 0px; - padding: 0px; - display: inline; - background-image: none; -} - -a { - text-decoration: underline; - color: #444; -} - -pre { - overflow: auto; -} - -a:hover { - text-decoration: underline; - /*font-weight: bold;*/ -} - -/* This style defines how the permalink character - appears by itself and when hovered over with - the mouse. */ - -[alt='Permalink'] { color: #eee; } -[alt='Permalink']:hover { color: black; } - - -div.informalfigure, -div.informalexample, -div.informaltable, -div.figure, -div.table, -div.example { - margin: 1em 0em; - padding: 1em; - page-break-inside: avoid; -} - - -div.informalfigure p.title b, -div.informalexample p.title b, -div.informaltable p.title b, -div.figure p.title b, -div.example p.title b, -div.table p.title b{ - padding-top: 0em; - margin-top: 0em; - font-size: 100%; - font-weight: normal; -} - -.mediaobject .caption, -.mediaobject .caption p { - text-align: center; - font-size: 80%; - padding-top: 0.5em; - padding-bottom: 0.5em; -} - -.epigraph { - padding-left: 55%; - margin-bottom: 1em; -} - -.epigraph p { - text-align: left; -} - -.epigraph .quote { - font-style: italic; -} -.epigraph .attribution { - font-style: normal; - text-align: right; -} - -span.application { - font-style: italic; -} - -.programlisting { - font-family: monospace; - font-size: 80%; - white-space: pre; - margin: 1.33em 0em; - padding: 1.33em; -} - -.tip, -.warning, -.caution, -.note { - margin-top: 1em; - margin-bottom: 1em; - -} - -/* force full width of table within div */ -.tip table, -.warning table, -.caution table, -.note table { - border: none; - width: 100%; -} - - -.tip table th, -.warning table th, -.caution table th, -.note table th { - padding: 0.8em 0.0em 0.0em 0.0em; - margin : 0em 0em 0em 0em; -} - -.tip p, -.warning p, -.caution p, -.note p { - margin-top: 0.5em; - margin-bottom: 0.5em; - padding-right: 1em; - text-align: left; -} - -.acronym { - text-transform: uppercase; -} - -b.keycap, -.keycap { - padding: 0.09em 0.3em; - margin: 0em; -} - -.itemizedlist li { - clear: none; -} - -.filename { - font-size: medium; - font-family: Courier, monospace; -} - - -div.navheader, div.heading{ - position: absolute; - left: 0em; - top: 0em; - width: 100%; - background-color: #cdf; - width: 100%; -} - -div.navfooter, div.footing{ - position: fixed; - left: 0em; - bottom: 0em; - background-color: #eee; - width: 100%; -} - - -div.navheader td, -div.navfooter td { - font-size: 66%; -} - -div.navheader table th { - /*font-family: Georgia, Times, serif;*/ - /*font-size: x-large;*/ - font-size: 80%; -} - -div.navheader table { - border-left: 0em; - border-right: 0em; - border-top: 0em; - width: 100%; -} - -div.navfooter table { - border-left: 0em; - border-right: 0em; - border-bottom: 0em; - width: 100%; -} - -div.navheader table td a, -div.navfooter table td a { - color: #777; - text-decoration: none; -} - -/* normal text in the footer */ -div.navfooter table td { - color: black; -} - -div.navheader table td a:visited, -div.navfooter table td a:visited { - color: #444; -} - - -/* links in header and footer */ -div.navheader table td a:hover, -div.navfooter table td a:hover { - text-decoration: underline; - background-color: transparent; - color: #33a; -} - -div.navheader hr, -div.navfooter hr { - display: none; -} - - -.qandaset tr.question td p { - margin: 0em 0em 1em 0em; - padding: 0em 0em 0em 0em; -} - -.qandaset tr.answer td p { - margin: 0em 0em 1em 0em; - padding: 0em 0em 0em 0em; -} -.answer td { - padding-bottom: 1.5em; -} - -.emphasis { - font-weight: bold; -} - - - /************* / - / decorations / -/ *************/ - -.titlepage { -} - -.part .title { -} - -.subtitle { - border: none; -} - -/* -h1 { - border: none; -} - -h2 { - border-top: solid 0.2em; - border-bottom: solid 0.06em; -} - -h3 { - border-top: 0em; - border-bottom: solid 0.06em; -} - -h4 { - border: 0em; - border-bottom: solid 0.06em; -} - -h5 { - border: 0em; -} -*/ - -.programlisting { - border: solid 1px; -} - -div.figure, -div.table, -div.informalfigure, -div.informaltable, -div.informalexample, -div.example { - border: 1px solid; -} - - - -.tip, -.warning, -.caution, -.note { - border: 1px solid; -} - -.tip table th, -.warning table th, -.caution table th, -.note table th { - border-bottom: 1px solid; -} - -.question td { - border-top: 1px solid black; -} - -.answer { -} - - -b.keycap, -.keycap { - border: 1px solid; -} - - -div.navheader, div.heading{ - border-bottom: 1px solid; -} - - -div.navfooter, div.footing{ - border-top: 1px solid; -} - - /********* / - / colors / -/ *********/ - -body { - color: #333; - background: white; -} - -a { - background: transparent; -} - -a:hover { - background-color: #dedede; -} - - -h1, -h2, -h3, -h4, -h5, -h6, -h7, -h8 { - background-color: transparent; -} - -hr { - border-color: #aaa; -} - - -.tip, .warning, .caution, .note { - border-color: #fff; -} - - -.tip table th, -.warning table th, -.caution table th, -.note table th { - border-bottom-color: #fff; -} - - -.warning { - background-color: #f0f0f2; -} - -.caution { - background-color: #f0f0f2; -} - -.tip { - background-color: #f0f0f2; -} - -.note { - background-color: #f0f0f2; -} - -.glossary dl dt, -.variablelist dl dt, -.variablelist dl dt span.term { - color: #044; -} - -div.figure, -div.table, -div.example, -div.informalfigure, -div.informaltable, -div.informalexample { - border-color: #aaa; -} - -pre.programlisting { - color: black; - background-color: #fff; - border-color: #aaa; - border-width: 2px; -} - -.guimenu, -.guilabel, -.guimenuitem { - background-color: #eee; -} - - -b.keycap, -.keycap { - background-color: #eee; - border-color: #999; -} - - -div.navheader { - border-color: black; -} - - -div.navfooter { - border-color: black; -} - -.writernotes { - color: red; -} - - - /*********** / - / graphics / -/ ***********/ - -/* -body { - background-image: url("images/body_bg.jpg"); - background-attachment: fixed; -} - -.navheader, -.note, -.tip { - background-image: url("images/note_bg.jpg"); - background-attachment: fixed; -} - -.warning, -.caution { - background-image: url("images/warning_bg.jpg"); - background-attachment: fixed; -} - -.figure, -.informalfigure, -.example, -.informalexample, -.table, -.informaltable { - background-image: url("images/figure_bg.jpg"); - background-attachment: fixed; -} - -*/ -h1, -h2, -h3, -h4, -h5, -h6, -h7{ -} - -/* -Example of how to stick an image as part of the title. - -div.article .titlepage .title -{ - background-image: url("figures/white-on-black.png"); - background-position: center; - background-repeat: repeat-x; -} -*/ - -div.preface .titlepage .title, -div.colophon .title, -div.chapter .titlepage .title, -div.article .titlepage .title -{ -} - -div.section div.section .titlepage .title, -div.sect2 .titlepage .title { - background: none; -} - - -h1.title { - background-color: transparent; - background-repeat: no-repeat; - height: 256px; - text-indent: -9000px; - overflow:hidden; -} - -h2.subtitle { - background-color: transparent; - text-indent: -9000px; - overflow:hidden; - width: 0px; - display: none; -} - - /*************************************** / - / pippin.gimp.org specific alterations / -/ ***************************************/ - -/* -div.heading, div.navheader { - color: #777; - font-size: 80%; - padding: 0; - margin: 0; - text-align: left; - position: absolute; - top: 0px; - left: 0px; - width: 100%; - height: 50px; - background: url('/gfx/heading_bg.png') transparent; - background-repeat: repeat-x; - background-attachment: fixed; - border: none; -} - -div.heading a { - color: #444; -} - -div.footing, div.navfooter { - border: none; - color: #ddd; - font-size: 80%; - text-align:right; - - width: 100%; - padding-top: 10px; - position: absolute; - bottom: 0px; - left: 0px; - - background: url('/gfx/footing_bg.png') transparent; -} -*/ - - - - /****************** / - / nasty ie tweaks / -/ ******************/ - -/* -div.heading, div.navheader { - width:expression(document.body.clientWidth + "px"); -} - -div.footing, div.navfooter { - width:expression(document.body.clientWidth + "px"); - margin-left:expression("-5em"); -} -body { - padding:expression("4em 5em 0em 5em"); -} -*/ - - /**************************************** / - / mozilla vendor specific css extensions / -/ ****************************************/ -/* -div.navfooter, div.footing{ - -moz-opacity: 0.8em; -} - -div.figure, -div.table, -div.informalfigure, -div.informaltable, -div.informalexample, -div.example, -.tip, -.warning, -.caution, -.note { - -moz-border-radius: 0.5em; -} - -b.keycap, -.keycap { - -moz-border-radius: 0.3em; -} -*/ - -table tr td table tr td { - display: none; -} - - -hr { - display: none; -} - -table { - border: 0em; -} - - .photo { - float: right; - margin-left: 1.5em; - margin-bottom: 1.5em; - margin-top: 0em; - max-width: 17em; - border: 1px solid gray; - padding: 3px; - background: white; -} - .seperator { - padding-top: 2em; - clear: both; - } - - #validators { - margin-top: 5em; - text-align: right; - color: #777; - } - @media print { - body { - font-size: 8pt; - } - .noprint { - display: none; - } - } - - -.tip, -.note { - background: #f0f0f2; - color: #333; - padding: 20px; - margin: 20px; -} - -.tip h3, -.note h3 { - padding: 0em; - margin: 0em; - font-size: 2em; - font-weight: bold; - color: #333; -} - -.tip a, -.note a { - color: #333; - text-decoration: underline; -} - -.footnote { - font-size: small; - color: #333; -} - -/* Changes the announcement text */ -.tip h3, -.warning h3, -.caution h3, -.note h3 { - font-size:large; - color: #00557D; -} diff --git a/poky/documentation/overview-manual/overview-manual-yp-intro.rst b/poky/documentation/overview-manual/overview-manual-yp-intro.rst new file mode 100644 index 0000000000..f1c725ac27 --- /dev/null +++ b/poky/documentation/overview-manual/overview-manual-yp-intro.rst @@ -0,0 +1,941 @@ +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK + +***************************** +Introducing the Yocto Project +***************************** + +What is the Yocto Project? +========================== + +The Yocto Project is an open source collaboration project that helps +developers create custom Linux-based systems that are designed for +embedded products regardless of the product's hardware architecture. +Yocto Project provides a flexible toolset and a development environment +that allows embedded device developers across the world to collaborate +through shared technologies, software stacks, configurations, and best +practices used to create these tailored Linux images. + +Thousands of developers worldwide have discovered that Yocto Project +provides advantages in both systems and applications development, +archival and management benefits, and customizations used for speed, +footprint, and memory utilization. The project is a standard when it +comes to delivering embedded software stacks. The project allows +software customizations and build interchange for multiple hardware +platforms as well as software stacks that can be maintained and scaled. + +.. image:: figures/key-dev-elements.png + :align: center + +For further introductory information on the Yocto Project, you might be +interested in this +`article <https://www.embedded.com/electronics-blogs/say-what-/4458600/Why-the-Yocto-Project-for-my-IoT-Project->`__ +by Drew Moseley and in this short introductory +`video <https://www.youtube.com/watch?v=utZpKM7i5Z4>`__. + +The remainder of this section overviews advantages and challenges tied +to the Yocto Project. + +.. _gs-features: + +Features +-------- + +The following list describes features and advantages of the Yocto +Project: + +- *Widely Adopted Across the Industry:* Semiconductor, operating + system, software, and service vendors exist whose products and + services adopt and support the Yocto Project. For a look at the Yocto + Project community and the companies involved with the Yocto Project, + see the "COMMUNITY" and "ECOSYSTEM" tabs on the + :yocto_home:`Yocto Project <>` home page. + +- *Architecture Agnostic:* Yocto Project supports Intel, ARM, MIPS, + AMD, PPC and other architectures. Most ODMs, OSVs, and chip vendors + create and supply BSPs that support their hardware. If you have + custom silicon, you can create a BSP that supports that architecture. + + Aside from lots of architecture support, the Yocto Project fully + supports a wide range of device emulation through the Quick EMUlator + (QEMU). + +- *Images and Code Transfer Easily:* Yocto Project output can easily + move between architectures without moving to new development + environments. Additionally, if you have used the Yocto Project to + create an image or application and you find yourself not able to + support it, commercial Linux vendors such as Wind River, Mentor + Graphics, Timesys, and ENEA could take it and provide ongoing + support. These vendors have offerings that are built using the Yocto + Project. + +- *Flexibility:* Corporations use the Yocto Project many different + ways. One example is to create an internal Linux distribution as a + code base the corporation can use across multiple product groups. + Through customization and layering, a project group can leverage the + base Linux distribution to create a distribution that works for their + product needs. + +- *Ideal for Constrained Embedded and IoT devices:* Unlike a full Linux + distribution, you can use the Yocto Project to create exactly what + you need for embedded devices. You only add the feature support or + packages that you absolutely need for the device. For devices that + have display hardware, you can use available system components such + as X11, GTK+, Qt, Clutter, and SDL (among others) to create a rich + user experience. For devices that do not have a display or where you + want to use alternative UI frameworks, you can choose to not install + these components. + +- *Comprehensive Toolchain Capabilities:* Toolchains for supported + architectures satisfy most use cases. However, if your hardware + supports features that are not part of a standard toolchain, you can + easily customize that toolchain through specification of + platform-specific tuning parameters. And, should you need to use a + third-party toolchain, mechanisms built into the Yocto Project allow + for that. + +- *Mechanism Rules Over Policy:* Focusing on mechanism rather than + policy ensures that you are free to set policies based on the needs + of your design instead of adopting decisions enforced by some system + software provider. + +- *Uses a Layer Model:* The Yocto Project `layer + infrastructure <#the-yocto-project-layer-model>`__ groups related + functionality into separate bundles. You can incrementally add these + grouped functionalities to your project as needed. Using layers to + isolate and group functionality reduces project complexity and + redundancy, allows you to easily extend the system, make + customizations, and keep functionality organized. + +- *Supports Partial Builds:* You can build and rebuild individual + packages as needed. Yocto Project accomplishes this through its + `shared-state cache <#shared-state-cache>`__ (sstate) scheme. Being + able to build and debug components individually eases project + development. + +- *Releases According to a Strict Schedule:* Major releases occur on a + :doc:`six-month cycle <../ref-manual/ref-release-process>` + predictably in October and April. The most recent two releases + support point releases to address common vulnerabilities and + exposures. This predictability is crucial for projects based on the + Yocto Project and allows development teams to plan activities. + +- *Rich Ecosystem of Individuals and Organizations:* For open source + projects, the value of community is very important. Support forums, + expertise, and active developers who continue to push the Yocto + Project forward are readily available. + +- *Binary Reproducibility:* The Yocto Project allows you to be very + specific about dependencies and achieves very high percentages of + binary reproducibility (e.g. 99.8% for ``core-image-minimal``). When + distributions are not specific about which packages are pulled in and + in what order to support dependencies, other build systems can + arbitrarily include packages. + +- *License Manifest:* The Yocto Project provides a :ref:`license + manifest <dev-manual/dev-manual-common-tasks:maintaining open source license compliance during your product's lifecycle>` + for review by people who need to track the use of open source + licenses (e.g. legal teams). + +.. _gs-challenges: + +Challenges +---------- + +The following list presents challenges you might encounter when +developing using the Yocto Project: + +- *Steep Learning Curve:* The Yocto Project has a steep learning curve + and has many different ways to accomplish similar tasks. It can be + difficult to choose how to proceed when varying methods exist by + which to accomplish a given task. + +- *Understanding What Changes You Need to Make For Your Design Requires + Some Research:* Beyond the simple tutorial stage, understanding what + changes need to be made for your particular design can require a + significant amount of research and investigation. For information + that helps you transition from trying out the Yocto Project to using + it for your project, see the ":ref:`what-i-wish-id-known:what i wish i'd known about yocto project`" and + ":ref:`transitioning-to-a-custom-environment:transitioning to a custom environment for systems development`" + documents on the Yocto Project website. + +- *Project Workflow Could Be Confusing:* The `Yocto Project + workflow <#overview-development-environment>`__ could be confusing if + you are used to traditional desktop and server software development. + In a desktop development environment, mechanisms exist to easily pull + and install new packages, which are typically pre-compiled binaries + from servers accessible over the Internet. Using the Yocto Project, + you must modify your configuration and rebuild to add additional + packages. + +- *Working in a Cross-Build Environment Can Feel Unfamiliar:* When + developing code to run on a target, compilation, execution, and + testing done on the actual target can be faster than running a + BitBake build on a development host and then deploying binaries to + the target for test. While the Yocto Project does support development + tools on the target, the additional step of integrating your changes + back into the Yocto Project build environment would be required. + Yocto Project supports an intermediate approach that involves making + changes on the development system within the BitBake environment and + then deploying only the updated packages to the target. + + The Yocto Project :term:`OpenEmbedded Build System` + produces packages + in standard formats (i.e. RPM, DEB, IPK, and TAR). You can deploy + these packages into the running system on the target by using + utilities on the target such as ``rpm`` or ``ipk``. + +- *Initial Build Times Can be Significant:* Long initial build times + are unfortunately unavoidable due to the large number of packages + initially built from scratch for a fully functioning Linux system. + Once that initial build is completed, however, the shared-state + (sstate) cache mechanism Yocto Project uses keeps the system from + rebuilding packages that have not been "touched" since the last + build. The sstate mechanism significantly reduces times for + successive builds. + +The Yocto Project Layer Model +============================= + +The Yocto Project's "Layer Model" is a development model for embedded +and IoT Linux creation that distinguishes the Yocto Project from other +simple build systems. The Layer Model simultaneously supports +collaboration and customization. Layers are repositories that contain +related sets of instructions that tell the :term:`OpenEmbedded Build System` +what to do. You can +collaborate, share, and reuse layers. + +Layers can contain changes to previous instructions or settings at any +time. This powerful override capability is what allows you to customize +previously supplied collaborative or community layers to suit your +product requirements. + +You use different layers to logically separate information in your +build. As an example, you could have BSP, GUI, distro configuration, +middleware, or application layers. Putting your entire build into one +layer limits and complicates future customization and reuse. Isolating +information into layers, on the other hand, helps simplify future +customizations and reuse. You might find it tempting to keep everything +in one layer when working on a single project. However, the more modular +your Metadata, the easier it is to cope with future changes. + +.. note:: + + - Use Board Support Package (BSP) layers from silicon vendors when + possible. + + - Familiarize yourself with the `Yocto Project curated layer + index <https://www.yoctoproject.org/software-overview/layers/>`__ + or the `OpenEmbedded layer + index <http://layers.openembedded.org/layerindex/branch/master/layers/>`__. + The latter contains more layers but they are less universally + validated. + + - Layers support the inclusion of technologies, hardware components, + and software components. The :ref:`Yocto Project + Compatible <dev-manual/dev-manual-common-tasks:making sure your layer is compatible with yocto project>` + designation provides a minimum level of standardization that + contributes to a strong ecosystem. "YP Compatible" is applied to + appropriate products and software components such as BSPs, other + OE-compatible layers, and related open-source projects, allowing + the producer to use Yocto Project badges and branding assets. + +To illustrate how layers are used to keep things modular, consider +machine customizations. These types of customizations typically reside +in a special layer, rather than a general layer, called a BSP Layer. +Furthermore, the machine customizations should be isolated from recipes +and Metadata that support a new GUI environment, for example. This +situation gives you a couple of layers: one for the machine +configurations, and one for the GUI environment. It is important to +understand, however, that the BSP layer can still make machine-specific +additions to recipes within the GUI environment layer without polluting +the GUI layer itself with those machine-specific changes. You can +accomplish this through a recipe that is a BitBake append +(``.bbappend``) file, which is described later in this section. + +.. note:: + + For general information on BSP layer structure, see the + :doc:`../bsp-guide/bsp-guide` + . + +The :term:`Source Directory` +contains both general layers and BSP layers right out of the box. You +can easily identify layers that ship with a Yocto Project release in the +Source Directory by their names. Layers typically have names that begin +with the string ``meta-``. + +.. note:: + + It is not a requirement that a layer name begin with the prefix + meta- + , but it is a commonly accepted standard in the Yocto Project + community. + +For example, if you were to examine the `tree +view <https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/>`__ of the +``poky`` repository, you will see several layers: ``meta``, +``meta-skeleton``, ``meta-selftest``, ``meta-poky``, and +``meta-yocto-bsp``. Each of these repositories represents a distinct +layer. + +For procedures on how to create layers, see the +":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`" +section in the Yocto Project Development Tasks Manual. + +Components and Tools +==================== + +The Yocto Project employs a collection of components and tools used by +the project itself, by project developers, and by those using the Yocto +Project. These components and tools are open source projects and +metadata that are separate from the reference distribution +(:term:`Poky`) and the +:term:`OpenEmbedded Build System`. Most of the +components and tools are downloaded separately. + +This section provides brief overviews of the components and tools +associated with the Yocto Project. + +.. _gs-development-tools: + +Development Tools +----------------- + +The following list consists of tools that help you develop images and +applications using the Yocto Project: + +- *CROPS:* `CROPS <https://github.com/crops/poky-container/>`__ is an + open source, cross-platform development framework that leverages + `Docker Containers <https://www.docker.com/>`__. CROPS provides an + easily managed, extensible environment that allows you to build + binaries for a variety of architectures on Windows, Linux and Mac OS + X hosts. + +- *devtool:* This command-line tool is available as part of the + extensible SDK (eSDK) and is its cornerstone. You can use ``devtool`` + to help build, test, and package software within the eSDK. You can + use the tool to optionally integrate what you build into an image + built by the OpenEmbedded build system. + + The ``devtool`` command employs a number of sub-commands that allow + you to add, modify, and upgrade recipes. As with the OpenEmbedded + build system, "recipes" represent software packages within + ``devtool``. When you use ``devtool add``, a recipe is automatically + created. When you use ``devtool modify``, the specified existing + recipe is used in order to determine where to get the source code and + how to patch it. In both cases, an environment is set up so that when + you build the recipe a source tree that is under your control is used + in order to allow you to make changes to the source as desired. By + default, both new recipes and the source go into a "workspace" + directory under the eSDK. The ``devtool upgrade`` command updates an + existing recipe so that you can build it for an updated set of source + files. + + You can read about the ``devtool`` workflow in the Yocto Project + Application Development and Extensible Software Development Kit + (eSDK) Manual in the + ":ref:`sdk-manual/sdk-extensible:using \`\`devtool\`\` in your sdk workflow`" + section. + +- *Extensible Software Development Kit (eSDK):* The eSDK provides a + cross-development toolchain and libraries tailored to the contents of + a specific image. The eSDK makes it easy to add new applications and + libraries to an image, modify the source for an existing component, + test changes on the target hardware, and integrate into the rest of + the OpenEmbedded build system. The eSDK gives you a toolchain + experience supplemented with the powerful set of ``devtool`` commands + tailored for the Yocto Project environment. + + For information on the eSDK, see the :doc:`../sdk-manual/sdk-manual` Manual. + +- *Toaster:* Toaster is a web interface to the Yocto Project + OpenEmbedded build system. Toaster allows you to configure, run, and + view information about builds. For information on Toaster, see the + :doc:`../toaster-manual/toaster-manual`. + +.. _gs-production-tools: + +Production Tools +---------------- + +The following list consists of tools that help production related +activities using the Yocto Project: + +- *Auto Upgrade Helper:* This utility when used in conjunction with the + :term:`OpenEmbedded Build System` + (BitBake and + OE-Core) automatically generates upgrades for recipes that are based + on new versions of the recipes published upstream. See + :ref:`dev-manual/dev-manual-common-tasks:using the auto upgrade helper (auh)` + for how to set it up. + +- *Recipe Reporting System:* The Recipe Reporting System tracks recipe + versions available for Yocto Project. The main purpose of the system + is to help you manage the recipes you maintain and to offer a dynamic + overview of the project. The Recipe Reporting System is built on top + of the `OpenEmbedded Layer + Index <http://layers.openembedded.org/layerindex/layers/>`__, which + is a website that indexes OpenEmbedded-Core layers. + +- *Patchwork:* `Patchwork <http://jk.ozlabs.org/projects/patchwork/>`__ + is a fork of a project originally started by + `OzLabs <http://ozlabs.org/>`__. The project is a web-based tracking + system designed to streamline the process of bringing contributions + into a project. The Yocto Project uses Patchwork as an organizational + tool to handle patches, which number in the thousands for every + release. + +- *AutoBuilder:* AutoBuilder is a project that automates build tests + and quality assurance (QA). By using the public AutoBuilder, anyone + can determine the status of the current "master" branch of Poky. + + .. note:: + + AutoBuilder is based on buildbot. + + A goal of the Yocto Project is to lead the open source industry with + a project that automates testing and QA procedures. In doing so, the + project encourages a development community that publishes QA and test + plans, publicly demonstrates QA and test plans, and encourages + development of tools that automate and test and QA procedures for the + benefit of the development community. + + You can learn more about the AutoBuilder used by the Yocto Project + Autobuilder `here <&YOCTO_AB_URL;>`__. + +- *Cross-Prelink:* Prelinking is the process of pre-computing the load + addresses and link tables generated by the dynamic linker as compared + to doing this at runtime. Doing this ahead of time results in + performance improvements when the application is launched and reduced + memory usage for libraries shared by many applications. + + Historically, cross-prelink is a variant of prelink, which was + conceived by `Jakub + Jelínek <http://people.redhat.com/jakub/prelink.pdf>`__ a number of + years ago. Both prelink and cross-prelink are maintained in the same + repository albeit on separate branches. By providing an emulated + runtime dynamic linker (i.e. ``glibc``-derived ``ld.so`` emulation), + the cross-prelink project extends the prelink software's ability to + prelink a sysroot environment. Additionally, the cross-prelink + software enables the ability to work in sysroot style environments. + + The dynamic linker determines standard load address calculations + based on a variety of factors such as mapping addresses, library + usage, and library function conflicts. The prelink tool uses this + information, from the dynamic linker, to determine unique load + addresses for executable and linkable format (ELF) binaries that are + shared libraries and dynamically linked. The prelink tool modifies + these ELF binaries with the pre-computed information. The result is + faster loading and often lower memory consumption because more of the + library code can be re-used from shared Copy-On-Write (COW) pages. + + The original upstream prelink project only supports running prelink + on the end target device due to the reliance on the target device's + dynamic linker. This restriction causes issues when developing a + cross-compiled system. The cross-prelink adds a synthesized dynamic + loader that runs on the host, thus permitting cross-prelinking + without ever having to run on a read-write target filesystem. + +- *Pseudo:* Pseudo is the Yocto Project implementation of + `fakeroot <http://man.he.net/man1/fakeroot>`__, which is used to run + commands in an environment that seemingly has root privileges. + + During a build, it can be necessary to perform operations that + require system administrator privileges. For example, file ownership + or permissions might need definition. Pseudo is a tool that you can + either use directly or through the environment variable + ``LD_PRELOAD``. Either method allows these operations to succeed as + if system administrator privileges exist even when they do not. + + You can read more about Pseudo in the "`Fakeroot and + Pseudo <#fakeroot-and-pseudo>`__" section. + +.. _gs-openembedded-build-system: + +Open-Embedded Build System Components +------------------------------------- + +The following list consists of components associated with the +:term:`OpenEmbedded Build System`: + +- *BitBake:* BitBake is a core component of the Yocto Project and is + used by the OpenEmbedded build system to build images. While BitBake + is key to the build system, BitBake is maintained separately from the + Yocto Project. + + BitBake is a generic task execution engine that allows shell and + Python tasks to be run efficiently and in parallel while working + within complex inter-task dependency constraints. In short, BitBake + is a build engine that works through recipes written in a specific + format in order to perform sets of tasks. + + You can learn more about BitBake in the :doc:`BitBake User + Manual <bitbake:index>`. + +- *OpenEmbedded-Core:* OpenEmbedded-Core (OE-Core) is a common layer of + metadata (i.e. recipes, classes, and associated files) used by + OpenEmbedded-derived systems, which includes the Yocto Project. The + Yocto Project and the OpenEmbedded Project both maintain the + OpenEmbedded-Core. You can find the OE-Core metadata in the Yocto + Project :yocto_git:`Source Repositories </cgit/cgit.cgi/poky/tree/meta>`. + + Historically, the Yocto Project integrated the OE-Core metadata + throughout the Yocto Project source repository reference system + (Poky). After Yocto Project Version 1.0, the Yocto Project and + OpenEmbedded agreed to work together and share a common core set of + metadata (OE-Core), which contained much of the functionality + previously found in Poky. This collaboration achieved a long-standing + OpenEmbedded objective for having a more tightly controlled and + quality-assured core. The results also fit well with the Yocto + Project objective of achieving a smaller number of fully featured + tools as compared to many different ones. + + Sharing a core set of metadata results in Poky as an integration + layer on top of OE-Core. You can see that in this + `figure <#yp-key-dev-elements>`__. The Yocto Project combines various + components such as BitBake, OE-Core, script "glue", and documentation + for its build system. + +.. _gs-reference-distribution-poky: + +Reference Distribution (Poky) +----------------------------- + +Poky is the Yocto Project reference distribution. It contains the +:term:`OpenEmbedded Build System` +(BitBake and OE-Core) as well as a set of metadata to get you started +building your own distribution. See the +`figure <#what-is-the-yocto-project>`__ in "What is the Yocto Project?" +section for an illustration that shows Poky and its relationship with +other parts of the Yocto Project. + +To use the Yocto Project tools and components, you can download +(``clone``) Poky and use it to bootstrap your own distribution. + +.. note:: + + Poky does not contain binary files. It is a working example of how to + build your own custom Linux distribution from source. + +You can read more about Poky in the "`Reference Embedded Distribution +(Poky) <#reference-embedded-distribution>`__" section. + +.. _gs-packages-for-finished-targets: + +Packages for Finished Targets +----------------------------- + +The following lists components associated with packages for finished +targets: + +- *Matchbox:* Matchbox is an Open Source, base environment for the X + Window System running on non-desktop, embedded platforms such as + handhelds, set-top boxes, kiosks, and anything else for which screen + space, input mechanisms, or system resources are limited. + + Matchbox consists of a number of interchangeable and optional + applications that you can tailor to a specific, non-desktop platform + to enhance usability in constrained environments. + + You can find the Matchbox source in the Yocto Project + :yocto_git:`Source Repositories <>`. + +- *Opkg:* Open PacKaGe management (opkg) is a lightweight package + management system based on the itsy package (ipkg) management system. + Opkg is written in C and resembles Advanced Package Tool (APT) and + Debian Package (dpkg) in operation. + + Opkg is intended for use on embedded Linux devices and is used in + this capacity in the + `OpenEmbedded <http://www.openembedded.org/wiki/Main_Page>`__ and + `OpenWrt <https://openwrt.org/>`__ projects, as well as the Yocto + Project. + + .. note:: + + As best it can, opkg maintains backwards compatibility with ipkg + and conforms to a subset of Debian's policy manual regarding + control files. + + You can find the opkg source in the Yocto Project + :yocto_git:`Source Repositories <>`. + +.. _gs-archived-components: + +Archived Components +------------------- + +The Build Appliance is a virtual machine image that enables you to build +and boot a custom embedded Linux image with the Yocto Project using a +non-Linux development system. + +Historically, the Build Appliance was the second of three methods by +which you could use the Yocto Project on a system that was not native to +Linux. + +1. *Hob:* Hob, which is now deprecated and is no longer available since + the 2.1 release of the Yocto Project provided a rudimentary, + GUI-based interface to the Yocto Project. Toaster has fully replaced + Hob. + +2. *Build Appliance:* Post Hob, the Build Appliance became available. It + was never recommended that you use the Build Appliance as a + day-to-day production development environment with the Yocto Project. + Build Appliance was useful as a way to try out development in the + Yocto Project environment. + +3. *CROPS:* The final and best solution available now for developing + using the Yocto Project on a system not native to Linux is with + `CROPS <#gs-crops-overview>`__. + +.. _gs-development-methods: + +Development Methods +=================== + +The Yocto Project development environment usually involves a +:term:`Build Host` and target +hardware. You use the Build Host to build images and develop +applications, while you use the target hardware to test deployed +software. + +This section provides an introduction to the choices or development +methods you have when setting up your Build Host. Depending on the your +particular workflow preference and the type of operating system your +Build Host runs, several choices exist that allow you to use the Yocto +Project. + +.. note:: + + For additional detail about the Yocto Project development + environment, see the ":doc:`overview-manual-development-environment`" + chapter. + +- *Native Linux Host:* By far the best option for a Build Host. A + system running Linux as its native operating system allows you to + develop software by directly using the + :term:`BitBake` tool. You can + accomplish all aspects of development from a familiar shell of a + supported Linux distribution. + + For information on how to set up a Build Host on a system running + Linux as its native operating system, see the + ":ref:`dev-manual/dev-manual-start:setting up a native linux host`" + section in the Yocto Project Development Tasks Manual. + +- *CROss PlatformS (CROPS):* Typically, you use + `CROPS <https://github.com/crops/poky-container/>`__, which leverages + `Docker Containers <https://www.docker.com/>`__, to set up a Build + Host that is not running Linux (e.g. Microsoft Windows or macOS). + + .. note:: + + You can, however, use CROPS on a Linux-based system. + + CROPS is an open source, cross-platform development framework that + provides an easily managed, extensible environment for building + binaries targeted for a variety of architectures on Windows, macOS, + or Linux hosts. Once the Build Host is set up using CROPS, you can + prepare a shell environment to mimic that of a shell being used on a + system natively running Linux. + + For information on how to set up a Build Host with CROPS, see the + ":ref:`dev-manual/dev-manual-start:setting up to use cross platforms (crops)`" + section in the Yocto Project Development Tasks Manual. + +- *Windows Subsystem For Linux (WSLv2):* You may use Windows Subsystem + For Linux v2 to set up a build host using Windows 10. + + .. note:: + + The Yocto Project is not compatible with WSLv1, it is compatible + but not officially supported nor validated with WSLv2, if you + still decide to use WSL please upgrade to WSLv2. + + The Windows Subsystem For Linux allows Windows 10 to run a real Linux + kernel inside of a lightweight utility virtual machine (VM) using + virtualization technology. + + For information on how to set up a Build Host with WSLv2, see the + ":ref:`dev-manual/dev-manual-start:setting up to use windows subsystem for linux (wslv2)`" + section in the Yocto Project Development Tasks Manual. + +- *Toaster:* Regardless of what your Build Host is running, you can use + Toaster to develop software using the Yocto Project. Toaster is a web + interface to the Yocto Project's :term:`OpenEmbedded Build System`. + The interface + enables you to configure and run your builds. Information about + builds is collected and stored in a database. You can use Toaster to + configure and start builds on multiple remote build servers. + + For information about and how to use Toaster, see the + :doc:`../toaster-manual/toaster-manual`. + +.. _reference-embedded-distribution: + +Reference Embedded Distribution (Poky) +====================================== + +"Poky", which is pronounced *Pock*-ee, is the name of the Yocto +Project's reference distribution or Reference OS Kit. Poky contains the +:term:`OpenEmbedded Build System` +(:term:`BitBake` and +:term:`OpenEmbedded-Core (OE-Core)`) as well as a set +of :term:`Metadata` to get you started +building your own distro. In other words, Poky is a base specification +of the functionality needed for a typical embedded system as well as the +components from the Yocto Project that allow you to build a distribution +into a usable binary image. + +Poky is a combined repository of BitBake, OpenEmbedded-Core (which is +found in ``meta``), ``meta-poky``, ``meta-yocto-bsp``, and documentation +provided all together and known to work well together. You can view +these items that make up the Poky repository in the +:yocto_git:`Source Repositories </cgit/cgit.cgi/poky/tree/>`. + +.. note:: + + If you are interested in all the contents of the + poky + Git repository, see the ":ref:`ref-manual/ref-structure:top-level core components`" + section in the Yocto Project Reference Manual. + +The following figure illustrates what generally comprises Poky: + +.. image:: figures/poky-reference-distribution.png + :align: center + +- BitBake is a task executor and scheduler that is the heart of the + OpenEmbedded build system. + +- ``meta-poky``, which is Poky-specific metadata. + +- ``meta-yocto-bsp``, which are Yocto Project-specific Board Support + Packages (BSPs). + +- OpenEmbedded-Core (OE-Core) metadata, which includes shared + configurations, global variable definitions, shared classes, + packaging, and recipes. Classes define the encapsulation and + inheritance of build logic. Recipes are the logical units of software + and images to be built. + +- Documentation, which contains the Yocto Project source files used to + make the set of user manuals. + +.. note:: + + While Poky is a "complete" distribution specification and is tested + and put through QA, you cannot use it as a product "out of the box" + in its current form. + +To use the Yocto Project tools, you can use Git to clone (download) the +Poky repository then use your local copy of the reference distribution +to bootstrap your own distribution. + +.. note:: + + Poky does not contain binary files. It is a working example of how to + build your own custom Linux distribution from source. + +Poky has a regular, well established, six-month release cycle under its +own version. Major releases occur at the same time major releases (point +releases) occur for the Yocto Project, which are typically in the Spring +and Fall. For more information on the Yocto Project release schedule and +cadence, see the ":doc:`../ref-manual/ref-release-process`" chapter in the +Yocto Project Reference Manual. + +Much has been said about Poky being a "default configuration". A default +configuration provides a starting image footprint. You can use Poky out +of the box to create an image ranging from a shell-accessible minimal +image all the way up to a Linux Standard Base-compliant image that uses +a GNOME Mobile and Embedded (GMAE) based reference user interface called +Sato. + +One of the most powerful properties of Poky is that every aspect of a +build is controlled by the metadata. You can use metadata to augment +these base image types by adding metadata +`layers <#the-yocto-project-layer-model>`__ that extend functionality. +These layers can provide, for example, an additional software stack for +an image type, add a board support package (BSP) for additional +hardware, or even create a new image type. + +Metadata is loosely grouped into configuration files or package recipes. +A recipe is a collection of non-executable metadata used by BitBake to +set variables or define additional build-time tasks. A recipe contains +fields such as the recipe description, the recipe version, the license +of the package and the upstream source repository. A recipe might also +indicate that the build process uses autotools, make, distutils or any +other build process, in which case the basic functionality can be +defined by the classes it inherits from the OE-Core layer's class +definitions in ``./meta/classes``. Within a recipe you can also define +additional tasks as well as task prerequisites. Recipe syntax through +BitBake also supports both ``_prepend`` and ``_append`` operators as a +method of extending task functionality. These operators inject code into +the beginning or end of a task. For information on these BitBake +operators, see the +":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:appending and prepending (override style syntax)`" +section in the BitBake User's Manual. + +.. _openembedded-build-system-workflow: + +The OpenEmbedded Build System Workflow +====================================== + +The :term:`OpenEmbedded Build System` uses a "workflow" to +accomplish image and SDK generation. The following figure overviews that +workflow: + +.. image:: figures/YP-flow-diagram.png + :align: center + +Following is a brief summary of the "workflow": + +1. Developers specify architecture, policies, patches and configuration + details. + +2. The build system fetches and downloads the source code from the + specified location. The build system supports standard methods such + as tarballs or source code repositories systems such as Git. + +3. Once source code is downloaded, the build system extracts the sources + into a local work area where patches are applied and common steps for + configuring and compiling the software are run. + +4. The build system then installs the software into a temporary staging + area where the binary package format you select (DEB, RPM, or IPK) is + used to roll up the software. + +5. Different QA and sanity checks run throughout entire build process. + +6. After the binaries are created, the build system generates a binary + package feed that is used to create the final root file image. + +7. The build system generates the file system image and a customized + Extensible SDK (eSDK) for application development in parallel. + +For a very detailed look at this workflow, see the "`OpenEmbedded Build +System Concepts <#openembedded-build-system-build-concepts>`__" section. + +Some Basic Terms +================ + +It helps to understand some basic fundamental terms when learning the +Yocto Project. Although a list of terms exists in the ":doc:`Yocto Project +Terms <../ref-manual/ref-terms>`" section of the Yocto Project +Reference Manual, this section provides the definitions of some terms +helpful for getting started: + +- *Configuration Files:* Files that hold global definitions of + variables, user-defined variables, and hardware configuration + information. These files tell the :term:`OpenEmbedded Build System` + what to build and + what to put into the image to support a particular platform. + +- *Extensible Software Development Kit (eSDK):* A custom SDK for + application developers. This eSDK allows developers to incorporate + their library and programming changes back into the image to make + their code available to other application developers. For information + on the eSDK, see the :doc:`../sdk-manual/sdk-manual` manual. + +- *Layer:* A collection of related recipes. Layers allow you to + consolidate related metadata to customize your build. Layers also + isolate information used when building for multiple architectures. + Layers are hierarchical in their ability to override previous + specifications. You can include any number of available layers from + the Yocto Project and customize the build by adding your layers after + them. You can search the Layer Index for layers used within Yocto + Project. + + For more detailed information on layers, see the + ":ref:`dev-manual/dev-manual-common-tasks:understanding and creating layers`" + section in the Yocto Project Development Tasks Manual. For a + discussion specifically on BSP Layers, see the + ":ref:`bsp-guide/bsp:bsp layers`" section in the Yocto + Project Board Support Packages (BSP) Developer's Guide. + +- *Metadata:* A key element of the Yocto Project is the Metadata that + is used to construct a Linux distribution and is contained in the + files that the OpenEmbedded build system parses when building an + image. In general, Metadata includes recipes, configuration files, + and other information that refers to the build instructions + themselves, as well as the data used to control what things get built + and the effects of the build. Metadata also includes commands and + data used to indicate what versions of software are used, from where + they are obtained, and changes or additions to the software itself + (patches or auxiliary files) that are used to fix bugs or customize + the software for use in a particular situation. OpenEmbedded-Core is + an important set of validated metadata. + +- *OpenEmbedded Build System:* The terms "BitBake" and "build system" + are sometimes used for the OpenEmbedded Build System. + + BitBake is a task scheduler and execution engine that parses + instructions (i.e. recipes) and configuration data. After a parsing + phase, BitBake creates a dependency tree to order the compilation, + schedules the compilation of the included code, and finally executes + the building of the specified custom Linux image (distribution). + BitBake is similar to the ``make`` tool. + + During a build process, the build system tracks dependencies and + performs a native or cross-compilation of the package. As a first + step in a cross-build setup, the framework attempts to create a + cross-compiler toolchain (i.e. Extensible SDK) suited for the target + platform. + +- *OpenEmbedded-Core (OE-Core):* OE-Core is metadata comprised of + foundation recipes, classes, and associated files that are meant to + be common among many different OpenEmbedded-derived systems, + including the Yocto Project. OE-Core is a curated subset of an + original repository developed by the OpenEmbedded community that has + been pared down into a smaller, core set of continuously validated + recipes. The result is a tightly controlled and quality-assured core + set of recipes. + + You can see the Metadata in the ``meta`` directory of the Yocto + Project `Source + Repositories <http://git.yoctoproject.org/cgit/cgit.cgi>`__. + +- *Packages:* In the context of the Yocto Project, this term refers to + a recipe's packaged output produced by BitBake (i.e. a "baked + recipe"). A package is generally the compiled binaries produced from + the recipe's sources. You "bake" something by running it through + BitBake. + + It is worth noting that the term "package" can, in general, have + subtle meanings. For example, the packages referred to in the + ":ref:`ref-manual/ref-system-requirements:required packages for the build host`" + section in the Yocto Project Reference Manual are compiled binaries + that, when installed, add functionality to your Linux distribution. + + Another point worth noting is that historically within the Yocto + Project, recipes were referred to as packages - thus, the existence + of several BitBake variables that are seemingly mis-named, (e.g. + :term:`PR`, + :term:`PV`, and + :term:`PE`). + +- *Poky:* Poky is a reference embedded distribution and a reference + test configuration. Poky provides the following: + + - A base-level functional distro used to illustrate how to customize + a distribution. + + - A means by which to test the Yocto Project components (i.e. Poky + is used to validate the Yocto Project). + + - A vehicle through which you can download the Yocto Project. + + Poky is not a product level distro. Rather, it is a good starting + point for customization. + + .. note:: + + Poky is an integration layer on top of OE-Core. + +- *Recipe:* The most common form of metadata. A recipe contains a list + of settings and tasks (i.e. instructions) for building packages that + are then used to build the binary image. A recipe describes where you + get source code and which patches to apply. Recipes describe + dependencies for libraries or for other recipes as well as + configuration and compilation options. Related recipes are + consolidated into a layer. diff --git a/poky/documentation/overview-manual/overview-manual-yp-intro.xml b/poky/documentation/overview-manual/overview-manual-yp-intro.xml deleted file mode 100644 index 1b60a30302..0000000000 --- a/poky/documentation/overview-manual/overview-manual-yp-intro.xml +++ /dev/null @@ -1,1332 +0,0 @@ -<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" -"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" -[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > - -<chapter id='overview-yp'> - <title>Introducing the Yocto Project</title> - - <section id='what-is-the-yocto-project'> - <title>What is the Yocto Project?</title> - - <para> - The Yocto Project is an open source collaboration project - that helps developers create custom Linux-based systems that are - designed for embedded products regardless of the product's hardware - architecture. - Yocto Project provides a flexible toolset and a development - environment that allows embedded device developers across the - world to collaborate through shared technologies, software stacks, - configurations, and best practices used to create these tailored - Linux images. - </para> - - <para> - Thousands of developers worldwide have discovered that Yocto - Project provides advantages in both systems and applications - development, archival and management benefits, and customizations - used for speed, footprint, and memory utilization. - The project is a standard when it comes to delivering embedded - software stacks. - The project allows software customizations and build interchange - for multiple hardware platforms as well as software stacks that - can be maintained and scaled. - </para> - - <para id='yp-key-dev-elements'> - <imagedata fileref="figures/key-dev-elements.png" format="PNG" align='center' width="8in"/> - </para> - - <para> - For further introductory information on the Yocto Project, you - might be interested in this - <ulink url='https://www.embedded.com/electronics-blogs/say-what-/4458600/Why-the-Yocto-Project-for-my-IoT-Project-'>article</ulink> - by Drew Moseley and in this short introductory - <ulink url='https://www.youtube.com/watch?v=utZpKM7i5Z4'>video</ulink>. - </para> - - <para> - The remainder of this section overviews advantages and challenges - tied to the Yocto Project. - </para> - - <section id='gs-features'> - <title>Features</title> - - <para> - The following list describes features and advantages of the - Yocto Project: - <itemizedlist> - <listitem><para> - <emphasis>Widely Adopted Across the Industry:</emphasis> - Semiconductor, operating system, software, and - service vendors exist whose products and services - adopt and support the Yocto Project. - For a look at the Yocto Project community and - the companies involved with the Yocto - Project, see the "COMMUNITY" and "ECOSYSTEM" tabs - on the - <ulink url='&YOCTO_HOME_URL;'>Yocto Project</ulink> - home page. - </para></listitem> - <listitem><para> - <emphasis>Architecture Agnostic:</emphasis> - Yocto Project supports Intel, ARM, MIPS, AMD, PPC - and other architectures. - Most ODMs, OSVs, and chip vendors create and supply - BSPs that support their hardware. - If you have custom silicon, you can create a BSP - that supports that architecture.</para> - - <para>Aside from lots of architecture support, the - Yocto Project fully supports a wide range of device - emulation through the Quick EMUlator (QEMU). - </para></listitem> - <listitem><para> - <emphasis>Images and Code Transfer Easily:</emphasis> - Yocto Project output can easily move between - architectures without moving to new development - environments. - Additionally, if you have used the Yocto Project to - create an image or application and you find yourself - not able to support it, commercial Linux vendors such - as Wind River, Mentor Graphics, Timesys, and ENEA could - take it and provide ongoing support. - These vendors have offerings that are built using - the Yocto Project. - </para></listitem> - <listitem><para> - <emphasis>Flexibility:</emphasis> - Corporations use the Yocto Project many different ways. - One example is to create an internal Linux distribution - as a code base the corporation can use across multiple - product groups. - Through customization and layering, a project group - can leverage the base Linux distribution to create - a distribution that works for their product needs. - </para></listitem> - <listitem><para> - <emphasis>Ideal for Constrained Embedded and IoT devices:</emphasis> - Unlike a full Linux distribution, you can use the - Yocto Project to create exactly what you need for - embedded devices. - You only add the feature support or packages that you - absolutely need for the device. - For devices that have display hardware, you can use - available system components such as X11, GTK+, Qt, - Clutter, and SDL (among others) to create a rich user - experience. - For devices that do not have a display or where you - want to use alternative UI frameworks, you can choose - to not install these components. - </para></listitem> - <listitem><para> - <emphasis>Comprehensive Toolchain Capabilities:</emphasis> - Toolchains for supported architectures satisfy most - use cases. - However, if your hardware supports features that are - not part of a standard toolchain, you can easily - customize that toolchain through specification of - platform-specific tuning parameters. - And, should you need to use a third-party toolchain, - mechanisms built into the Yocto Project allow for that. - </para></listitem> - <listitem><para> - <emphasis>Mechanism Rules Over Policy:</emphasis> - Focusing on mechanism rather than policy ensures that - you are free to set policies based on the needs of your - design instead of adopting decisions enforced by some - system software provider. - </para></listitem> - <listitem><para> - <emphasis>Uses a Layer Model:</emphasis> - The Yocto Project - <link linkend='the-yocto-project-layer-model'>layer infrastructure</link> - groups related functionality into separate bundles. - You can incrementally add these grouped functionalities - to your project as needed. - Using layers to isolate and group functionality - reduces project complexity and redundancy, allows you - to easily extend the system, make customizations, - and keep functionality organized. - </para></listitem> - <listitem><para> - <emphasis>Supports Partial Builds:</emphasis> - You can build and rebuild individual packages as - needed. - Yocto Project accomplishes this through its - <link linkend='shared-state-cache'>shared-state cache</link> - (sstate) scheme. - Being able to build and debug components individually - eases project development. - </para></listitem> - <listitem><para> - <emphasis>Releases According to a Strict Schedule:</emphasis> - Major releases occur on a - <ulink url='&YOCTO_DOCS_REF_URL;#ref-release-process'>six-month cycle</ulink> - predictably in October and April. - The most recent two releases support point releases - to address common vulnerabilities and exposures. - This predictability is crucial for projects based on - the Yocto Project and allows development teams to - plan activities. - </para></listitem> - <listitem><para> - <emphasis>Rich Ecosystem of Individuals and Organizations:</emphasis> - For open source projects, the value of community is - very important. - Support forums, expertise, and active developers who - continue to push the Yocto Project forward are readily - available. - </para></listitem> - <listitem><para> - <emphasis>Binary Reproducibility:</emphasis> - The Yocto Project allows you to be very specific about - dependencies and achieves very high percentages of - binary reproducibility (e.g. 99.8% for - <filename>core-image-minimal</filename>). - When distributions are not specific about which - packages are pulled in and in what order to support - dependencies, other build systems can arbitrarily - include packages. - </para></listitem> - <listitem><para> - <emphasis>License Manifest:</emphasis> - The Yocto Project provides a - <ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>license manifest</ulink> - for review by people who need to track the use of open - source licenses (e.g.legal teams). - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='gs-challenges'> - <title>Challenges</title> - - <para> - The following list presents challenges you might encounter - when developing using the Yocto Project: - <itemizedlist> - <listitem><para> - <emphasis>Steep Learning Curve:</emphasis> - The Yocto Project has a steep learning curve and has - many different ways to accomplish similar tasks. - It can be difficult to choose how to proceed when - varying methods exist by which to accomplish a given - task. - </para></listitem> - <listitem><para> - <emphasis>Understanding What Changes You Need to Make - For Your Design Requires Some Research:</emphasis> - Beyond the simple tutorial stage, understanding what - changes need to be made for your particular design - can require a significant amount of research and - investigation. - For information that helps you transition from - trying out the Yocto Project to using it for your - project, see the - "<ulink url='&YOCTO_DOCS_URL;/what-i-wish-id-known/'>What I wish I'd Known</ulink>" - and - "<ulink url='&YOCTO_DOCS_URL;/transitioning-to-a-custom-environment/'>Transitioning to a Custom Environment for Systems Development</ulink>" - documents on the Yocto Project website. - </para></listitem> - <listitem><para> - <emphasis>Project Workflow Could Be Confusing:</emphasis> - The - <link linkend='overview-development-environment'>Yocto Project workflow</link> - could be confusing if you are used to traditional - desktop and server software development. - In a desktop development environment, mechanisms exist - to easily pull and install new packages, which are - typically pre-compiled binaries from servers accessible - over the Internet. - Using the Yocto Project, you must modify your - configuration and rebuild to add additional packages. - </para></listitem> - <listitem><para> - <emphasis>Working in a Cross-Build Environment Can - Feel Unfamiliar:</emphasis> - When developing code to run on a target, compilation, - execution, and testing done on the actual target - can be faster than running a BitBake build on a - development host and then deploying binaries to the - target for test. - While the Yocto Project does support development tools - on the target, the additional step of integrating your - changes back into the Yocto Project build environment - would be required. - Yocto Project supports an intermediate approach that - involves making changes on the development system - within the BitBake environment and then deploying only - the updated packages to the target.</para> - - <para>The Yocto Project - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> - produces packages in standard formats (i.e. RPM, - DEB, IPK, and TAR). - You can deploy these packages into the running system - on the target by using utilities on the target such - as <filename>rpm</filename> or - <filename>ipk</filename>. - </para></listitem> - <listitem><para> - <emphasis>Initial Build Times Can be Significant:</emphasis> - Long initial build times are unfortunately unavoidable - due to the large number of packages initially built - from scratch for a fully functioning Linux system. - Once that initial build is completed, however, the - shared-state (sstate) cache mechanism Yocto Project - uses keeps the system from rebuilding packages that - have not been "touched" since the last build. - The sstate mechanism significantly reduces times - for successive builds. - </para></listitem> - </itemizedlist> - </para> - </section> - </section> - - <section id='the-yocto-project-layer-model'> - <title>The Yocto Project Layer Model</title> - - <para> - The Yocto Project's "Layer Model" is a development model for - embedded and IoT Linux creation that distinguishes the - Yocto Project from other simple build systems. - The Layer Model simultaneously supports collaboration and - customization. - Layers are repositories that contain related sets of instructions - that tell the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> - what to do. - You can collaborate, share, and reuse layers. - </para> - - <para> - Layers can contain changes to previous instructions or settings - at any time. - This powerful override capability is what allows you to customize - previously supplied collaborative or community layers to suit your - product requirements. - </para> - - <para> - You use different layers to logically separate information in your - build. - As an example, you could have BSP, GUI, distro configuration, - middleware, or application layers. - Putting your entire build into one layer limits and complicates - future customization and reuse. - Isolating information into layers, on the other hand, helps - simplify future customizations and reuse. - You might find it tempting to keep everything in one layer when - working on a single project. - However, the more modular your Metadata, the easier - it is to cope with future changes. - <note><title>Notes</title> - <itemizedlist> - <listitem><para> - Use Board Support Package (BSP) layers from silicon - vendors when possible. - </para></listitem> - <listitem><para> - Familiarize yourself with the - <ulink url='https://caffelli-staging.yoctoproject.org/software-overview/layers/'>Yocto Project curated layer index</ulink> - or the - <ulink url='http://layers.openembedded.org/layerindex/branch/master/layers/'>OpenEmbedded layer index</ulink>. - The latter contains more layers but they are less - universally validated. - </para></listitem> - <listitem><para> - Layers support the inclusion of technologies, hardware - components, and software components. - The - <ulink url='&YOCTO_DOCS_DEV_URL;#making-sure-your-layer-is-compatible-with-yocto-project'>Yocto Project Compatible</ulink> - designation provides a minimum level of standardization - that contributes to a strong ecosystem. - "YP Compatible" is applied to appropriate products and - software components such as BSPs, other OE-compatible - layers, and related open-source projects, allowing the - producer to use Yocto Project badges and branding - assets. - </para></listitem> - </itemizedlist> - </note> - </para> - - <para> - To illustrate how layers are used to keep things modular, consider - machine customizations. - These types of customizations typically reside in a special layer, - rather than a general layer, called a BSP Layer. - Furthermore, the machine customizations should be isolated from - recipes and Metadata that support a new GUI environment, - for example. - This situation gives you a couple of layers: one for the machine - configurations, and one for the GUI environment. - It is important to understand, however, that the BSP layer can - still make machine-specific additions to recipes within the GUI - environment layer without polluting the GUI layer itself - with those machine-specific changes. - You can accomplish this through a recipe that is a BitBake append - (<filename>.bbappend</filename>) file, which is described later - in this section. - <note> - For general information on BSP layer structure, see the - <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Packages (BSP) Developer's Guide</ulink>. - </note> - </para> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> - contains both general layers and BSP layers right out of the box. - You can easily identify layers that ship with a Yocto Project - release in the Source Directory by their names. - Layers typically have names that begin with the string - <filename>meta-</filename>. - <note> - It is not a requirement that a layer name begin with the - prefix <filename>meta-</filename>, but it is a commonly - accepted standard in the Yocto Project community. - </note> - For example, if you were to examine the - <ulink url='https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/'>tree view</ulink> - of the <filename>poky</filename> repository, you will see several - layers: <filename>meta</filename>, - <filename>meta-skeleton</filename>, - <filename>meta-selftest</filename>, - <filename>meta-poky</filename>, and - <filename>meta-yocto-bsp</filename>. - Each of these repositories represents a distinct layer. - </para> - - <para> - For procedures on how to create layers, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>" - section in the Yocto Project Development Tasks Manual. - </para> - </section> - - <section id='components-and-tools'> - <title>Components and Tools</title> - - <para> - The Yocto Project employs a collection of components and - tools used by the project itself, by project developers, - and by those using the Yocto Project. - These components and tools are open source projects and - metadata that are separate from the reference distribution - (<ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink>) - and the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>. - Most of the components and tools are downloaded separately. - </para> - - <para> - This section provides brief overviews of the components and - tools associated with the Yocto Project. - </para> - - <section id='gs-development-tools'> - <title>Development Tools</title> - - <para> - The following list consists of tools that help you develop - images and applications using the Yocto Project: - <itemizedlist> - <listitem><para id='gs-crops-overview'> - <emphasis>CROPS:</emphasis> - <ulink url='https://github.com/crops/poky-container/'>CROPS</ulink> - is an open source, cross-platform development framework - that leverages - <ulink url='https://www.docker.com/'>Docker Containers</ulink>. - CROPS provides an easily managed, extensible environment - that allows you to build binaries for a variety of - architectures on Windows, Linux and Mac OS X hosts. - </para></listitem> - <listitem><para> - <emphasis><filename>devtool</filename>:</emphasis> - This command-line tool is available as part of the - extensible SDK (eSDK) and is its cornerstone. - You can use <filename>devtool</filename> to help build, - test, and package software within the eSDK. - You can use the tool to optionally integrate what you - build into an image built by the OpenEmbedded build - system.</para> - - <para>The <filename>devtool</filename> command employs - a number of sub-commands that allow you to add, modify, - and upgrade recipes. - As with the OpenEmbedded build system, “recipes” - represent software packages within - <filename>devtool</filename>. - When you use <filename>devtool add</filename>, a recipe - is automatically created. - When you use <filename>devtool modify</filename>, the - specified existing recipe is used in order to determine - where to get the source code and how to patch it. - In both cases, an environment is set up so that when - you build the recipe a source tree that is under your - control is used in order to allow you to make changes - to the source as desired. - By default, both new recipes and the source go into - a “workspace” directory under the eSDK. - The <filename>devtool upgrade</filename> command - updates an existing recipe so that you can build it - for an updated set of source files.</para> - - <para>You can read about the - <filename>devtool</filename> workflow in the Yocto - Project Application Development and Extensible - Software Development Kit (eSDK) Manual in the - "<ulink url='&YOCTO_DOCS_SDK_URL;#using-devtool-in-your-sdk-workflow'>Using <filename>devtool</filename> in Your SDK Workflow'</ulink>" - section. - </para></listitem> - <listitem><para> - <emphasis>Extensible Software Development Kit (eSDK):</emphasis> - The eSDK provides a cross-development toolchain and - libraries tailored to the contents of a specific image. - The eSDK makes it easy to add new applications and - libraries to an image, modify the source for an - existing component, test changes on the target - hardware, and integrate into the rest of the - OpenEmbedded build system. - The eSDK gives you a toolchain experience supplemented - with the powerful set of <filename>devtool</filename> - commands tailored for the Yocto Project environment. - </para> - - <para>For information on the eSDK, see the - <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> - Manual. - </para></listitem> - <listitem><para> - <emphasis>Toaster:</emphasis> - Toaster is a web interface to the Yocto Project - OpenEmbedded build system. - Toaster allows you to configure, run, and view - information about builds. - For information on Toaster, see the - <ulink url='&YOCTO_DOCS_TOAST_URL;'>Toaster User Manual</ulink>. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='gs-production-tools'> - <title>Production Tools</title> - - <para> - The following list consists of tools that help production - related activities using the Yocto Project: - <itemizedlist> - <listitem><para> - <emphasis>Auto Upgrade Helper:</emphasis> - This utility when used in conjunction with the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> - (BitBake and OE-Core) automatically generates upgrades - for recipes that are based on new versions of the - recipes published upstream. - </para></listitem> - <listitem><para> - <emphasis>Recipe Reporting System:</emphasis> - The Recipe Reporting System tracks recipe versions - available for Yocto Project. - The main purpose of the system is to help you - manage the recipes you maintain and to offer a dynamic - overview of the project. - The Recipe Reporting System is built on top of the - <ulink url="http://layers.openembedded.org/layerindex/layers/">OpenEmbedded Layer Index</ulink>, - which is a website that indexes OpenEmbedded-Core - layers. - </para></listitem> - <listitem><para> - <emphasis>Patchwork:</emphasis> - <ulink url='http://jk.ozlabs.org/projects/patchwork/'>Patchwork</ulink> - is a fork of a project originally started by - <ulink url='http://ozlabs.org/'>OzLabs</ulink>. - The project is a web-based tracking system designed - to streamline the process of bringing contributions - into a project. - The Yocto Project uses Patchwork as an organizational - tool to handle patches, which number in the thousands - for every release. - </para></listitem> - <listitem><para> - <emphasis>AutoBuilder:</emphasis> - AutoBuilder is a project that automates build tests - and quality assurance (QA). - By using the public AutoBuilder, anyone can determine - the status of the current "master" branch of Poky. - <note> - AutoBuilder is based on - <ulink url='https://buildbot.net/'>buildbot</ulink>. - </note></para> - - <para>A goal of the Yocto Project is to lead the - open source industry with a project that automates - testing and QA procedures. - In doing so, the project encourages a development - community that publishes QA and test plans, publicly - demonstrates QA and test plans, and encourages - development of tools that automate and test and QA - procedures for the benefit of the development - community.</para> - - <para>You can learn more about the AutoBuilder used - by the Yocto Project - <ulink url='&YOCTO_AB_URL;'>here</ulink>. - </para></listitem> - <listitem><para> - <emphasis>Cross-Prelink:</emphasis> - Prelinking is the process of pre-computing the load - addresses and link tables generated by the dynamic - linker as compared to doing this at runtime. - Doing this ahead of time results in performance - improvements when the application is launched and - reduced memory usage for libraries shared by many - applications.</para> - - <para>Historically, cross-prelink is a variant of - prelink, which was conceived by - <ulink url='http://people.redhat.com/jakub/prelink.pdf'>Jakub Jelínek</ulink> - a number of years ago. - Both prelink and cross-prelink are maintained in the - same repository albeit on separate branches. - By providing an emulated runtime dynamic linker - (i.e. <filename>glibc</filename>-derived - <filename>ld.so</filename> emulation), the - cross-prelink project extends the prelink software’s - ability to prelink a sysroot environment. - Additionally, the cross-prelink software enables the - ability to work in sysroot style environments.</para> - - <para>The dynamic linker determines standard load - address calculations based on a variety of factors - such as mapping addresses, library usage, and library - function conflicts. - The prelink tool uses this information, from the - dynamic linker, to determine unique load addresses - for executable and linkable format (ELF) binaries - that are shared libraries and dynamically linked. - The prelink tool modifies these ELF binaries with the - pre-computed information. - The result is faster loading and often lower memory - consumption because more of the library code can - be re-used from shared Copy-On-Write (COW) pages. - </para> - - <para>The original upstream prelink project only - supports running prelink on the end target device - due to the reliance on the target device’s dynamic - linker. - This restriction causes issues when developing a - cross-compiled system. - The cross-prelink adds a synthesized dynamic loader - that runs on the host, thus permitting cross-prelinking - without ever having to run on a read-write target - filesystem. - </para></listitem> - <listitem><para> - <emphasis>Pseudo:</emphasis> - Pseudo is the Yocto Project implementation of - <ulink url='http://man.he.net/man1/fakeroot'>fakeroot</ulink>, - which is used to run commands in an environment - that seemingly has root privileges.</para> - - <para>During a build, it can be necessary to perform - operations that require system administrator - privileges. - For example, file ownership or permissions might need - definition. - Pseudo is a tool that you can either use directly or - through the environment variable - <filename>LD_PRELOAD</filename>. - Either method allows these operations to succeed as - if system administrator privileges exist even - when they do not.</para> - - <para>You can read more about Pseudo in the - "<link linkend='fakeroot-and-pseudo'>Fakeroot and Pseudo</link>" - section. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='gs-openembedded-build-system'> - <title>Open-Embedded Build System Components</title> - - <para> - The following list consists of components associated with the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink>: - <itemizedlist> - <listitem><para> - <emphasis>BitBake:</emphasis> - BitBake is a core component of the Yocto Project and is - used by the OpenEmbedded build system to build images. - While BitBake is key to the build system, BitBake - is maintained separately from the Yocto Project.</para> - - <para>BitBake is a generic task execution engine that - allows shell and Python tasks to be run efficiently - and in parallel while working within complex inter-task - dependency constraints. - In short, BitBake is a build engine that works - through recipes written in a specific format in order - to perform sets of tasks.</para> - - <para>You can learn more about BitBake in the - <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>. - </para></listitem> - <listitem><para> - <emphasis>OpenEmbedded-Core:</emphasis> - OpenEmbedded-Core (OE-Core) is a common layer of - metadata (i.e. recipes, classes, and associated files) - used by OpenEmbedded-derived systems, which includes - the Yocto Project. - The Yocto Project and the OpenEmbedded Project both - maintain the OpenEmbedded-Core. - You can find the OE-Core metadata in the Yocto Project - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta'>Source Repositories</ulink>. - </para> - - <para>Historically, the Yocto Project integrated the - OE-Core metadata throughout the Yocto Project - source repository reference system (Poky). - After Yocto Project Version 1.0, the Yocto Project - and OpenEmbedded agreed to work together and share a - common core set of metadata (OE-Core), which contained - much of the functionality previously found in Poky. - This collaboration achieved a long-standing - OpenEmbedded objective for having a more tightly - controlled and quality-assured core. - The results also fit well with the Yocto Project - objective of achieving a smaller number of fully - featured tools as compared to many different ones. - </para> - - <para>Sharing a core set of metadata results in Poky - as an integration layer on top of OE-Core. - You can see that in this - <link linkend='yp-key-dev-elements'>figure</link>. - The Yocto Project combines various components such as - BitBake, OE-Core, script “glue”, and documentation - for its build system. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='gs-reference-distribution-poky'> - <title>Reference Distribution (Poky)</title> - - <para> - Poky is the Yocto Project reference distribution. - It contains the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>Open-Embedded build system</ulink> - (BitBake and OE-Core) as well as a set of metadata to get you - started building your own distribution. - See the - <link linkend='what-is-the-yocto-project'>figure</link> in - "What is the Yocto Project?" section for an illustration - that shows Poky and its relationship with other parts of the - Yocto Project.</para> - - <para>To use the Yocto Project tools and components, you - can download (<filename>clone</filename>) Poky and use it - to bootstrap your own distribution. - <note> - Poky does not contain binary files. - It is a working example of how to build your own custom - Linux distribution from source. - </note> - You can read more about Poky in the - "<link linkend='reference-embedded-distribution'>Reference Embedded Distribution (Poky)</link>" - section. - </para> - </section> - - <section id='gs-packages-for-finished-targets'> - <title>Packages for Finished Targets</title> - - <para> - The following lists components associated with packages - for finished targets: - <itemizedlist> - <listitem><para> - <emphasis>Matchbox:</emphasis> - Matchbox is an Open Source, base environment for the - X Window System running on non-desktop, embedded - platforms such as handhelds, set-top boxes, kiosks, - and anything else for which screen space, input - mechanisms, or system resources are limited.</para> - - <para>Matchbox consists of a number of interchangeable - and optional applications that you can tailor to a - specific, non-desktop platform to enhance usability - in constrained environments.</para> - - <para>You can find the Matchbox source in the Yocto - Project - <ulink url='&YOCTO_GIT_URL;'>Source Repositories</ulink>. - </para></listitem> - <listitem><para> - <emphasis>Opkg</emphasis> - Open PacKaGe management (opkg) is a lightweight - package management system based on the itsy package - (ipkg) management system. - Opkg is written in C and resembles Advanced Package - Tool (APT) and Debian Package (dpkg) in operation. - </para> - - <para>Opkg is intended for use on embedded Linux - devices and is used in this capacity in the - <ulink url='http://www.openembedded.org/wiki/Main_Page'>OpenEmbedded</ulink> - and - <ulink url='https://openwrt.org/'>OpenWrt</ulink> - projects, as well as the Yocto Project. - <note> - As best it can, opkg maintains backwards - compatibility with ipkg and conforms to a subset - of Debian’s policy manual regarding control files. - </note> - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='gs-archived-components'> - <title>Archived Components</title> - - <para> - The Build Appliance is a virtual machine image that enables - you to build and boot a custom embedded Linux image with - the Yocto Project using a non-Linux development system. - </para> - - <para> - Historically, the Build Appliance was the second of three - methods by which you could use the Yocto Project on a system - that was not native to Linux. - <orderedlist> - <listitem><para> - <emphasis>Hob:</emphasis> - Hob, which is now deprecated and is no longer available - since the 2.1 release of the Yocto Project provided - a rudimentary, GUI-based interface to the Yocto - Project. - Toaster has fully replaced Hob. - </para></listitem> - <listitem><para> - <emphasis>Build Appliance:</emphasis> - Post Hob, the Build Appliance became available. - It was never recommended that you use the Build - Appliance as a day-to-day production development - environment with the Yocto Project. - Build Appliance was useful as a way to try out - development in the Yocto Project environment. - </para></listitem> - <listitem><para> - <emphasis>CROPS:</emphasis> - The final and best solution available now for - developing using the Yocto Project on a system - not native to Linux is with - <link linkend='gs-crops-overview'>CROPS</link>. - </para></listitem> - </orderedlist> - </para> - </section> - </section> - - <section id='gs-development-methods'> - <title>Development Methods</title> - - <para> - The Yocto Project development environment usually involves a - <ulink url='&YOCTO_DOCS_REF_URL;#hardware-build-system-term'>Build Host</ulink> - and target hardware. - You use the Build Host to build images and develop applications, - while you use the target hardware to test deployed software. - </para> - - <para> - This section provides an introduction to the choices or - development methods you have when setting up your Build Host. - Depending on the your particular workflow preference and the - type of operating system your Build Host runs, several choices - exist that allow you to use the Yocto Project. - <note> - For additional detail about the Yocto Project development - environment, see the - "<link linkend='overview-development-environment'>The Yocto Project Development Environment</link>" - chapter. - </note> - <itemizedlist> - <listitem><para> - <emphasis>Native Linux Host:</emphasis> - By far the best option for a Build Host. - A system running Linux as its native operating system - allows you to develop software by directly using the - <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> - tool. - You can accomplish all aspects of development from a - familiar shell of a supported Linux distribution.</para> - - <para>For information on how to set up a Build Host on - a system running Linux as its native operating system, - see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-a-native-linux-host'>Setting Up a Native Linux Host</ulink>" - section in the Yocto Project Development Tasks Manual. - </para></listitem> - <listitem><para> - <emphasis>CROss PlatformS (CROPS):</emphasis> - Typically, you use - <ulink url='https://github.com/crops/poky-container/'>CROPS</ulink>, - which leverages - <ulink url='https://www.docker.com/'>Docker Containers</ulink>, - to set up a Build Host that is not running Linux (e.g. - <trademark class='registered'>Microsoft</trademark> - <trademark class='trademark'>Windows</trademark> - or - <trademark class='registered'>macOS</trademark>). - <note> - You can, however, use CROPS on a Linux-based system. - </note> - CROPS is an open source, cross-platform development - framework that provides an easily managed, extensible - environment for building binaries targeted for a variety - of architectures on Windows, macOS, or Linux hosts. - Once the Build Host is set up using CROPS, you can prepare - a shell environment to mimic that of a shell being used - on a system natively running Linux.</para> - - <para>For information on how to set up a Build Host with - CROPS, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-to-use-crops'>Setting Up to Use CROss PlatformS (CROPS)</ulink>" - section in the Yocto Project Development Tasks Manual. - </para></listitem> - <listitem><para> - <emphasis>Windows Subsystem For Linux (WSLv2):</emphasis> - You may use Windows Subsystem For Linux v2 to set up a build - host using Windows 10. - <note> - The Yocto Project is not compatible with WSLv1, it is - compatible but not officially supported nor validated - with WSLv2, if you still decide to use WSL please upgrade - to WSLv2. - </note> - The Windows Subsystem For Linux allows Windows 10 to run a real - Linux kernel inside of a lightweight utility virtual - machine (VM) using virtualization technology.</para> - <para>For information on how to set up a Build Host with - WSLv2, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#setting-up-to-use-wsl'>Setting Up to Use Windows Subsystem For Linux</ulink>" - section in the Yocto Project Development Tasks Manual. - </para></listitem> - <listitem><para> - <emphasis>Toaster:</emphasis> - Regardless of what your Build Host is running, you can - use Toaster to develop software using the Yocto Project. - Toaster is a web interface to the Yocto Project's - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>Open-Embedded build system</ulink>. - The interface enables you to configure and run your - builds. - Information about builds is collected and stored in a - database. - You can use Toaster to configure and start builds on - multiple remote build servers.</para> - - <para>For information about and how to use Toaster, - see the - <ulink url='&YOCTO_DOCS_TOAST_URL;'>Toaster User Manual</ulink>. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='reference-embedded-distribution'> - <title>Reference Embedded Distribution (Poky)</title> - - <para> - "Poky", which is pronounced <emphasis>Pock</emphasis>-ee, is the - name of the Yocto Project's reference distribution or Reference OS - Kit. - Poky contains the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded Build System</ulink> - (<ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> and - <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core</ulink>) - as well as a set of - <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>metadata</ulink> to get - you started building your own distro. - In other words, Poky is a base specification of the functionality - needed for a typical embedded system as well as the components - from the Yocto Project that allow you to build a distribution into - a usable binary image. - </para> - - <para> - Poky is a combined repository of BitBake, OpenEmbedded-Core - (which is found in <filename>meta</filename>), - <filename>meta-poky</filename>, - <filename>meta-yocto-bsp</filename>, and documentation provided - all together and known to work well together. - You can view these items that make up the Poky repository in the - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/'>Source Repositories</ulink>. - <note> - If you are interested in all the contents of the - <filename>poky</filename> Git repository, see the - "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core'>Top-Level Core Components</ulink>" - section in the Yocto Project Reference Manual. - </note> - </para> - - <para id='gs-poky-reference-distribution'> - The following figure illustrates what generally comprises Poky: - <imagedata fileref="figures/poky-reference-distribution.png" format="PNG" align='center' width="8in"/> - <itemizedlist> - <listitem><para> - BitBake is a task executor and scheduler that is the heart of - the OpenEmbedded build system. - </para></listitem> - <listitem><para> - <filename>meta-poky</filename>, which is Poky-specific - metadata. - </para></listitem> - <listitem><para> - <filename>meta-yocto-bsp</filename>, which are Yocto - Project-specific Board Support Packages (BSPs). - </para></listitem> - <listitem><para> - OpenEmbedded-Core (OE-Core) metadata, which includes - shared configurations, global variable definitions, - shared classes, packaging, and recipes. - Classes define the encapsulation and inheritance of build - logic. - Recipes are the logical units of software and images - to be built. - </para></listitem> - <listitem><para> - Documentation, which contains the Yocto Project source - files used to make the set of user manuals. - </para></listitem> - </itemizedlist> - <note> - While Poky is a "complete" distribution specification and is - tested and put through QA, you cannot use it as a product - "out of the box" in its current form. - </note> - </para> - - <para> - To use the Yocto Project tools, you can use Git to clone (download) - the Poky repository then use your local copy of the reference - distribution to bootstrap your own distribution. - <note> - Poky does not contain binary files. - It is a working example of how to build your own custom Linux distribution - from source. - </note> - </para> - - <para> - Poky has a regular, well established, six-month release cycle - under its own version. - Major releases occur at the same time major releases (point - releases) occur for the Yocto Project, which are typically in the - Spring and Fall. - For more information on the Yocto Project release schedule and - cadence, see the - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-release-process'>Yocto Project Releases and the Stable Release Process</ulink>" - chapter in the Yocto Project Reference Manual. - </para> - - <para> - Much has been said about Poky being a "default configuration." - A default configuration provides a starting image footprint. - You can use Poky out of the box to create an image ranging from a - shell-accessible minimal image all the way up to a Linux - Standard Base-compliant image that uses a GNOME Mobile and - Embedded (GMAE) based reference user interface called Sato. - </para> - - <para> - One of the most powerful properties of Poky is that every aspect - of a build is controlled by the metadata. - You can use metadata to augment these base image types by - adding metadata - <link linkend='the-yocto-project-layer-model'>layers</link> - that extend functionality. - These layers can provide, for example, an additional software - stack for an image type, add a board support package (BSP) for - additional hardware, or even create a new image type. - </para> - - <para> - Metadata is loosely grouped into configuration files or package - recipes. - A recipe is a collection of non-executable metadata used by - BitBake to set variables or define additional build-time tasks. - A recipe contains fields such as the recipe description, the recipe - version, the license of the package and the upstream source - repository. - A recipe might also indicate that the build process uses autotools, - make, distutils or any other build process, in which case the basic - functionality can be defined by the classes it inherits from - the OE-Core layer's class definitions in - <filename>./meta/classes</filename>. - Within a recipe you can also define additional tasks as well as - task prerequisites. - Recipe syntax through BitBake also supports both - <filename>_prepend</filename> and <filename>_append</filename> - operators as a method of extending task functionality. - These operators inject code into the beginning or end of a task. - For information on these BitBake operators, see the - "<ulink url='&YOCTO_DOCS_BB_URL;#appending-and-prepending-override-style-syntax'>Appending and Prepending (Override Style Syntax)</ulink>" - section in the BitBake User's Manual. - </para> - </section> - - <section id='openembedded-build-system-workflow'> - <title>The OpenEmbedded Build System Workflow</title> - - <para> - The - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> - uses a "workflow" to accomplish image and SDK generation. - The following figure overviews that workflow: - <imagedata fileref="figures/YP-flow-diagram.png" - format="PNG" align='center' width="8in"/> - Following is a brief summary of the "workflow": - <orderedlist> - <listitem><para> - Developers specify architecture, policies, patches and - configuration details. - </para></listitem> - <listitem><para> - The build system fetches and downloads the source code - from the specified location. - The build system supports standard methods such as tarballs - or source code repositories systems such as Git. - </para></listitem> - <listitem><para> - Once source code is downloaded, the build system extracts - the sources into a local work area where patches are - applied and common steps for configuring and compiling - the software are run. - </para></listitem> - <listitem><para> - The build system then installs the software into a - temporary staging area where the binary package format you - select (DEB, RPM, or IPK) is used to roll up the software. - </para></listitem> - <listitem><para> - Different QA and sanity checks run throughout entire - build process. - </para></listitem> - <listitem><para> - After the binaries are created, the build system - generates a binary package feed that is used to create - the final root file image. - </para></listitem> - <listitem><para> - The build system generates the file system image and a - customized Extensible SDK (eSDK) for application - development in parallel. - </para></listitem> - </orderedlist> - </para> - - <para> - For a very detailed look at this workflow, see the - "<link linkend='openembedded-build-system-build-concepts'>OpenEmbedded Build System Concepts</link>" - section. - </para> - </section> - - - <section id='some-basic-terms'> - <title>Some Basic Terms</title> - - <para> - It helps to understand some basic fundamental terms when - learning the Yocto Project. - Although a list of terms exists in the - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-terms'>Yocto Project Terms</ulink>" - section of the Yocto Project Reference Manual, this section - provides the definitions of some terms helpful for getting started: - <itemizedlist> - <listitem><para> - <emphasis>Configuration Files:</emphasis> - Files that hold global definitions of variables, - user-defined variables, and hardware configuration - information. - These files tell the - <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>Open-Embedded build system</ulink> - what to build and what to put into the image to support a - particular platform. - </para></listitem> - <listitem><para> - <emphasis>Extensible Software Development Kit (eSDK):</emphasis> - A custom SDK for application developers. - This eSDK allows developers to incorporate their library - and programming changes back into the image to make - their code available to other application developers. - For information on the eSDK, see the - <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> - manual. - </para></listitem> - <listitem><para> - <emphasis>Layer:</emphasis> - A collection of related recipes. - Layers allow you to consolidate related metadata to - customize your build. - Layers also isolate information used when building - for multiple architectures. - Layers are hierarchical in their ability to override - previous specifications. - You can include any number of available layers from the - Yocto Project and customize the build by adding your - layers after them. - You can search the Layer Index for layers used within - Yocto Project.</para> - - <para>For more detailed information on layers, see the - "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>" - section in the Yocto Project Development Tasks Manual. - For a discussion specifically on BSP Layers, see the - "<ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP Layers</ulink>" - section in the Yocto Project Board Support Packages (BSP) - Developer's Guide. - </para></listitem> - <listitem><para> - <emphasis>Metadata:</emphasis> - A key element of the Yocto Project is the Metadata that - is used to construct a Linux distribution and is contained - in the files that the OpenEmbedded build system parses - when building an image. - In general, Metadata includes recipes, configuration - files, and other information that refers to the build - instructions themselves, as well as the data used to - control what things get built and the effects of the - build. - Metadata also includes commands and data used to - indicate what versions of software are used, from - where they are obtained, and changes or additions to the - software itself (patches or auxiliary files) that - are used to fix bugs or customize the software for use - in a particular situation. - OpenEmbedded-Core is an important set of validated - metadata. - </para></listitem> - <listitem><para id='gs-term-openembedded-build-system'> - <emphasis>OpenEmbedded Build System:</emphasis> - The terms "BitBake" and "build system" are sometimes - used for the OpenEmbedded Build System.</para> - - <para>BitBake is a task scheduler and execution engine - that parses instructions (i.e. recipes) and configuration - data. - After a parsing phase, BitBake creates a dependency tree - to order the compilation, schedules the compilation of - the included code, and finally executes the building - of the specified custom Linux image (distribution). - BitBake is similar to the <filename>make</filename> - tool.</para> - - <para>During a build process, the build system tracks - dependencies and performs a native or cross-compilation - of the package. - As a first step in a cross-build setup, the framework - attempts to create a cross-compiler toolchain - (i.e. Extensible SDK) suited for the target platform. - </para></listitem> - <listitem><para> - <emphasis>OpenEmbedded-Core (OE-Core):</emphasis> - OE-Core is metadata comprised of foundation recipes, - classes, and associated files that are meant to be - common among many different OpenEmbedded-derived systems, - including the Yocto Project. - OE-Core is a curated subset of an original repository - developed by the OpenEmbedded community that has been - pared down into a smaller, core set of continuously - validated recipes. - The result is a tightly controlled and quality-assured - core set of recipes.</para> - - <para>You can see the Metadata in the - <filename>meta</filename> directory of the Yocto Project - <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi'>Source Repositories</ulink>. - </para></listitem> - <listitem><para> - <emphasis>Packages:</emphasis> - In the context of the Yocto Project, this term refers to a - recipe's packaged output produced by BitBake (i.e. a - "baked recipe"). - A package is generally the compiled binaries produced from the - recipe's sources. - You "bake" something by running it through BitBake.</para> - - <para>It is worth noting that the term "package" can, - in general, have subtle meanings. - For example, the packages referred to in the - "<ulink url='&YOCTO_DOCS_REF_URL;#required-packages-for-the-build-host'>Required Packages for the Build Host</ulink>" - section in the Yocto Project Reference Manual are compiled - binaries that, when installed, add functionality to your - Linux distribution.</para> - - <para>Another point worth noting is that historically within - the Yocto Project, recipes were referred to as packages - thus, - the existence of several BitBake variables that are seemingly - mis-named, - (e.g. <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>, - and - <ulink url='&YOCTO_DOCS_REF_URL;#var-PE'><filename>PE</filename></ulink>). - </para></listitem> - <listitem><para> - <emphasis>Poky:</emphasis> - Poky is a reference embedded distribution and a reference - test configuration. - Poky provides the following: - <itemizedlist> - <listitem><para> - A base-level functional distro used to illustrate - how to customize a distribution. - </para></listitem> - <listitem><para> - A means by which to test the Yocto Project - components (i.e. Poky is used to validate - the Yocto Project). - </para></listitem> - <listitem><para> - A vehicle through which you can download - the Yocto Project. - </para></listitem> - </itemizedlist> - Poky is not a product level distro. - Rather, it is a good starting point for customization. - <note> - Poky is an integration layer on top of OE-Core. - </note> - </para></listitem> - <listitem><para> - <emphasis>Recipe:</emphasis> - The most common form of metadata. - A recipe contains a list of settings and tasks - (i.e. instructions) for building packages that are then - used to build the binary image. - A recipe describes where you get source code and which - patches to apply. - Recipes describe dependencies for libraries or for other - recipes as well as configuration and compilation options. - Related recipes are consolidated into a layer. - </para></listitem> - </itemizedlist> - </para> - </section> -</chapter> -<!-- -vim: expandtab tw=80 ts=4 ---> diff --git a/poky/documentation/overview-manual/overview-manual.rst b/poky/documentation/overview-manual/overview-manual.rst new file mode 100644 index 0000000000..f20b20e328 --- /dev/null +++ b/poky/documentation/overview-manual/overview-manual.rst @@ -0,0 +1,19 @@ +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK + +========================================== +Yocto Project Overview and Concepts Manual +========================================== + +| + +.. toctree:: + :caption: Table of Contents + :numbered: + + overview-manual-intro + overview-manual-yp-intro + overview-manual-development-environment + overview-manual-concepts + history + +.. include:: /boilerplate.rst diff --git a/poky/documentation/overview-manual/overview-manual.xml b/poky/documentation/overview-manual/overview-manual.xml deleted file mode 100755 index 7c75e5086c..0000000000 --- a/poky/documentation/overview-manual/overview-manual.xml +++ /dev/null @@ -1,149 +0,0 @@ -<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" -"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" -[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > - -<book id='overview-manual' lang='en' - xmlns:xi="http://www.w3.org/2003/XInclude" - xmlns="http://docbook.org/ns/docbook" - > - <bookinfo> - - <mediaobject> - <imageobject> - <imagedata fileref='figures/overview-manual-title.png' - format='SVG' - align='left' scalefit='1' width='100%'/> - </imageobject> - </mediaobject> - - <title> - Yocto Project Overview and Concepts Manual - </title> - - <authorgroup> - <author> - <affiliation> - <orgname>&ORGNAME;</orgname> - </affiliation> - <email>&ORGEMAIL;</email> - </author> - </authorgroup> - - <revhistory> - <revision> - <revnumber>2.5</revnumber> - <date>May 2018</date> - <revremark>The initial document released with the Yocto Project 2.5 Release.</revremark> - </revision> - <revision> - <revnumber>2.6</revnumber> - <date>November 2018</date> - <revremark>Released with the Yocto Project 2.6 Release.</revremark> - </revision> - <revision> - <revnumber>2.7</revnumber> - <date>May 2019</date> - <revremark>Released with the Yocto Project 2.7 Release.</revremark> - </revision> - <revision> - <revnumber>3.0</revnumber> - <date>October 2019</date> - <revremark>Released with the Yocto Project 3.0 Release.</revremark> - </revision> - <revision> - <revnumber>3.1</revnumber> - <date>April 2020</date> - <revremark>Released with the Yocto Project 3.1 Release.</revremark> - </revision> - <revision> - <revnumber>3.1.1</revnumber> - <date>June 2020</date> - <revremark>Released with the Yocto Project 3.1.1 Release.</revremark> - </revision> - <revision> - <revnumber>3.1.2</revnumber> - <date>August 2020</date> - <revremark>Released with the Yocto Project 3.1.2 Release.</revremark> - </revision> - <revision> - <revnumber>3.1.3</revnumber> - <date>October 2020</date> - <revremark>Released with the Yocto Project 3.1.3 Release.</revremark> - </revision> - <revision> - <revnumber>3.1.4</revnumber> - <date>&REL_MONTH_YEAR;</date> - <revremark>Released with the Yocto Project 3.1.4 Release.</revremark> - </revision> - </revhistory> - - <copyright> - <year>©RIGHT_YEAR;</year> - <holder>Linux Foundation</holder> - </copyright> - - <legalnotice> - <para> - Permission is granted to copy, distribute and/or modify this document under - the terms of the <ulink type="http" url="http://creativecommons.org/licenses/by-sa/2.0/uk/"> - Creative Commons Attribution-Share Alike 2.0 UK: England & Wales</ulink> as published by - Creative Commons. - </para> - <note><title>Manual Notes</title> - <itemizedlist> - <listitem><para> - This version of the - <emphasis>Yocto Project Overview and Concepts Manual</emphasis> - is for the &YOCTO_DOC_VERSION; release of the - Yocto Project. - To be sure you have the latest version of the manual - for this release, go to the - <ulink url='&YOCTO_DOCS_URL;'>Yocto Project documentation page</ulink> - and select the manual from that site. - Manuals from the site are more up-to-date than manuals - derived from the Yocto Project released TAR files. - </para></listitem> - <listitem><para> - If you located this manual through a web search, the - version of the manual might not be the one you want - (e.g. the search might have returned a manual much - older than the Yocto Project version with which you - are working). - You can see all Yocto Project major releases by - visiting the - <ulink url='&YOCTO_WIKI_URL;/wiki/Releases'>Releases</ulink> - page. - If you need a version of this manual for a different - Yocto Project release, visit the - <ulink url='&YOCTO_DOCS_URL;'>Yocto Project documentation page</ulink> - and select the manual set by using the - "ACTIVE RELEASES DOCUMENTATION" or "DOCUMENTS ARCHIVE" - pull-down menus. - </para></listitem> - <listitem> - <para> - To report any inaccuracies or problems with this - (or any other Yocto Project) manual, send an email to - the Yocto Project documentation mailing list at - <filename>docs@lists.yoctoproject.org</filename> or - log into the freenode <filename>#yocto</filename> channel. - </para> - </listitem> - </itemizedlist> - </note> - </legalnotice> - - </bookinfo> - - <xi:include href="overview-manual-intro.xml"/> - - <xi:include href="overview-manual-yp-intro.xml"/> - - <xi:include href="overview-manual-development-environment.xml"/> - - <xi:include href="overview-manual-concepts.xml" /> - -</book> -<!-- -vim: expandtab tw=80 ts=4 ---> |