Moulin User reference manual

Invoking moulin

moulin has one mandatory parameter - the file name of the build description. It should be in YAML format. You may use a regular local file or URL. moulin detects URL by the presence of a protocol prefix, like https://. If you use the URL to GitHub or another network repository, you can use the URL for the raw file only, not for a web page with that file. For example, this URL points to the correct YAML file: https://raw.githubusercontent.com/xen-troops/meta-xt-prod-devel-rcar/master/prod-devel-rcar.yaml. But the following URL can’t be used, because it points to GitHub’s web page: https://github.com/xen-troops/meta-xt-prod-devel-rcar/blob/master/prod-devel-rcar.yaml. Pay attention, that file will be downloaded only if a file with the same name doesn’t exist in the current folder. This is done to preserve possible local changes made by a user.

As a result, moulin will generate ninja.build file. You can then invoke ninja to perform the actual build. moulin adds generator rule in ninja.build, so it is not mandatory to invoke moulin after you make changes to your YAML file. Ninja will detect any changes to this file and invoke moulin automatically to re-create ninja.build.

If the YAML file contains parameters key, it is possible to invoke moulin with additional command line options. The set of these options depends on the contents of the YAML file and can be viewed using --help-config command line option.

Verbose output

Use -v or --verbose command line option to increase the verbosity of the output. With this option enabled, moulin will give more information about what it is doing.

Dumping intermediate state

--dump command line option can be used to force moulin to dump the intermediate state of the processed YAML file. You will see the contents of your build config after applying all parameters and expanding all variables. This can come in handy during the debugging of your build, because you can see what exactly is passed to fetchers and builders.

Internal command line options

There is --fetcherdep command line option which is internal, and it is even hidden from -h output. It is used by moulin to generate dynamic dependency files for Ninja, so Ninja can track changes inside components.

This option is not meant to be used by a user.

YAML sections

The YAML file should consist of several predefined keys or sections, which are discussed below. Any unknown keys are ignored. Right now, only the following top-level keys are supported:

  • desc - mandatory

  • min_ver - optional

  • components - mandatory

  • images - optional. See rouge documentation.

  • variables - optional

  • parameters - optional

Minimal Version

Optional min_ver section should hold the minimal required version of moulin. This is a text string that conforms to PEP-440. For example min_ver: "0.2". moulin will compare this with its own version and will stop if the required version is newer.

Mandatory sections: “desc” and “components”

There are only two mandatory sections: desc and components. desc should contain a text string that describes the build. It is displayed when moulin is invoked with --help-config command line option.

components should contain a dictionary where the key is a component name and value is another dictionary with component settings:

components:
    component1:
        build-dir: "component-build-dir" # Optional
        default: true # Optional
        sources:
            ......
        builder:
            .....
    component2:
        sources:
            ......
        builder:
            .....

There are two main parts of each component description: sources and builder.

sources is optional and can contain a list of source code definitions, which will be fetched before starting a build:

sources:
 - type: git
   url: "git://git.yoctoproject.org/poky"
   rev: gatesgarth
 - type: repo
   url: https://github.com/xen-troops/android_manifest.git
   rev: android-11-master
   manifest: doma.xml

All supported fetchers are listed in the Fetchers section.

builder contains build configuration for a given component. There are multiple builder types supported. They are described in the Builders section.

Apart from two mandatory options, component description can contain the following optional keys:

  • build_dir - build directory name. By default, the component’s name is used.

  • default - if set to true - tells Ninja that this component is a default build target. This can be omitted and Ninja will choose the build target on its own rules.

Variables

variables section is optional. It can contain a dictionary of variable name-value pairs:

variables:
  A: "a"
  B: "1%{A}%{A}" # will be expanded to "1aa"
  C: "2%{B}%{B}" # will be expanded to "21aa1aa"

Variables can be used anywhere in the YAML file. During internal pre-processing all variable references in the form of %{variable_name} will be replaced with the actual variable value.

% is a special symbol. It can be escaped by doubling it: %%.

Variables should be used to decrease the amount of hard-coded values. Good candidates that should be moved to variables are path names, branches, hardware identifiers, etc.

Parameters

Often, it is desired to have some options for a build. For example, one can want to support several different HW boards or to enable additional features. It would not be feasible to have separate YAML for every board-feature combination. This is where parameters come to help. All parameters should be stored in parameters section:

parameters:
  parameter1:
    desc: "parameter 1 description"
    option1:
      default: true
      overrides:
        ...
    option2:
      overrides:
        ...
    option3:
      overrides:
        ...
  parameter2:
    desc: "parameter 2 description"
    option1:
      overrides:
        ...
    option2:
      overrides:
        ...
    option3:
      default: true
      overrides:
        ...

Every parameter should include mandatory desc key. Parameter can have one or more options, one of which should have default flag enabled.

Central part of each option is the overrides section. Contents of this section should correspond to the top-level layout of the YAML file. All contents of this section will be overlaid on the contents of the YAML file during the pre-processing stage. Rules of this process are:

  • Dictionaries are extended with new keys from overrides section.

  • If the dictionary already has the key:

    • If type of original value differs from type of overrides section value, error is generated.

    • If key’s value is a scalar (number, boolean, string) that it is replaced with value from overrides section.

    • If key’s value is an another dictionary, process start recursively.

    • If key’s value is a list, it is expanded with values from overrides section.

  • Order of parameter application is not specified.

Basically, these rules follow the intuitive idea of extending/overwriting original config: primitive values will be overwritten, all other values will be extended.

User can choose parameter options using command line arguments, as described in the Invoking moulin section.

Fetchers

Fetchers are the moulin plugins responsible for downloading sources listed in sources section of a component.

moulin will generate phony Ninja target fetch-{component_name} for every component. It can be used to just fetch sources without building anything.

git fetcher

git fetcher is used to download code from a remote or local git repository. There is a complete list of supported parameters:

type: git # Selects `git` fetcher
url: "url://for.repository/project.git"
rev: revision_name
dir: "directory/where/store/code"
depth: 1
submodules: true

  • type - mandatory - should be git to enable git fetcher.

  • url - mandatory - repository URL. You can provide any URL that is supported by git itself.

  • rev - optional - revision that should be checked out after cloning. Can be any git tree-ish like branch name, tag, or commit ID. If this option is omitted, git will checkout the default branch.

  • dir - optional - directory name which should be used for cloning. If this option is missed, moulin will try to guess the directory name from url. This path is relative to the component’s build directory.

  • submodules - optional - boolean. Fetch submodules along with main repository.

  • depth - optional - cloning depth. Corresponds to --depth option for git clone. If used together with submodules enabled, it will call git with --shallow-submodules

repo fetcher

repo fetcher is used to download code using Google’s repo tool. Full list of supported options:

type: repo # Selects `repo` fetcher
url: https://manifest.address/repo.git
rev: manifest-revision
manifest: manifest-file.xml
depth: 1
groups: all
dir: "."

  • type - mandatory - should be repo to enable repo fetcher.

  • url - mandatory - manifest repository URL. You can provide any URL that is supported by repo itself. This corresponds to repo’s -u option.

  • rev - optional - manifest revision. Corresponds to repo’s -b option.

  • manifest - optional - manifest file name. Corresponds to repo’s -m option.

  • depth - optional - cloning depth of internal repositories. Corresponds to repo’s --depth option. Setting it to 1 will sufficiently decrease the fetching time.

  • groups - optional - name of manifest groups that should be synced. Corresponds to repo’s -g option. You can use it to choose which project groups need to be synced.

  • dir - optional - directory name which should be used for code storage. If it is missing, moulin will use "." to initialize repo repository right in the component’s build directory, as this is a main repo use case.

http fetcher

http fetcher is used to download a file via HTTP or HTTPS protocol. It uses curl tool to do so. Complete list of supported options:

type: http # Selects `http` fetcher
url: "https://example.com/file.txt"
filename: "file.txt"
dir: "."

  • type - mandatory - should be http to use http fetcher. Use the same type even if you are downloading over the HTTPS protocol.

  • url - mandatory - URL of a file to be downloaded

  • filename - optional (in most cases) - name of the output file. If omitted, moulin will try to guess it from a URL. But if you can’t do so, it will ask you to provide the filename manually.

  • dir - optional - directory name where to store a downloaded file. If it is omitted, moulin will use "." to download a file right into the component’s root directory.

unpack fetcher

unpack fetcher is used to unpack already available archives to a specified directory. An example use case is when we need to use a 3rd-party code/resources that are not available in the git repository. Complete list of supported options:

type: unpack # Selects `unpack` fetcher
archive_type: tar
file: my_file.tar.gz
dir: "."

  • type - mandatory - should be unpack to enable unpack fetcher.

  • archive_type - mandatory - type or archive. Now tar and zip are supported.

  • file - mandatory - name of the archive file

  • dir - optional - directory name which should be used for code storage. If it is missing, moulin will use "." to unpack the archive right into the component directory.

Currently, unpack fetcher supports two archive types: tar and zip.

  • tar actually supports not only plain .tar archives, but also compressed archives like .tar.gz, .tar.bz2, and so on. We rely on tar ability to select the right decompressor automatically.

  • zip - this is classic zip format. unzip tool is used to decompress this kind of archive, so it should be present on the user’s machine.

west fetcher

west fetcher is used to download code using Zephyr’s west meta-tool. Complete list of supported options:

type: west # Selects `west` fetcher
url: https://manifest.address/zephyr
rev: manifest-revision
file: manifest-file.yml

  • type - mandatory - should be west to enable west fetcher.

  • url - optional - manifest repository URL. You can provide any URL that is supported by west itself. This corresponds to west init’s -m option.

  • rev - optional - manifest revision. Corresponds to west init’s --mr option.

  • file - optional - manifest file name. Corresponds to west init’s --mf option.

For additional details, see documentation on west init: https://docs.zephyrproject.org/latest/develop/west/built-in.html#west-init

Regarding installation of west, please see: https://docs.zephyrproject.org/latest/develop/west/install.html

null fetcher

null fetcher does nothing. It can be used for testing or in some tricky situation when you want to have a component without fetchers.

type: "null" # Selects `none` fetcher

  • type - mandatory - should be "null" to use null fetcher. Please note that you need to use quotes; otherwise, the YAML parser will treat it as a null type.

Builders

Builders are the moulin plugins responsible for actual image building.

moulin will generate phony Ninja target {component_name} for every component. It can be used to build a certain component. Please note that this will not build only given component. Any prerequisites will be fetched and built as well.

Builder configuration heavily depends on the builder type and is described in the next subsections.

yocto builder

Yocto builder is used to build OpenEmbedded-based images. It expects that poky repository is cloned in {build_dir}/poky and uses it’s poky/oe-init-build-env script to initialize build environment. Then bitbake-layers tool is used to add additional layers and bitbake used to perform the build.

builder:
  type: yocto       # Should be `yocto`
  base_distro: poky # Optional
  work_dir: "build" # Optional
  build_target: core-image-minimal # Mandatory
  conf:             # Mandatory
    - [MACHINE, "machine-name"]
    - [DISTRO_FEATURES_remove, "feature_to_remove"]
    - [DISTRO_FEATURES_append, "feature_to_add"]
  layers:           # Optional
    - "../poky/meta-yocto-bsp"
    - "../meta-other-layer/"
  external_src:     # Optional
    "package-name": "path-to-package-sources"
    "another-package-name": ["path part1", "path part2", "path part3"]
  target_images:    # Mandatory
    - "tmp/deploy/images/machine-name/Image"
  additional_deps:  # Optional
    - "path/to/file/generated/by/other/component"

Mandatory options:

  • type - Builder type. Should be yocto for this type of builder.

  • build_target - bitbake’s build target. This will be used to run the build: $ bitbake {build_target}

  • target_images - list of image files that should be generated by this component as a result of invoking $ bitbake {build_target}. Every component should generate at least one image file.

Optional parameters. Those provide advanced features that may be needed if you are building multiple VMs with cross-dependencies.

  • base_distro - provides name of base distro directory. Default value is poky for compatibility reasons. For newer Yocto releases you want to use openembedded-core. Currently this only affects location of oe-init-build-env script.

  • conf - list of additional local.conf options. Please note that each entry in conf list is not a key:value pair, but another list of two items. We use this format because it is possible to have multiple local.conf entries with the same key. Those entries will not be written straight into local.conf. Instead, a new file moulin.conf will be created. This file will then be included from local.conf.

  • layers - list of additional layers. Those layers will be added to the build using bitbake-layers add-layer {layers} command.

  • work_dir - bitbake’s work directory. The default value is “build”. This is where files like “conf/local.conf” are stored. You can overwrite so you can produce multiple builds from the same (or different) set of Yocto layers.

  • additional_deps - list of additional dependencies. This is basically target_images produced by other components. You can use those to implement build dependencies between components. For example, if your system needs to have DomU’s kernel image on Dom0’s file system, you might want to add the path to DomU’s kernel into additional_deps of Dom0’s config. This will ensure that Dom0 will be built after DomU.

  • external_src - list of external sources for packages. This option will make moulin generate EXTERNALSRC:pn-{package} in local.conf. This feature is used to provide the Yocto build with artifacts that were built outside of the tree. Such artifacts can be provided by another component, for example.

bazel builder

Bazel builder is used to build projects based on the Bazel build system provided by Google. It expects that a project with source and Bazel configuration files is present in the build directory.

builder:
  type: "bazel"         # Mandatory and must be `bazel`
  tool: "tools/bazel"   # Optional
  startup-options:      # Optional
    - "--max_idle_secs=1"
  command: run          # Optional
  args:                 # Optional
    - "--verbose_failures"
    - "--sandbox_debug"
  target:               # Mandatory
  target-patterns:      # Optional
    - "--dist_dir=path_to_dist"
  target_images:        # Mandatory
    - "out/deploy/virtual-device/virtual_device_aarch64/Image"
    - "out/deploy/virtual-device/virtual_device_aarch64/initramfs.img"

Mandatory parameters:

  • type - Builder type. It must be bazel for this type of builder.

  • target - target name that should be described in the corresponding BUILD.bazel file and must satisfy bazel rules.

  • target_images - list of artifact files that should be generated by this component as a result of the build. Every component should generate at least one image file.

Optional parameters:

  • tool - the relative path to the Bazel tool in relation to the ‘build-dir’. If this parameter is not defined, the system-installed Bazel tool will be used.

  • startup-options - Bazel startup options that appear before the command and are parsed by the client.

  • command - bazel command. By default, build is used.

  • args - bazel arguments related to the concrete command.

  • target-patterns - bazel target patterns to be built or parameters to the executable target.

android builder

Android builder is used to build the Android Open Source Project (AOSP). It expects that AOSP is present in the build directory. In most cases, AOSP is cloned using the repo fetcher.

builder:
  type: android # Should be 'android'
  env:          # Optional
    - "TARGET_BOARD_PLATFORM=r8a7795"
  lunch_target: xenvm-userdebug
  target_images:
    - "out/xenvm/userdebug/boot.img"
    - "out/xenvm/userdebug/system.img"
  additional_deps:  # Optional
    - "path/to/file/generated/by/other/component"

Mandatory options:

  • type - Builder type. Should be android for this type of builder.

  • lunch_target - lunch’s build target. This will be used to run the build: $ lunch {lunch-target}

  • target_images - list of image files that should be generated by this component as a result of invoking $ m. Every component should generate at least one image file.

Optional parameters:

  • env - list of additional environment variables that should be exported before calling lunch.

android_kernel builder

Android Kernel builder is used to build the kernel and kernel modules for Android Open Source Project (AOSP). It expects that the correct directory layout is present in the build directory. In most cases, AOSP is cloned using the repo fetcher.

builder:
  type: android_kernel # Should be 'android_kernel'
  env:                 # Optional
    - "TARGET_BOARD_PLATFORM=r8a7795"
    - "BUILD_CONFIG=common/build.config.xenvm"
    - "SKIP_MRPROPER=1"
  target_images:
    - "out/android12-5.4/common/arch/arm64/boot/Image"

Mandatory options:

  • type - Builder type. Should be android_kernel for this type of builder.

  • target_images - list of image files that this component should generate as a result of invoking build.sh script. Every component should generate at least one image file.

Optional parameters:

  • env - list of additional environment variables that should be exported before calling build.sh.

  • additional_deps - list of additional dependencies. This is basically target_images produced by other components. You can use those to implement build dependencies between components. For example, if your Android build needs a Linux kernel built by some other component, you might want to add path to linux kernel image provided by this component into additional_deps. This will ensure that Linux kernel will be built before Android.

archive builder

Archive builder is intended to create an archive from other components. It can be used to gather build artifacts, for example. This builder uses tar to create archive files. Archives can be optionally compressed as, tar is invoked with –auto-compress option.

builder:
  type: archive        # Should be 'archive'
  name: "artifacts.tar.bz2"
  base_dir: "yocto/build/tmp/deploy/images/"
  items:
    # items are relative to base_dir
    - "generic-armv8-xt/Image"
    - "generic-armv8-xt/uInitramfs"

Mandatory options:

  • type - Builder type. It should be archive for this type of builder.

  • name - Name of an archive file. Add a suffix like tar.bz2 to make tar compress archive with desired compressing algorithm.

  • base_dir - Optional parameter specifying tar’s base directory. The default value is “.” if not specified. This is passed to tar as -C option. As result, the final archive will contain paths relative to base_dir. Avoid using .. because specified items will be archived by tar but all .. will be stripped. As a result, the archive will contain items with unexpected paths.

  • items - list of files or directories that should be added to the archive. Please ensure that those files or directories are present in other components target_images sections, so Ninja can build correct dependencies. All paths are relative to the base_dir.

zephyr builder

This builder is used to build applications based on Zephyr OS. It uses Zephyr OS meta-tool west. Required code is expected to be fetched by west fetcher.

builder:
  type: zephyr
  board: xenvm
  shields:          # Optional
    - "shield1"
    - "shield2"
  target: samples/synchronization
  work_dir: build_dir
  target_images:
    - "zephyr/build/zephyr/zephyr.bin"
  vars:
    - "VAR1=var1_value"
  env:
    - "MY_ENV_VAR=my_value"
  additional_deps:  # Optional
    - "path/to/file/generated/by/other/component"

Mandatory options:

  • type - builder type. Should be zephyr for this type of builder.

  • board - target board name. For example: xenvm or xenvm_gicv3 for Xen-based builds. Corresponds to west build’s -b option. See Zephyr’s documentation for the list of allowed values.

  • target - build target. This will be used to run the build: $ west build {target}. For example: samples/synchronization or samples/hello_world.

  • target_images - list of image files that this builder should generate. For the standard build, it is expected to be “zephyr/build/zephyr/zephyr.bin”

Optional parameters:

  • env - list of additional environment variables that should be exported before calling west build.

  • work_dir - build system’s work directory. Default value is “build”. This is where files produced by the build system are stored.

  • additional_deps - list of additional dependencies. This is basically target_images produced by other components. You can use those to implement build dependencies between components. For example, if your system needs to have DomU’s kernel image in your zephyr image, you might want to add the path to DomU’s kernel into additional_deps of zephyr’s config. This will ensure that zephyr will be built after DomU.

  • shields - list of shields should be integrated to zephyr board(For Zephyr < 3.4.0).

  • snippets - list of snippets should be integrated to zephyr board(For Zephyr >= 3.4.0). Please note that only one of shields and snippets can be used at the same time.

  • vars - list of additional variables that should be passed to CMake via west build.

Please note that this builder uses --pristine=auto command-line option.

Proper versions of CMake and the Zephyr SDK must be installed on the host.

For additional details, please see https://docs.zephyrproject.org/latest/develop/west/build-flash-debug.html#building-west-build

custom_script builder

The custom-script builder is designed to perform custom actions. This builder actually calls the script pointed in option script. Builder node is stored in the file in work_dir directory, and that file is passed to the script as parameter

builder:
  type: custom_script        # Should be 'custom_script'
  work_dir : "script_workdir"
  script: "path/to/script/custom_script.py"
  args:
    - "argument1"
    - "argument2"
  config:
    items:
      "rootfs": "images/spider/rootfs.tar.bz2"
    manifest:
      "id": ""
      "vendorVersion": "0.2.0"
      "fileName": "archive.squashfs"
      "description": "DomD image"
      "bundleType": "full"
  target_images:
    - "custom_script_targets"
  additional_deps:  # Optional
    - "path/to/file_name"

Mandatory options:

  • type - Builder type. It should be custom_script for this type of builder.

  • work_dir - build script work directory. Default value is “script_workdir”. This is where files produced by the build system are stored.

  • script - path to script which performs custom actions. Whole yaml node will be stored to a file, and the name of that file will be passed to script as the last command line argument.

  • target_images - list of files that should be generated by script.

Optional parameters:

  • args - additional arguments should be passed to script. Can be passed as a string or a list

  • additional_deps - list of additional dependencies. This is basically target_images produced by other components. You can use those to implement build dependencies between components. For example, if your system needs to have DomU’s kernel image for your fota archive, you might want to add the path to DomU’s kernel into additional_deps of Dom0’s config. This will ensure that the FOTA archive will be built after DomU.

  • Remaining parameters should be parsed and used by the script pointed in script option.

null builder

“null” builder does nothing at all. It does not even generate dependencies. It can be used for testing or in cases when you need to call the fetcher only. Please note that Ninja will not call fetcher for the component if fetcher’s output file is not used by anything.

builder:
  type: "null"        # Should be "null"

Mandatory options:

  • type - Builder type. It should be "null" for this type of builder. Please note that you need to use quotes; otherwise, the YAML parser will treat it as a null type.