6 Building software for embedded devices is a complex process. Projects often have
7 custom toolchains, target different hardware platforms, and require additional
8 configuration and post-processing of artifacts.
10 As a modern embedded framework, Pigweed's goal is to collect these embedded use
11 cases into a powerful and flexible build system, then extend it with support for
12 modern software development practices.
14 What's in a build system?
15 =========================
16 A quality build system provides a variety of features beyond compiling code.
17 Throughout our experience with embedded development, we've found several build
18 features to be especially useful, and designed Pigweed's build system with them
21 Simple toolchain configuration
22 ------------------------------
23 Embedded projects often use custom build toolchains for their specific hardware.
24 Configuring these should be a simple process, both in their initial setup and
29 Virtually every consumer product has firmware that targets different boards or
30 MCUs during development. While building for a single board is simple enough, the
31 complexity of supporting different targets ranges from changing compiler flags
32 to swapping out entire libraries of firmware and drivers. This is often done by
33 running multiple builds, configuring each one accordingly. In Pigweed, we've
34 designed our build system with first-class multi-target support in mind,
35 allowing any number of target configurations to be built simultaneously.
37 Multi-language support
38 ----------------------
39 Embedded projects are typically written in C, C++, and assembly. However, it is
40 possible to have firmware written in other languages, such as Rust.
41 Additionally, projects may have host-side tooling written in a wide variety of
42 languages. Having all of these build together proves to be a large time saver.
46 Embedded projects often require post-processing of build artifacts; these may
49 * Extracting ELF sections into a different container
50 * Injecting metadata into firmware images
52 * Creating databases of symbols for debugging
53 * Extracting string tokens into a database (for example, with
54 :ref:`module-pw_tokenizer`)
56 These are run as steps during a build, facilitated by the build system.
61 * :ref:`module-pw_build-python-action`
65 Python is a favorite scripting language of many development teams, and here at
66 Pigweed, we're no exception. Much of Pigweed's host-side tooling is written in
67 Python. While Python works great for local development, problems can arise when
68 scripts need to be packaged and distributed for vendors or factory teams. Having
69 proper support for packaging Python within a build system allows teams to focus
70 on writing code instead of worrying about distribution.
74 On embedded devices, memory is everything. Most projects have some sort of
75 custom tooling to determine how much flash and RAM space their firmware uses.
76 Being able to run size reports as part of a build ensures that they are always
77 up-to-date and allows space usage to be tracked over time.
82 * :ref:`module-pw_bloat`
86 An oft-neglected part of software development, documentation is invaluable for
87 future maintainers of a project. As such, Pigweed has integrated documentation
88 which builds alongside its code and combines with other build features, such as
89 size reports, to provide high quality, up-to-date references for developers.
94 * :ref:`module-pw_docgen`
98 Unit tests are essential to ensure that the functionality of code remains
99 consistent as changes are made to avoid accidental regressions. Running unit
100 tests as part of a build keeps developers constantly aware of the impact of
105 Though Pigweed targets embedded devices, a lot of its code can be run and tested
106 on a host desktop by swapping out backends to host platform libraries. This is
107 highly beneficial during development, as it allows tests to consistently run
108 without having to go through the process of flashing a device.
110 Device-side unit tests
111 ^^^^^^^^^^^^^^^^^^^^^^
112 As useful as host-side tests are, they are not sufficient for developing actual
113 firmware, and it is critical to run tests on the actual hardware. Pigweed has
114 invested into creating a test framework and build integration for running tests
115 across physical devices as part of a build.
120 * :ref:`module-pw_unit_test`
121 * :ref:`module-pw_target_runner`
125 In web development, it is common to have a file system watcher listening for
126 source file changes and triggering a build for quick iteration. When combined
127 with a fast incremental build system, this becomes a powerful feature, allowing
128 things such as unit tests and size reports to re-run whenever any dependent
131 While initially seen as somewhat of a gimmick, Pigweed's watcher has become a
132 staple of Pigweed development, with most Pigweed users having it permanently
133 running in a terminal window.
138 * :ref:`module-pw_watch`
140 Pigweed's build systems
141 =======================
142 Pigweed can be used either as a monolith or à la carte, slotting into an
143 existing project. To this end, Pigweed supports multiple build systems, allowing
144 Pigweed-based projects to choose the most suitable one for them.
146 Of the supported build systems, GN is the most full-featured, followed by CMake,
151 A well-known name in C/C++ development, `CMake`_ is widely used by all kinds of
152 projects, including embedded devices. Pigweed's CMake support is provided
153 primarily for projects that have an existing CMake build and wish to integrate
158 The open source version of Google's internal build system. `Bazel`_ has been
159 growing in popularity within the open source world, as well as being adopted by
160 various proprietary projects. Its modular structure makes it a great fit for
165 A perhaps unfamiliar name, `GN (Generate Ninja)`_ is a meta-build system that
166 outputs `Ninja`_ build files, originally designed for use in Chromium. Pigweed
167 first experimented with GN after hearing about it from another team, and we
168 quickly came to appreciate its speed and simplicity. GN has become Pigweed's
169 primary build system; it is used for all upstream development and strongly
170 recommended for Pigweed-based projects where possible.
172 .. _CMake: https://cmake.org/
173 .. _Bazel: https://bazel.build/
174 .. _GN (Generate Ninja): https://gn.googlesource.com/gn
175 .. _Ninja: https://ninja-build.org/
179 This section describes Pigweed's GN build structure, how it is used upstream,
180 build conventions, and recommendations for Pigweed-based projects. While
181 containing some details about how GN works in general, this section is not
182 intended to be a guide on how to use GN. To learn more about the tool itself,
183 refer to the official `GN reference`_.
185 .. _GN reference: https://gn.googlesource.com/gn/+/master/docs/reference.md
188 A quick note on terminology: the word "target" is overloaded within GN (and
189 Pigweed)---it can refer to either a GN build target, such as a ``source_set``
190 or ``executable``, or to an output platform (e.g. a specific board, device, or
193 To avoid confusing the two, we refer to the former as "GN targets" and the
194 latter as "Pigweed targets".
198 The entrypoint to a GN build is the ``.gn`` file, which defines a project's root
199 directory (henceforth ``//``).
201 ``.gn`` must point to the location of a ``BUILDCONFIG.gn`` file for the project.
202 In Pigweed upstream, this is its only purpose.
204 Downstream projects may additionally use ``.gn`` to set global overrides for
205 Pigweed's build arguments, which apply across all Pigweed targets. For example,
206 a project could configure the protobuf libraries that it uses. This is done by
207 defining a ``default_args`` scope containing the overrides.
211 # The location of the BUILDCONFIG file.
212 buildconfig = "//BUILDCONFIG.gn"
214 # Build arguments set across all Pigweed targets.
216 dir_pw_third_party_nanopb = "//third_party/nanopb-0.4.2"
219 Configuration: BUILDCONFIG.gn
220 -----------------------------
221 The file ``BUILDCONFIG.gn`` configures the GN build by defining any desired
222 global variables/options. It can be located anywhere in the build tree, but is
223 conventionally placed at the root. ``.gn`` points GN to this file.
225 ``BUILDCONFIG.gn`` is evaluated before any other GN files, and variables defined
226 within it are placed into GN's global scope, becoming available in every file
227 without requiring imports.
229 The options configured in this file differ from those in ``.gn`` in two ways:
231 1. ``BUILDCONFIG.gn`` is evaluated for every GN toolchain (and Pigweed target),
232 whereas ``.gn`` is only evaluated once. This allows ``BUILDCONFIG.gn`` to set
233 different options for each Pigweed target.
234 2. In ``.gn``, only GN build arguments can be overridden. ``BUILDCONFIG.gn``
235 allows defining arbitrary variables.
237 Generally, it is preferable to expose configuration options through build args
238 instead of globals in ``BUILDCONFIG.gn`` (something Pigweed's build previously
239 did), as they are more flexible, greppable, and easier to manage. However, it
240 may make sense to define project-specific constants in ``BUILDCONFIG.gn``.
242 Pigweed's upstream ``BUILDCONFIG.gn`` does not define any variables; it just
243 sets Pigweed's default toolchain, which GN requires.
247 Top-level GN targets: //BUILD.gn
248 --------------------------------
249 The root ``BUILD.gn`` file defines all of the libraries, images, tests, and
250 binaries built by a Pigweed project. This file is evaluated immediately after
251 ``BUILDCONFIG.gn``, with the active toolchain (which is the default toolchain
252 at the start of a build).
254 ``//BUILD.gn`` is responsible for enumerating each of the Pigweed targets built
255 by a project. This is done by instantiating a version of each of the project's
256 GN target groups with each Pigweed target's toolchain. For example, in upstream,
257 all of Pigweed's GN targets are contained within the ``pigweed_default`` group.
258 This group is instantiated multiple times, with different Pigweed target
261 These groups include the following:
263 * ``host`` -- builds ``pigweed_default`` with Clang or GCC, depending on the
265 * ``host_clang`` -- builds ``pigweed_default`` for the host with Clang
266 * ``host_gcc`` -- builds ``pigweed_default`` for the host with GCC
267 * ``stm32f429i`` -- builds ``pigweed_default`` for STM32F429i Discovery board
268 * ``docs`` -- builds the Pigweed documentation and size reports
270 Pigweed projects are recommended to follow this pattern, creating a top-level
271 group for each of their Pigweed targets that builds a common GN target with the
272 appropriate toolchain.
274 It is important that no dependencies are listed under the default toolchain
275 within ``//BUILD.gn``, as it does not configure any build parameters, and
276 therefore should not evaluate any other GN files. The pattern that Pigweed uses
277 to achieve this is to wrap all dependencies within a condition checking the
282 group("my_application_images") {
283 deps = [] # Empty in the default toolchain.
285 if (current_toolchain != default_toolchain) {
286 # This is only evaluated by Pigweed target toolchains, which configure
287 # all of the required options to build Pigweed code.
288 deps += [ "//images:evt" ]
292 # The images group is instantiated for each of the project's Pigweed targets.
293 group("my_pigweed_target") {
294 deps = [ ":my_application_images(//toolchains:my_pigweed_target)" ]
298 Pigweed's default toolchain is never used, so it is set to a dummy toolchain
299 which doesn't define any tools. ``//BUILD.gn`` contains conditions which check
300 that the current toolchain is not the default before declaring any GN target
301 dependencies to prevent the default toolchain from evaluating any other BUILD
302 files. All GN targets added to the build must be placed under one of these
307 The root ``BUILD.gn`` file can define a special group named ``default``. If
308 present, Ninja will build this group when invoked without arguments.
311 Defining a ``default`` group makes using ``pw watch`` simple!
315 Pigweed's ``//BUILD.gn`` defines the ``pw_default_optimization_level`` build
316 arg, which specifies the optimization level to use for the default groups
317 (``host``, ``stm32f429i``, etc.). The supported values for
318 ``pw_default_optimization_level`` are:
320 * ``debug`` -- create debugging-friendly binaries (``-Og``)
321 * ``size_optimized`` -- optimize for size (``-Os``)
322 * ``speed_optimized`` -- optimized for speed, without increasing code size
325 Pigweed defines versions of its groups in ``//BUILD.gn`` for each optimization
326 level. Rather than relying on ``pw_default_optimization_level``, you may
327 directly build a group at the desired optimization level:
328 ``<group>_<optimization>``. Examples include ``host_clang_debug``,
329 ``host_gcc_size_optimized``, and ``stm32f429i_speed_optimized``.
331 Upstream GN target groups
332 ^^^^^^^^^^^^^^^^^^^^^^^^^
333 In upstream, Pigweed splits its top-level GN targets into a few logical groups,
334 which are described below. In order to build a GN target, it *must* be listed in
335 one of the groups in this file.
339 This group defines the application images built in Pigweed. It lists all of the
340 common images built across all Pigweed targets, such as modules' example
341 executables. Each Pigweed target can additionally provide its own specific
342 images through the ``pw_TARGET_APPLICATIONS`` build arg, which is included by
347 This group defines host-side tooling binaries built for Pigweed.
351 This group lists the main libraries for all of Pigweed's modules.
355 All modules' unit tests are collected here, so that they can all be run at once.
359 This group defines everything built in a Pigweed build invocation by collecting
360 the above groups and conditionally depending on them based on the active Pigweed
361 target's configuration. Generally, new dependencies should not be added here;
362 instead, use one of the groups listed above.
364 The ``pigweed_default`` group is instantiated for each upstream Pigweed target's
367 Pigweed target instantiations
368 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
369 These groups wrap ``pigweed_default`` with a specific target toolchain. They are
370 named after the Pigweed target, e.g. ``host_clang``, ``stm32f429i``, etc.
372 Other BUILD files: //\*\*/BUILD.gn
373 ----------------------------------
374 The rest of the ``BUILD.gn`` files in the tree define libraries, configs, and
375 build args for each of the modules in a Pigweed project.
377 Project configuration: //build_overrides/pigweed.gni
378 ----------------------------------------------------
379 Each Pigweed project must contain a Pigweed configuration file at a known
380 location in the GN build tree. Currently, this file only contains a single build
381 argument, which must be set to the GN build path to the root of the Pigweed
382 repository within the project.
386 As Pigweed is inteded to be a subcomponent of a larger project, it cannot assume
387 where it or its modules is located. Therefore, Pigweed's upstream BUILD.gn files
388 do not use absolute paths; instead, variables are defined pointing to each of
389 Pigweed's modules, set relative to a project-specific ``dir_pigweed``.
391 To depend on Pigweed modules from GN code, import Pigweed's overrides file and
392 reference these module variables.
396 # This must be imported before .gni files from any other Pigweed modules. To
397 # prevent gn format from reordering this import, it must be separated by a
398 # blank line from other imports.
400 import("//build_overrides/pigweed.gni")
402 GN target type wrappers
403 -----------------------
404 To faciliate injecting global configuration options, Pigweed defines wrappers
405 around builtin GN target types such as ``source_set`` and ``executable``. These
406 are defined within ``$dir_pw_build/target_types.gni``.
409 To take advantage of Pigweed's flexible target configuration system, use
410 ``pw_*`` target types (e.g. ``pw_source_set``) in your BUILD.gn files instead
415 To build for a specific hardware platform, builds define Pigweed targets. These
416 are essentially GN toolchains which set special arguments telling Pigweed how to
417 build. For information on Pigweed's target system, refer to
422 Pigweed's ``BUILDCONFIG.gn`` sets the project's default toolchain to a "dummy"
423 toolchain which does not specify any compilers or override any build arguments.
424 Downstream projects are recommended to do the same, following the steps
425 described in :ref:`top-level-build` to configure builds for each of their
428 .. admonition:: Why use a dummy?
430 To support some of its advanced (and useful!) build features, Pigweed requires
431 the ability to generate new toolchains on the fly. This requires having
432 knowledge of the full configuration of the current toolchain (which is easy if
433 it's all defined within a scope), something which is impractical to achieve
434 using the default toolchain.
436 Additionally, there are some cases where GN treats default and non-default
437 toolchains differently. By not using the default toolchain, we avoid having
438 to deal with these inconsistencies.
440 It is possible to build Pigweed using only the default toolchain, but it
441 requires a more complicated setup to get everything working and should be
442 avoided unless necessary (for example, when integrating with a large existing
445 Upstream development examples
446 -----------------------------
447 If developing for upstream Pigweed, some common build use cases are described
450 Building a custom executable/app image
451 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
453 1. Define your executable GN target using the ``pw_executable`` template.
458 pw_executable("foo") {
459 sources = [ "main.cc" ]
463 2. In the root ``BUILD.gn`` file, add the executable's GN target to the ``apps``
472 "//foo", # Shorthand for //foo:foo
476 3. Run the ninja build to compile your executable. The apps group is built by
477 default, so there's no need to provide a target. The executable will be
478 compiled for every supported Pigweed target.
484 Alternatively, build your executable by itself by specifying its path to
485 Ninja. When building a GN target manually, the Pigweed target for which it
486 is built must be specified on the Ninja command line.
488 For example, to build for the Pigweed target ``host_gcc_debug``:
492 ninja -C out host_gcc_debug/obj/foo/bin/foo
496 The path passed to Ninja is a filesystem path within the ``out`` directory,
497 rather than a GN path. This path can be found by running ``gn outputs``.
499 4. Retrieve your compiled binary from the out directory. It is located at the
504 out/<pw_target>/obj/<gn_path>/{bin,test}/<executable>
506 where ``pw_target`` is the Pigweed target for which the binary was built,
507 ``gn_path`` is the GN path to the BUILD.gn file defining the executable,
508 and ``executable`` is the executable's GN target name (potentially with an
509 extension). Note that the executable is located within a ``bin`` subdirectory
510 in the module (or ``test`` for unit tests defined with ``pw_test``).
512 For example, the ``foo`` executable defined above and compiled for the
513 Pigweed target stm32f429i_disc1_debug is found at:
517 out/stm32f429i_disc1_debug/obj/foo/bin/foo