Merge pull request #91 from pcc/reload-manifest

[platform/upstream/ninja.git] / doc / manual.asciidoc

Ninja
=====
Evan Martin <martine@danga.com>


Introduction
------------

Ninja is yet another build system.  It takes as input the
interdependencies of files (typically source code and output
executables) and orchestrates building them, _quickly_.

Ninja joins a sea of other build systems.  Its distinguishing goal is
to be fast.  It is born from my work on the Chromium browser project,
which has over 30,000 source files and whose other build systems
(including one built from custom non-recursive Makefiles) can take ten
seconds to start building after changing one file.  Ninja is under a
second.


Philosophical overview
~~~~~~~~~~~~~~~~~~~~~~

Build systems get slow when they need to make decisions.  When you are
in a edit-compile cycle you want it to be as fast as possible -- you
want the build system to do the minimum work necessary to figure out
what needs to be built immediately.

Ninja contains the barest functionality necessary to describe
arbitrary dependency graphs.  Its lack of syntax makes it impossible
to express complex decisions.  Instead, Ninja is intended to be used
with a separate program generating its input files.  The generator
program (like the `./configure` found in autotools projects) can
analyze system dependencies and make as many decisions as possible up
front so that incremental builds stay fast.  Going beyond autotools,
even build-time decisions like "which compiler flags should I use?"
or "should I build a debug or release-mode binary?"  belong in the
`.ninja` file generator.

Conceptual overview
~~~~~~~~~~~~~~~~~~~

Ninja evaluates a graph of dependencies between files, and runs
whichever commands are necessary to make your build target up to date.
If you are familiar with Make, Ninja is very similar.

A build file (default name: `build.ninja`) provides a list of _rules_
-- short names for longer commands, like how to run the compiler --
along with a list of _build_ statements saying how to build files
using the rules -- which rule to apply to which inputs to produce
which outputs.

Conceptually, `build` statements describe the dependency graph of your
project, while `rule` statements describe how to generate the files
along a given edge of the graph.

Design goals
~~~~~~~~~~~~

Here are some of the design goals of Ninja:

* very fast (i.e., instant) incremental builds, even for very large
  projects.

* very little policy about how code is built; "explicit is better than
  implicit".

* get dependencies correct, and in particular situations that are
  difficult to get right with Makefiles (e.g. outputs need an implicit
  dependency on the command line used to generate them; to build C
  source code you need to use gcc's `-M` flags for header
  dependencies).

* when convenience and speed are in conflict, prefer speed.

Some explicit _non-goals_:

* convenient syntax for writing build files by hand.  _You should
  generate your ninja files using another program_.  This is how we
  can sidestep many policy decisions.

* built-in rules. _Out of the box, Ninja has no rules for
  e.g. compiling C code._

* build-time customization of the build. _Options belong in
  the program that generates the ninja files_.

* build-time decision-making ability such as conditionals or search
  paths. _Making decisions is slow._

To restate, Ninja is faster than other build systems because it is
painfully simple.  You must tell Ninja exactly what to do when you
create your project's `.ninja` files.

Comparison to GNU make
~~~~~~~~~~~~~~~~~~~~~~

Ninja is closest in spirit and functionality to make.  But
fundamentally, make has a lot of _features_: suffix rules, functions,
built-in rules that e.g. search for RCS files when building source.
Many projects find make alone adequate for their build problems.  In
contrast, Ninja has almost no features; just those necessary to get
builds correct while punting most complexity to generation of the
ninja input files.  Ninja by itself is unlikely to be useful for most
projects.

Here are some of the features ninja adds to make.  (These sorts of
features can often be implemented using more complicated Makefiles,
but they are not part of make itself.)

* A Ninja rule may point at a path for extra implicit dependency
  information.  This makes it easy to get header dependencies correct
  for C/C++ code.

* A build edge may have multiple outputs.

* Outputs implicitly depend on the command line that was used to generate
  them, which means that changing e.g. compilation flags will cause
  the outputs to rebuild.

* Output directories are always implicitly created before running the
  command that relies on them.

* Rules can provide shorter descriptions of the command being run, so
  you can print e.g. `CC foo.o` instead of a long command line while
  building.

* Command output is always buffered.  This means commands running in
  parallel don't interleave their output, and when a command fails we
  can print its failure output next to the full command line that
  produced the failure.


Getting started
---------------

Start by downloading the code from
https://github.com/martine/ninja[the github repo].

The included `bootstrap.sh` should hopefully produce a working `ninja`
binary, by first blindly compiling all non-test files together then
re-building Ninja using itself.

Usage is currently just

----------------
ninja target
----------------

where `target` is a known output described by `build.ninja` in the
current directory.

There is no installation step; the only files of interest to a user
are the resulting binary and this manual.


Creating .ninja files
---------------------
Here's a basic `.ninja` file that demonstrates most of the syntax.
It will be used as an example for the following sections.

---------------------------------
cflags = -Wall

rule cc
  command = gcc $cflags -c $in -o $out

build foo.o: cc foo.c
---------------------------------

Variables
~~~~~~~~~
Despite the non-goal of being convenient to write by hand, to keep
build files readable (debuggable), Ninja supports declaring shorter
reusable names for strings.  A declaration like the following

----------------
cflags = -g
----------------

can be used on the right side of an equals sign, dereferencing it with
a dollar sign, like this:

----------------
rule cc
  command = gcc $cflags -c $in -o $out
----------------

Variables can also be referenced using curly braces like `${in}`.

Variables might better be called "bindings", in that a given variable
cannot be changed, only shadowed.  There is more on how shadowing works
later in this document.

Rules
~~~~~

Rules declare a short name for a command line.  They begin with a line
consisting of the `rule` keyword and a name for the rule.  Then
follows an indented set of `variable = value` lines.

The basic example above declares a new rule named `cc`, along with the
command to run.  (In the context of a rule, the `command` variable is
special and defines the command to run.  A full list of special
variables is provided in <<ref_rule,the reference>>.)

Within the context of a rule, two additional special variables are
available: `$in` expands to the list of input files (`foo.c`) and
`$out` to the output file (`foo.o`) for the command.


Build statements
~~~~~~~~~~~~~~~~

Build statements declare a relationship between input and output
files.  They begin with the `build` keyword, and have the format
+build _outputs_: _rulename_ _inputs_+.  Such a declaration says that
all of the output files are derived from the input files.  When the
output files are missing or when the inputs change, Ninja will run the
rule to regenerate the outputs.

The basic example above describes how to build `foo.o`, using the `cc`
rule.

In the scope of a `build` block (including in the evaluation of its
associated `rule`), the variable `$in` is the list of inputs and the
variable `$out` is the list of outputs.

A build statement may be followed by an indented set of `key = value`
pairs, much like a rule.  These variables will shadow any variables
when evaluating the variables in the command.  For example:

----------------
cflags = -Wall -Werror
rule cc
  command = gcc $cflags -c $in -o $out

# If left unspecified, builds get the outer $cflags.
build foo.o: cc foo.c

# But you can can shadow variables like cflags for a particular build.
build special.o: cc special.c
  cflags = -Wall

# The variable was only shadowed for the scope of special.o;
# Subsequent build lines get the outer (original) cflags.
build bar.o: cc bar.c

----------------

For more discussion of how scoping works, consult <<ref_scope,the
reference>>.

If you need more complicated information passed from the build
statement to the rule (for example, if the rule needs "the file
extension of the first input"), pass that through as an extra
variable, like how `cflags` is passed above.

If the top-level Ninja file is specified as an output of any build
statement and it is out of date, Ninja will rebuild and reload it
before building the targets requested by the user.


The `phony` rule
~~~~~~~~~~~~~~~~

The special rule name `phony` can be used to create aliases for other
targets.  For example:

----------------
build foo: phony some/file/in/a/faraway/subdir/foo
----------------

This makes `ninja foo` build the longer path.  Semantically, the
`phony` rule is equivalent to a plain rule where the `command` does
nothing, but phony rules are handled specially in that they aren't
printed when run, logged (see below), nor do they contribute to the
command count printed as part of the build process.


Default target statements
~~~~~~~~~~~~~~~~~~~~~~~~~

By default, if no targets are specified on the command line, Ninja
will build every output that is not named as an input elsewhere.
You can override this behavior using a default target statement.
A default target statement causes Ninja to build only a given subset
of output files if none are specified on the command line.

Default target statements begin with the `default` keyword, and have
the format +default _targets_+.  A default target statement must appear
after the build statement that declares the target as an output file.
They are cumulative, so multiple statements may be used to extend
the list of default targets.  For example:

----------------
default foo bar
default baz
----------------

This causes Ninja to build the `foo`, `bar` and `baz` targets by
default.


The Ninja log
~~~~~~~~~~~~~

For each built file, Ninja keeps a log of the command used to build
it.  Using this log Ninja can know when an existing output was built
with a different command line than the build files specify (i.e., the
command line changed) and knows to rebuild the file.

The log file is kept in the build root in a file called `.ninja_log`.
If you provide a variable named `builddir` in the outermost scope,
`.ninja_log` will be kept in that directory instead.


Generating Ninja files
----------------------

The Ninja distribution includes a tiny (<100 line) Python module to
facilitate generating Ninja files.  It allows you to make Python calls
like `ninja.rule(name='foo', command='bar', depfile='$out.d')` and
it will generate the appropriate syntax.

Integration with other build systems
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

*gyp*. http://gyp.googlecode.com[gyp, the system used to generate build
files for the Chromium browser] can generate ninja files for Linux.
See http://code.google.com/p/chromium/wiki/NinjaBuild[the Chromium
Ninja documentation] for details.

*CMake*.  Various people have worked on generating Ninja files from
CMake.  Currently, it sounds like
https://github.com/pcc/CMake/tree/ninja-generator[pcc's branch] is the
one to try.

*Autotools*.  In theory, you could coax Automake into producing
.ninja files as well, but I haven't tried it.  It may very well be the
case that most projects use too much Makefile syntax in their `.am`
files for this to work.

Extra tools
-----------

The `-t` flag on the Ninja command line runs some tools that I have
found useful during Ninja's development.  The current tools are:

`query`:: dump the inputs and outputs of a given target.

`browse`:: browse the dependency graph in a web browser.  Clicking a
file focuses the view on that file, showing inputs and outputs.  This
feature requires a Python installation.

`graph`:: output a file in the syntax used by `graphviz`, a automatic
graph layout tool.  Use it like: +ninja -t graph _target_ | dot -Tpng
-ograph.png /dev/stdin+ .  In the Ninja source tree, `ninja graph`
generates an image for Ninja itself.  If no target is given generate a
graph for all root targets.

`targets`:: output a list of targets either by rule or by depth.  If used
like this +ninja -t targets rule _name_+ it prints the list of targets
using the given rule to be built.  If no rule is given, it prints the source
files (the leaves of the graph).  If used like this
+ninja -t targets depth _digit_+ it
prints the list of targets in a depth-first manner starting by the root
targets (the ones with no outputs). Indentation is used to mark dependencies.
If the depth is zero it prints all targets. If no arguments are provided
+ninja -t targets depth 1+ is assumed. In this mode targets may be listed
several times. If used like this +ninja -t targets all+ it
prints all the targets available without indentation and it is way faster
than the _depth_ mode.  It returns non-zero if an error occurs.

`rules`:: output the list of all rules with their description if they have
one.  It can be used to know which rule name to pass to
+ninja -t targets rule _name_+.

`clean`:: remove built files.  If used like this +ninja -t clean+ it
removes all the built files.  If used like this
+ninja -t clean _targets..._+ or like this
+ninja -t clean target _targets..._+ it removes the given targets and
recursively all files built for it.  If used like this
+ninja -t clean rule _rules_+ it removes all files built using the given
rules. The depfiles are not removed. Files created but not referenced in
the graph are not removed. This tool takes in account the +-v+ and the
+-n+ options (note that +-n+ implies +-v+).  It returns non-zero if an
error occurs.

Ninja file reference
--------------------

A file is a series of declarations.  A declaration can be one of:

1. A rule declaration, which begins with +rule _rulename_+, and
   then has a series of indented lines defining variables.

2. A build edge, which looks like +build _output1_ _output2_:
   _rulename_ _input1_ _input2_+. +
   Implicit dependencies may be tacked on the end with +|
   _dependency1_ _dependency2_+. +
   Order-only dependencies may be tacked on the end with +||
   _dependency1_ _dependency2_+.  (See <<ref_dependencies,the reference on
   dependency types>>.)

3. Variable declarations, which look like +_variable_ = _value_+.

4. Default target statements, which look like +default _target1_ _target2_+.

5. References to more files, which look like +subninja _path_+ or
   +include _path_+.  The difference between these is explained below
   <<ref_scope,in the discussion about scoping>>.

Comments begin with `#` and extend to the end of the line.

Newlines are significant, but they can be escaped by putting a `$`
before them.  To produce a literal `$` in a path or variable value use
`$$`.

Other whitespace is only significant if it's at the beginning of a
line.  If a line is intended more than the previous one, it's
considered part of its parent's scope; if it is indented less than the
previous one, it closes the previous scope.

Rule variables
~~~~~~~~~~~~~~
[[ref_rule]]

A `rule` block contains a list of `key = value` declarations that
affect the processing of the rule.  Here is a full list of special
keys.

`command` (_required_):: the command line to run.  This string (after
  $variables are expanded) is passed directly to `sh -c` without
  interpretation by Ninja.

`depfile`:: path to an optional `Makefile` that contains extra
  _implicit dependencies_ (see <<ref_dependencies,the reference on
  dependency types>>).  This is explicitly to support `gcc` and its `-M`
  family of flags, which output the list of headers a given `.c` file
  depends on.
+
Use it like in the following example:
+
----
rule cc
  depfile = $out.d
  command = gcc -MMD -MF $out.d [other gcc flags here]
----
+
When loading a `depfile`, Ninja implicitly adds edges such that it is
not an error if the listed dependency is missing.  This allows you to
delete a depfile-discovered header file and rebuild, without the build
aborting due to a missing input.

`description`:: a short description of the command, used to pretty-print
  the command as it's running.  The `-v` flag controls whether to print
  the full command or its description; if a command fails, the full command
  line will always be printed before the command's output.

Additionally, the special `$in` and `$out` variables expand to the
space-separated list of files provided to the `build` line referencing
this `rule`.

Build dependencies
~~~~~~~~~~~~~~~~~~
[[ref_dependencies]]

There are three types of build dependencies which are subtly different.

1. _Explicit dependencies_, as listed in a build line.  These are
   available as the `$in` variable in the rule.  Changes in these files
   cause the output to be rebuilt; if these file are missing and
   Ninja doesn't know how to build them, the build is aborted.
+
This is the standard form of dependency to be used for e.g. the
source file of a compile command.

2. _Implicit dependencies_, either as picked up from
   a `depfile` attribute on a rule or from the syntax +| _dep1_
   _dep2_+ on the end of a build line.  The semantics are identical to
   explicit dependencies, the only difference is that implicit dependencies
   don't show up in the `$in` variable.
+
This is for expressing dependencies that don't show up on the
command line of the command; for example, for a rule that runs a
script, the script itself should be an implicit dependency, as
changes to the script should cause the output to rebuild.
+
Note that dependencies as loaded through depfiles have slightly different
semantics, as described in the <<ref_rule,rule reference>>.

3. _Order-only dependencies_, expressed with the syntax +|| _dep1_
   _dep2_+ on the end of a build line.  When these are missing, the
   output is not rebuilt until they are built, but once they are
   available further changes to the files do not affect the output.
+
Order-only dependencies can be useful for bootstrapping dependencies
that are only discovered during build time: for example, to generate a
header file before starting a subsequent compilation step.  (Once the
header is used in compilation, a generated dependency file will then
express the implicit dependency.)

Evaluation and scoping
~~~~~~~~~~~~~~~~~~~~~~
[[ref_scope]]

Top-level variable declarations are scoped to the file they occur in.

The `subninja` keyword, used to include another `.ninja` file,
introduces a new scope.  The included `subninja` file may use the
variables from the parent file, and shadow their values for the file's
scope, but it won't affect values of the variables in the parent.

To include another `.ninja` file in the current scope, much like a C
`#include` statement, use `include` instead of `subninja`.

Variable declarations indented in a `build` block are scoped to the
`build` block.  This scope is inherited by the `rule`.  The full
lookup order for a variable referenced in a rule is:

1. Rule-level variables (i.e. `$in`, `$command`).

2. Build-level variables from the `build` that references this rule.

3. File-level variables from the file that the `build` line was in.

4. Variables from the file that included that file using the
   `subninja` keyword.

Variable expansion
~~~~~~~~~~~~~~~~~~

Variables are expanded in three cases: in the right side of a `name =
value` statement, in paths in a `build` statement and in paths in
a `default` statement.

When a `name = value` statement is evaluated, its right-hand side is
expanded once (according to the above scoping rules) immediately, and
from then on `$name` expands to the static string as the result of the
expansion.  It is never the case that you'll need to "double-escape" a
variable with some syntax like `$$foo`.

A `build` or `default` statement is first parsed as a space-separated
list of filenames and then each name is expanded.  This means that
spaces within a variable will result in spaces in the expanded
filename.

----
spaced = foo bar
build $spaced/baz other: ...
# The above build line has two outputs: "foo bar/baz" and "other".
----

In a `name = value` statement, whitespace at the beginning of a value
is always stripped.  Whitespace at the beginning of a line after a
line continuation is also stripped.

----
two_words_with_one_space = foo $
    bar
one_word_with_no_space = foo$
    bar
----


Future work
-----------

Some pieces I'd like to add:

_inotify_.  I had originally intended to make Ninja be memory-resident
and to use `inotify` to keep the build state hot at all times.  But
upon writing the code I found it was fast enough to run from scratch
each time.  Perhaps a slower computer would still benefit from
inotify; the data structures are set up such that using inotify
shouldn't be hard.

_build estimation and analysis_.  As part of build correctness, Ninja
keeps a log of the time spent on each build statement.  This log
format could be adjusted to instead store timing information across
multiple runs.  From that, the total time necessary to build could be
estimated, allowing Ninja to print status like "3 minutes until
complete" when building.  Additionally, a tool could output which
commands are the slowest or which directories take the most time
to build.

_many others_.  See the `todo` file included in the distribution.