1 <!--#include file="header.html" -->
3 <p><h1>Code style</h1></p>
5 <p>The primary goal of toybox is _simple_ code. Keeping the code small is
6 second, with speed and lots of features coming in somewhere after that.
7 (For more on that, see the <a href=design.html>design</a> page.)</p>
9 <p>A simple implementation usually takes up fewer lines of source code,
10 meaning more code can fit on the screen at once, meaning the programmer can
11 see more of it on the screen and thus keep more if in their head at once.
12 This helps code auditing and thus reduces bugs. That said, sometimes being
13 more explicit is preferable to being clever enough to outsmart yourself:
14 don't be so terse your code is unreadable.</p>
16 <p>Toybox source uses two spaces per indentation level, and wraps at 80
19 <p>Gotos are allowed for error handling, and for breaking out of
20 nested loops. In general, a goto should only jump forward (not back), and
21 should either jump to the end of an outer loop, or to error handling code
22 at the end of the function. Goto labels are never indented: they override the
23 block structure of the file. Putting them at the left edge makes them easy
24 to spot as overrides to the normal flow of control, which they are.</p>
26 <p><h1>Building Toybox:</h1></p>
28 <p>Toybox is configured using the Kconfig language pioneered by the Linux
29 kernel, and adopted by many other projects (uClibc, OpenEmbedded, etc).
30 This generates a ".config" file containing the selected options, which
31 controls which features are included when compiling toybox.</p>
33 <p>Each configuration option has a default value. The defaults indicate the
34 "maximum sane configuration", I.E. if the feature defaults to "n" then it
35 either isn't complete or is a special-purpose option (such as debugging
36 code) that isn't intended for general purpose use.</p>
38 <p>The standard build invocation is:</p>
41 <li>make defconfig #(or menuconfig)</li>
46 <p>Type "make help" to see all available build options.</p>
48 <p>The file "configure" contains a number of environment variable definitions
49 which influence the build, such as specifying which compiler to use or where
50 to install the resulting binaries. This file is included by the build, but
51 accepts existing definitions of the environment variables, so it may be sourced
52 or modified by the developer before building and the definitions exported
53 to the environment will take precedence.</p>
55 <p>(To clarify: "configure" describes the build and installation environment,
56 ".config" lists the features selected by defconfig/menuconfig.)</p>
58 <p><h1>Infrastructure:</h1></p>
60 <p>The toybox source code is in following directories:</p>
62 <li>The <a href="#top">top level directory</a> contains the file main.c (were
63 execution starts), the header file toys.h (included by every command), and
64 other global infrastructure.</li>
65 <li>The <a href="#lib">lib directory</a> contains common functions shared by
66 multiple commands:</li>
68 <li><a href="#lib_lib">lib/lib.c</a></li>
69 <li><a href="#lib_llist">lib/llist.c</a></li>
70 <li><a href="#lib_args">lib/args.c</a></li>
71 <li><a href="#lib_dirtree">lib/dirtree.c</a></li>
73 <li>The <a href="#toys">toys directory</a> contains the C files implementating
74 each command. Currently it contains three subdirectories:
75 posix, lsb, and other.</li>
76 <li>The <a href="#scripts">scripts directory</a> contains the build and
77 test infrastructure.</li>
78 <li>The <a href="#kconfig">kconfig directory</a> contains the configuration
79 infrastructure implementing menuconfig (copied from the Linux kernel).</li>
80 <li>The <a href="#generated">generated directory</a> contains intermediate
81 files generated from other parts of the source code.</li>
85 <p><h1>Adding a new command</h1></p>
86 <p>To add a new command to toybox, add a C file implementing that command under
87 the toys directory. No other files need to be modified; the build extracts
88 all the information it needs (such as command line arguments) from specially
89 formatted comments and macros in the C file. (See the description of the
90 <a href="#generated">"generated" directory</a> for details.)</p>
92 <p>Currently there are three subdirectories under "toys", one for commands
93 defined by the POSIX standard, one for commands defined by the Linux Standard
94 Base, and one for all other commands. (This is just for developer convenience
95 sorting them, the directories are otherwise functionally identical.)</p>
97 <p>An easy way to start a new command is copy the file "toys/other/hello.c" to
98 the name of the new command, and modify this copy to implement the new command.
99 This file is an example command meant to be used as a "skeleton" for
100 new commands (more or less by turning every instance of "hello" into the
101 name of your command, updating the command line arguments, globals, and
102 help data, and then filling out its "main" function with code that does
103 something interesting). It provides examples of all the build infrastructure
104 (including optional elements like command line argument parsing and global
105 variables that a "hello world" program doesn't strictly need).</p>
107 <p>Here's a checklist of steps to turn hello.c into another command:</p>
110 <li><p>First "cd toys/other" and "cp hello.c yourcommand.c". Note that the name
111 of this file is significant, it's the name of the new command you're adding
112 to toybox. Open your new file in your favorite editor.</p></li>
114 <li><p>Change the one line comment at the top of the file (currently
115 "hello.c - A hello world program") to describe your new file.</p></li>
117 <li><p>Change the copyright notice to your name, email, and the current
120 <li><p>Give a URL to the relevant standards document, where applicable.
121 (Sample links to SUSv4 and LSB are provided, feel free to link to other
122 documentation or standards as appropriate.)</p></li>
124 <li><p>Update the USE_YOURCOMMAND(NEWTOY(yourcommand,"blah",0)) line.
125 The NEWTOY macro fills out this command's <a href="#toy_list">toy_list</a>
126 structure. The arguments to the NEWTOY macro are:</p>
129 <li><p>the name used to run your command</p></li>
130 <li><p>the command line argument <a href="#lib_args">option parsing string</a> (NULL if none)</p></li>
131 <li><p>a bitfield of TOYFLAG values
132 (defined in toys.h) providing additional information such as where your
133 command should be installed on a running system, whether to blank umask
134 before running, whether or not the command must run as root (and thus should
135 retain root access if installed SUID), and so on.</p></li>
139 <li><p>Change the kconfig data (from "config YOURCOMMAND" to the end of the
140 comment block) to supply your command's configuration and help
141 information. The uppper case config symbols are used by menuconfig, and are
142 also what the CFG_ and USE_() macros are generated from (see [TODO]). The
143 help information here is used by menuconfig, and also by the "help" command to
144 describe your new command. (See [TODO] for details.) By convention,
145 unfinished commands default to "n" and finished commands default to "y",
146 so "make defconfig" selects all finished commands. (Note, "finished" means
147 "ready to be used", not that it'll never change again.)<p>
149 <p>Each help block should start with a "usage: yourcommand" line explaining
150 any command line arguments added by this config option. The "help" command
151 outputs this text, and scripts/config2help.c in the build infrastructure
152 collates these usage lines for commands with multiple configuration
153 options when producing generated/help.h.</p>
156 <li><p>Change the "#define FOR_hello" line to "#define FOR_yourcommand" right
157 before the "#include <toys.h>". (This selects the appropriate FLAG_ macros and
158 does a "#define TT this.yourcommand" so you can access the global variables
159 out of the space-saving union of structures. If you aren't using any command
160 flag bits and aren't defining a GLOBAL block, you can delete this line.)</p></li>
162 <li><p>Update the GLOBALS() macro to contain your command's global
163 variables. If your command has no global variables, delete this macro.</p>
165 <p>Variables in the GLOBALS() block are are stored in a space saving
166 <a href="#toy_union">union of structures</a> format, which may be accessed
167 using the TT macro as if TT were a global structure (so TT.membername).
168 If you specified two-character command line arguments in
169 NEWTOY(), the first few global variables will be initialized by the automatic
170 argument parsing logic, and the type and order of these variables must
171 correspond to the arguments specified in NEWTOY().
172 (See <a href="#lib_args">lib/args.c</a> for details.)</p></li>
174 <li><p>Rename hello_main() to yourcommand_main(). This is the main() function
175 where execution of your command starts. Your command line options are
176 already sorted into this.optflags, this.optargs, this.optc, and the GLOBALS()
177 as appropriate by the time this function is called. (See
178 <a href="#lib_args">get_optflags()</a> for details.</p></li>
181 <p><a name="top" /><h2>Top level directory.</h2></p>
183 <p>This directory contains global infrastructure.</p>
186 <p>Each command #includes "toys.h" as part of its standard prolog. It
187 may "#define FOR_commandname" before doing so to get some extra entries
188 specific to this command.</p>
190 <p>This file sucks in most of the commonly used standard #includes, so
191 individual files can just #include "toys.h" and not have to worry about
192 stdargs.h and so on. Individual commands still need to #include
193 special-purpose headers that may not be present on all systems (and thus would
194 prevent toybox from building that command on such a system with that command
195 enabled). Examples include regex support, any "linux/" or "asm/" headers, mtab
196 support (mntent.h and sys/mount.h), and so on.</p>
198 <p>The toys.h header also defines structures for most of the global variables
199 provided to each command by toybox_main(). These are described in
200 detail in the description for main.c, where they are initialized.</p>
202 <p>The global variables are grouped into structures (and a union) for space
203 savings, to more easily track the amount of memory consumed by them,
204 so that they may be automatically cleared/initialized as needed, and so
205 that access to global variables is more easily distinguished from access to
209 <p>Contains the main() function where execution starts, plus
210 common infrastructure to initialize global variables and select which command
211 to run. The "toybox" multiplexer command also lives here. (This is the
212 only command defined outside of the toys directory.)</p>
214 <p>Execution starts in main() which trims any path off of the first command
215 name and calls toybox_main(), which calls toy_exec(), which calls toy_find()
216 and toy_init() before calling the appropriate command's function from
217 toy_list[] (via toys.which->toy_main()).
218 If the command is "toybox", execution recurses into toybox_main(), otherwise
219 the call goes to the appropriate commandname_main() from a C file in the toys
222 <p>The following global variables are defined in main.c:</p>
224 <a name="toy_list" />
225 <li><p><b>struct toy_list toy_list[]</b> - array describing all the
226 commands currently configured into toybox. The first entry (toy_list[0]) is
227 for the "toybox" multiplexer command, which runs all the other built-in commands
228 without symlinks by using its first argument as the name of the command to
229 run and the rest as that command's argument list (ala "./toybox echo hello").
230 The remaining entries are the commands in alphabetical order (for efficient
233 <p>This is a read-only array initialized at compile time by
234 defining macros and #including generated/newtoys.h.</p>
236 <p>Members of struct toy_list (defined in "toys.h") include:</p>
238 <li><p>char *<b>name</b> - the name of this command.</p></li>
239 <li><p>void (*<b>toy_main</b>)(void) - function pointer to run this
241 <li><p>char *<b>options</b> - command line option string (used by
242 get_optflags() in lib/args.c to intialize toys.optflags, toys.optargs, and
243 entries in the toy's GLOBALS struct). When this is NULL, no option
244 parsing is done before calling toy_main().</p></li>
245 <li><p>int <b>flags</b> - Behavior flags for this command. The following flags are currently understood:</p>
248 <li><b>TOYFLAG_USR</b> - Install this command under /usr</li>
249 <li><b>TOYFLAG_BIN</b> - Install this command under /bin</li>
250 <li><b>TOYFLAG_SBIN</b> - Install this command under /sbin</li>
251 <li><b>TOYFLAG_NOFORK</b> - This command can be used as a shell builtin.</li>
252 <li><b>TOYFLAG_UMASK</b> - Call umask(0) before running this command.</li>
253 <li><b>TOYFLAG_STAYROOT</b> - Don't drop permissions for this command if toybox is installed SUID root.</li>
254 <li><b>TOYFLAG_NEEDROOT</b> - This command cannot function unless run with root access.</li>
258 <p>These flags are combined with | (or). For example, to install a command
259 in /usr/bin, or together TOYFLAG_USR|TOYFLAG_BIN.</p>
263 <li><p><b>struct toy_context toys</b> - global structure containing information
264 common to all commands, initializd by toy_init() and defined in "toys.h".
265 Members of this structure include:</p>
267 <li><p>struct toy_list *<b>which</b> - a pointer to this command's toy_list
268 structure. Mostly used to grab the name of the running command
269 (toys->which.name).</p>
271 <li><p>int <b>exitval</b> - Exit value of this command. Defaults to zero. The
272 error_exit() functions will return 1 if this is zero, otherwise they'll
273 return this value.</p></li>
274 <li><p>char **<b>argv</b> - "raw" command line options, I.E. the original
275 unmodified string array passed in to main(). Note that modifying this changes
276 "ps" output, and is not recommended. This array is null terminated; a NULL
277 entry indicates the end of the array.</p>
278 <p>Most commands don't use this field, instead the use optargs, optflags,
279 and the fields in the GLOBALS struct initialized by get_optflags().</p>
281 <li><p>unsigned <b>optflags</b> - Command line option flags, set by
282 <a href="#lib_args">get_optflags()</a>. Indicates which of the command line options listed in
283 toys->which.options occurred this time.</p>
285 <p>The rightmost command line argument listed in toys->which.options sets bit
286 1, the next one sets bit 2, and so on. This means the bits are set in the same
287 order the binary digits would be listed if typed out as a string. For example,
288 the option string "abcd" would parse the command line "-c" to set optflags to 2,
289 "-a" would set optflags to 8, and "-bd" would set optflags to 6 (4|2).</p>
291 <p>Only letters are relevant to optflags. In the string "a*b:c#d", d=1, c=2,
292 b=4, a=8. Punctuation after a letter initializes global variables at the
293 start of the GLOBALS() block (see <a href="#toy_union">union toy_union this</a>
296 <p>The build infrastructure creates FLAG_ macros for each option letter,
297 corresponding to the bit position, so you can check (toys.optflags & FLAG_x)
298 to see if a flag was specified. (The correct set of FLAG_ macros is selected
299 by defining FOR_mycommand before #including toys.h. The macros live in
300 toys/globals.h which is generated by scripts/make.sh.)</p>
302 <p>For more information on option parsing, see <a href="#lib_args">get_optflags()</a>.</p>
305 <li><p>char **<b>optargs</b> - Null terminated array of arguments left over
306 after get_optflags() removed all the ones it understood. Note: optarg[0] is
307 the first argument, not the command name. Use toys.which->name for the command
309 <li><p>int <b>optc</b> - Optarg count, equivalent to argc but for
311 <li><p>int <b>exithelp</b> - Whether error_exit() should print a usage message
312 via help_main() before exiting. (True during option parsing, defaults to
313 false afterwards.)</p></li>
316 <a name="toy_union" />
317 <li><p><b>union toy_union this</b> - Union of structures containing each
318 command's global variables.</p>
320 <p>Global variables are useful: they reduce the overhead of passing extra
321 command line arguments between functions, they conveniently start prezeroed to
322 save initialization costs, and the command line argument parsing infrastructure
323 can also initialize global variables with its results.</p>
325 <p>But since each toybox process can only run one command at a time, allocating
326 space for global variables belonging to other commands you aren't currently
327 running would be wasteful.</p>
329 <p>Toybox handles this by encapsulating each command's global variables in
330 a structure, and declaring a union of those structures with a single global
331 instance (called "this"). The GLOBALS() macro contains the global
332 variables that should go in the current command's global structure. Each
333 variable can then be accessed as "this.commandname.varname".
334 If you #defined FOR_commandname before including toys.h, the macro TT is
335 #defined to this.commandname so the variable can then be accessed as
336 "TT.variable". See toys/hello.c for an example.</p>
338 <p>A command that needs global variables should declare a structure to
339 contain them all, and add that structure to this union. A command should never
340 declare global variables outside of this, because such global variables would
341 allocate memory when running other commands that don't use those global
344 <p>The first few fields of this structure can be intialized by <a href="#lib_args">get_optargs()</a>,
345 as specified by the options field off this command's toy_list entry. See
346 the get_optargs() description in lib/args.c for details.</p>
349 <li><b>char toybuf[4096]</b> - a common scratch space buffer so
350 commands don't need to allocate their own. Any command is free to use this,
351 and it should never be directly referenced by functions in lib/ (although
352 commands are free to pass toybuf in to a library function as an argument).</li>
355 <p>The following functions are defined in main.c:</p>
357 <li><p>struct toy_list *<b>toy_find</b>(char *name) - Return the toy_list
358 structure for this command name, or NULL if not found.</p></li>
359 <li><p>void <b>toy_init</b>(struct toy_list *which, char *argv[]) - fill out
360 the global toys structure, calling get_optargs() if necessary.</p></li>
361 <li><p>void <b>toy_exec</b>(char *argv[]) - Run a built-in command with
363 <p>Calls toy_find() on argv[0] (which must be just a command name
364 without path). Returns if it can't find this command, otherwise calls
365 toy_init(), toys->which.toy_main(), and exit() instead of returning.</p>
367 <p>Use the library function xexec() to fall back to external executables
368 in $PATH if toy_exec() can't find a built-in command. Note that toy_exec()
369 does not strip paths before searching for a command, so "./command" will
370 never match an internal command.</li>
372 <li><p>void <b>toybox_main</b>(void) - the main function for the multiplexer
373 command (I.E. "toybox"). Given a command name as its first argument, calls
374 toy_exec() on its arguments. With no arguments, it lists available commands.
375 If the first argument starts with "-" it lists each command with its default
376 install path prepended.</p></li>
382 <p>Top level configuration file in a stylized variant of
383 <a href=http://kernel.org/doc/Documentation/kbuild/kconfig-language.txt>kconfig</a> format. Includes generated/Config.in.</p>
385 <p>These files are directly used by "make menuconfig" to select which commands
386 to build into toybox (thus generating a .config file), and by
387 scripts/config2help.py to create generated/help.h.</p>
389 <h3>Temporary files:</h3>
391 <p>There is one temporary file in the top level source directory:</p>
393 <li><p><b>.config</b> - Configuration file generated by kconfig, indicating
394 which commands (and options to commands) are currently enabled. Used
395 to make generated/config.h and determine which toys/*.c files to build.</p>
397 <p>You can create a human readable "miniconfig" version of this file using
398 <a href=http://landley.net/aboriginal/new_platform.html#miniconfig>these
399 instructions</a>.</p>
403 <a name="generated" />
404 <p>The "generated/" directory contains files generated from other source code
405 in toybox. All of these files can be recreated by the build system, although
406 some (such as generated/help.h) are shipped in release versions to reduce
407 environmental dependencies (I.E. so you don't need python on your build
411 <li><p><b>generated/config.h</b> - list of CFG_SYMBOL and USE_SYMBOL() macros,
412 generated from .config by a sed invocation in the top level Makefile.</p>
414 <p>CFG_SYMBOL is a comple time constant set to 1 for enabled symbols and 0 for
415 disabled symbols. This allows the use of normal if() statements to remove
416 code at compile time via the optimizer's dead code elimination (which removes
417 from the binary any code that cannot be reached). This saves space without
418 cluttering the code with #ifdefs or leading to configuration dependent build
419 breaks. (See the 1992 Usenix paper
420 <a href=http://doc.cat-v.org/henry_spencer/ifdef_considered_harmful.pdf>#ifdef
421 Considered Harmful</a> for more information.)</p>
423 <p>USE_SYMBOL(code) evaluates to the code in parentheses when the symbol
424 is enabled, and nothing when the symbol is disabled. This can be used
425 for things like varargs or variable declarations which can't always be
426 eliminated by a simple test on CFG_SYMBOL. Note that
427 (unlike CFG_SYMBOL) this is really just a variant of #ifdef, and can
428 still result in configuration dependent build breaks. Use with caution.</p>
432 <p><h2>Directory toys/</h2></p>
434 <h3>toys/Config.in</h3>
436 <p>Included from the top level Config.in, contains one or more
437 configuration entries for each command.</p>
439 <p>Each command has a configuration entry matching the command name (although
440 configuration symbols are uppercase and command names are lower case).
441 Options to commands start with the command name followed by an underscore and
442 the option name. Global options are attached to the "toybox" command,
443 and thus use the prefix "TOYBOX_". This organization is used by
444 scripts/cfg2files to select which toys/*.c files to compile for a given
447 <p>A command with multiple names (or multiple similar commands implemented in
448 the same .c file) should have config symbols prefixed with the name of their
449 C file. I.E. config symbol prefixes are NEWTOY() names. If OLDTOY() names
450 have config symbols they're options (symbols with an underscore and suffix)
451 to the NEWTOY() name. (See toys/toylist.h)</p>
453 <h3>toys/toylist.h</h3>
454 <p>The first half of this file prototypes all the structures to hold
455 global variables for each command, and puts them in toy_union. These
456 prototypes are only included if the macro NEWTOY isn't defined (in which
457 case NEWTOY is defined to a default value that produces function
460 <p>The second half of this file lists all the commands in alphabetical
461 order, along with their command line arguments and install location.
462 Each command has an appropriate configuration guard so only the commands that
463 are enabled wind up in the list.</p>
465 <p>The first time this header is #included, it defines structures and
466 produces function prototypes for the commands in the toys directory.</p>
469 <p>The first time it's included, it defines structures and produces function
472 is used to initialize toy_list in main.c, and later in that file to initialize
473 NEED_OPTIONS (to figure out whether the command like parsing logic is needed),
474 and to put the help entries in the right order in toys/help.c.</p>
478 <p>#defines two help text strings for each command: a single line
479 command_help and an additinal command_help_long. This is used by help_main()
480 in toys/help.c to display help for commands.</p>
482 <p>Although this file is generated from Config.in help entries by
483 scripts/config2help.py, it's shipped in release tarballs so you don't need
484 python on the build system. (If you check code out of source control, or
485 modify Config.in, then you'll need python installed to rebuild it.)</p>
487 <p>This file contains help for all commands, regardless of current
488 configuration, but only the currently enabled ones are entered into help_data[]
492 <h2>Directory lib/</h2>
494 <p>TODO: document lots more here.</p>
496 <p>lib: getmountlist(), error_msg/error_exit, xmalloc(),
497 strlcpy(), xexec(), xopen()/xread(), xgetcwd(), xabspath(), find_in_path(),
500 <h3>lib/portability.h</h3>
502 <p>This file is automatically included from the top of toys.h, and smooths
503 over differences between platforms (hardware targets, compilers, C libraries,
504 operating systems, etc).</p>
506 <p>This file provides SWAP macros (SWAP_BE16(x) and SWAP_LE32(x) and so on).</p>
508 <p>A macro like SWAP_LE32(x) means "The value in x is stored as a little
509 endian 32 bit value, so perform the translation to/from whatever the native
510 32-bit format is". You do the swap once on the way in, and once on the way
511 out. If your target is already little endian, the macro is a NOP.</p>
513 <p>The SWAP macros come in BE and LE each with 16, 32, and 64 bit versions.
514 In each case, the name of the macro refers to the _external_ representation,
515 and converts to/from whatever your native representation happens to be (which
516 can vary depending on what you're currently compiling for).</p>
518 <a name="lib_llist"><h3>lib/llist.c</h3>
520 <p>Some generic single and doubly linked list functions, which take
521 advantage of a couple properties of C:</p>
524 <li><p>Structure elements are laid out in memory in the order listed, and
525 the first element has no padding. This means you can always treat (typecast)
526 a pointer to a structure as a pointer to the first element of the structure,
527 even if you don't know anything about the data following it.</p></li>
529 <li><p>An array of length zero at the end of a structure adds no space
530 to the sizeof() the structure, but if you calculate how much extra space
531 you want when you malloc() the structure it will be available at the end.
532 Since C has no bounds checking, this means each struct can have one variable
533 length array.</p></li>
536 <p>Toybox's list structures always have their <b>next</b> pointer as
537 the first entry of each struct, and singly linked lists end with a NULL pointer.
538 This allows generic code to traverse such lists without knowing anything
539 else about the specific structs composing them: if your pointer isn't NULL
540 typecast it to void ** and dereference once to get the next entry.</p>
542 <p><b>lib/lib.h</b> defines three structure types:</p>
544 <li><p><b>struct string_list</b> - stores a single string (<b>char str[0]</b>),
545 memory for which is allocated as part of the node. (I.E. llist_traverse(list,
546 free); can clean up after this type of list.)</p></li>
548 <li><p><b>struct arg_list</b> - stores a pointer to a single string
549 (<b>char *arg</b>) which is stored in a separate chunk of memory.</p></li>
551 <li><p><b>struct double_list</b> - has a second pointer (<b>struct double_list
552 *prev</b> along with a <b>char *data</b> for payload.</p></li>
555 <b>List Functions</b>
558 <li><p>void *<b>llist_pop</b>(void **list) - advances through a list ala
559 <b>node = llist_pop(&list);</b> This doesn't modify the list contents,
560 but does advance the pointer you feed it (which is why you pass the _address_
561 of that pointer, not the pointer itself).</p></li>
563 <li><p>void <b>llist_traverse</b>(void *list, void (*using)(void *data)) -
564 iterate through a list calling a function on each node.</p></li>
566 <li><p>struct double_list *<b>dlist_add</b>(struct double_list **llist, char *data)
567 - append an entry to a circular linked list.
568 This function allocates a new struct double_list wrapper and returns the
569 pointer to the new entry (which you can usually ignore since it's llist->prev,
570 but if llist was NULL you need it). The argument is the ->data field for the
572 <ul><li><p>void <b>dlist_add_nomalloc</b>(struct double_list **llist,
573 struct double_list *new) - append existing struct double_list to
574 list, does not allocate anything.</p></li></ul>
577 <b>Trivia questions:</b>
580 <li><p><b>Why do arg_list and double_list contain a char * payload instead of
581 a void *?</b> - Because you always have to typecast a void * to use it, and
582 typecasting a char * does no harm. Thus having it default to the most common
583 pointer type saves a few typecasts (strings are the most common payload),
584 and doesn't hurt anything otherwise.</p>
587 <li><p><b>Why do the names ->str, ->arg, and ->data differ?</b> - To force
588 you to keep track of which one you're using, calling free(node->str) would
589 be bad, and _failing_ to free(node->arg) leaks memory.</p></li>
591 <li><p><b>Why does llist_pop() take a void * instead of void **?</b> -
592 because the stupid compiler complains about "type punned pointers" when
593 you typecast and dereference ont he same line,
594 due to insane FSF developers hardwiring limitations of their optimizer
595 into gcc's warning system. Since C automatically typecasts any other
596 pointer _down_ to a void *, the current code works fine. It's sad that it
597 won't warn you if you forget the &, but the code crashes pretty quickly in
600 <li><p><b>How do I assemble a singly-linked-list in order?</b> - use
601 a double_list, dlist_add() your entries, and then break the circle with
602 <b>list->prev->next = NULL;</b> when done.</li>
605 <a name="lib_args"><h3>lib/args.c</h3>
607 <p>Toybox's main.c automatically parses command line options before calling the
608 command's main function. Option parsing starts in get_optflags(), which stores
609 results in the global structures "toys" (optflags and optargs) and "this".</p>
611 <p>The option parsing infrastructure stores a bitfield in toys.optflags to
612 indicate which options the current command line contained. Arguments
613 attached to those options are saved into the command's global structure
614 ("this"). Any remaining command line arguments are collected together into
615 the null-terminated array toys.optargs, with the length in toys.optc. (Note
616 that toys.optargs does not contain the current command name at position zero,
617 use "toys.which->name" for that.) The raw command line arguments get_optflags()
618 parsed are retained unmodified in toys.argv[].</p>
620 <p>Toybox's option parsing logic is controlled by an "optflags" string, using
621 a format reminiscent of getopt's optargs but has several important differences.
622 Toybox does not use the getopt()
623 function out of the C library, get_optflags() is an independent implementation
624 which doesn't permute the original arguments (and thus doesn't change how the
625 command is displayed in ps and top), and has many features not present in
626 libc optargs() (such as the ability to describe long options in the same string
627 as normal options).</p>
629 <p>Each command's NEWTOY() macro has an optflags string as its middle argument,
630 which sets toy_list.options for that command to tell get_optflags() what
631 command line arguments to look for, and what to do with them.
632 If a command has no option
633 definition string (I.E. the argument is NULL), option parsing is skipped
634 for that command, which must look at the raw data in toys.argv to parse its
635 own arguments. (If no currently enabled command uses option parsing,
636 get_optflags() is optimized out of the resulting binary by the compiler's
637 --gc-sections option.)</p>
639 <p>You don't have to free the option strings, which point into the environment
640 space (I.E. the string data is not copied). A TOYFLAG_NOFORK command
641 that uses the linked list type "*" should free the list objects but not
642 the data they point to, via "llist_free(TT.mylist, NULL);". (If it's not
643 NOFORK, exit() will free all the malloced data anyway unless you want
644 to implement a CONFIG_TOYBOX_FREE cleanup for it.)</p>
646 <h4>Optflags format string</h4>
648 <p>Note: the optflags option description string format is much more
649 concisely described by a large comment at the top of lib/args.c.</p>
651 <p>The general theory is that letters set optflags, and punctuation describes
652 other actions the option parsing logic should take.</p>
654 <p>For example, suppose the command line <b>command -b fruit -d walrus -a 42</b>
655 is parsed using the optflags string "<b>a#b:c:d</b>". (I.E.
656 toys.which->options="a#b:c:d" and argv = ["command", "-b", "fruit", "-d",
657 "walrus", "-a", "42"]). When get_optflags() returns, the following data is
658 available to command_main():
661 <li><p>In <b>struct toys</b>:
663 <li>toys.optflags = 13; // -a = 8 | -b = 4 | -d = 1</li>
664 <li>toys.optargs[0] = "walrus"; // leftover argument</li>
665 <li>toys.optargs[1] = NULL; // end of list</li>
666 <li>toys.optc=1; // there was 1 leftover argument</li>
667 <li>toys.argv[] = {"-b", "fruit", "-d", "walrus", "-a", "42"}; // The original command line arguments
671 <li><p>In <b>union this</b> (treated as <b>long this[]</b>):
673 <li>this[0] = NULL; // -c didn't get an argument this time, so get_optflags() didn't change it and toys_init() zeroed "this" during setup.)</li>
674 <li>this[1] = (long)"fruit"; // argument to -b</li>
675 <li>this[2] = 42; // argument to -a</li>
680 <p>If the command's globals are:</p>
689 <p>That would mean TT.c == NULL, TT.b == "fruit", and TT.a == 42. (Remember,
690 each entry that receives an argument must be a long or pointer, to line up
691 with the array position. Right to left in the optflags string corresponds to
692 top to bottom in GLOBALS().</p>
694 <p><b>long toys.optflags</b></p>
696 <p>Each option in the optflags string corresponds to a bit position in
697 toys.optflags, with the same value as a corresponding binary digit. The
698 rightmost argument is (1<<0), the next to last is (1<<1) and so on. If
699 the option isn't encountered while parsing argv[], its bit remains 0.</p>
702 the optflags string "abcd" would parse the command line argument "-c" to set
703 optflags to 2, "-a" would set optflags to 8, "-bd" would set optflags to
704 6 (I.E. 4|2), and "-a -c" would set optflags to 10 (2|8).</p>
706 <p>Only letters are relevant to optflags, punctuation is skipped: in the
707 string "a*b:c#d", d=1, c=2, b=4, a=8. The punctuation after a letter
708 usually indicate that the option takes an argument.</p>
710 <p>Since toys.optflags is an unsigned int, it only stores 32 bits. (Which is
711 the amount a long would have on 32-bit platforms anyway; 64 bit code on
712 32 bit platforms is too expensive to require in common code used by almost
713 all commands.) Bit positions beyond the 1<<31 aren't recorded, but
714 parsing higher options can still set global variables.</p>
716 <p><b>Automatically setting global variables from arguments (union this)</b></p>
718 <p>The following punctuation characters may be appended to an optflags
719 argument letter, indicating the option takes an additional argument:</p>
722 <li><b>:</b> - plus a string argument, keep most recent if more than one.</li>
723 <li><b>*</b> - plus a string argument, appended to a linked list.</li>
724 <li><b>@</b> - plus an occurrence counter (stored in a long)</li>
725 <li><b>#</b> - plus a signed long argument.
726 <li><b>.</b> - plus a floating point argument (if CFG_TOYBOX_FLOAT).</li>
727 <ul>The following can be appended to a float or double:
728 <li><b><123</b> - error if argument is less than this</li>
729 <li><b>>123</b> - error if argument is greater than this</li>
730 <li><b>=123</b> - default value if argument not supplied</li>
732 <ul><li>Option parsing only understands <>= after . when CFG_TOYBOX_FLOAT
733 is enabled. (Otherwise the code to determine where floating point constants
734 end drops out. When disabled, it can reserve a global data slot for the
735 argument so offsets won't change, but will never fill it out.). You can handle
736 this by using the USE_BLAH() macros with C string concatenation, ala:
737 "abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"</li></ul>
740 <p>Arguments may occur with or without a space (I.E. "-a 42" or "-a42").
741 The command line argument "-abc" may be interepreted many different ways:
742 the optflags string "cba" sets toys.optflags = 7, "c:ba" sets toys.optflags=4
743 and saves "ba" as the argument to -c, and "cb:a" sets optflags to 6 and saves
744 "c" as the argument to -b.</p>
746 <p>Options which have an argument fill in the corresponding slot in the global
747 union "this" (see generated/globals.h), treating it as an array of longs
748 with the rightmost saved in this[0]. Again using "a*b:c#d", "-c 42" would set
749 this[0]=42; and "-b 42" would set this[1]="42"; each slot is left NULL if
750 the corresponding argument is not encountered.</p>
752 <p>This behavior is useful because the LP64 standard ensures long and pointer
753 are the same size. C99 guarantees structure members will occur in memory
754 in the same order they're declared, and that padding won't be inserted between
755 consecutive variables of register size. Thus the first few entries can
756 be longs or pointers corresponding to the saved arguments.</p>
758 <p><b>char *toys.optargs[]</b></p>
760 <p>Command line arguments in argv[] which are not consumed by option parsing
761 (I.E. not recognized either as -flags or arguments to -flags) will be copied
762 to toys.optargs[], with the length of that array in toys.optc.
763 (When toys.optc is 0, no unrecognized command line arguments remain.)
764 The order of entries is preserved, and as with argv[] this new array is also
765 terminated by a NULL entry.</p>
767 <p>Option parsing can require a minimum or maximum number of optargs left
768 over, by adding "<1" (read "at least one") or ">9" ("at most nine") to the
769 start of the optflags string.</p>
771 <p>The special argument "--" terminates option parsing, storing all remaining
772 arguments in optargs. The "--" itself is consumed.</p>
774 <p><b>Other optflags control characters</b></p>
776 <p>The following characters may occur at the start of each command's
777 optflags string, before any options that would set a bit in toys.optflags:</p>
780 <li><b>^</b> - stop at first nonoption argument (for nice, xargs...)</li>
781 <li><b>?</b> - allow unknown arguments (pass non-option arguments starting
782 with - through to optargs instead of erroring out).</li>
783 <li><b>&</b> - the first argument has imaginary dash (ala tar/ps. If given twice, all arguments have imaginary dash.)</li>
784 <li><b><</b> - must be followed by a decimal digit indicating at least this many leftover arguments are needed in optargs (default 0)</li>
785 <li><b>></b> - must be followed by a decimal digit indicating at most this many leftover arguments allowed (default MAX_INT)</li>
788 <p>The following characters may be appended to an option character, but do
789 not by themselves indicate an extra argument should be saved in this[].
790 (Technically any character not recognized as a control character sets an
791 optflag, but letters are never control characters.)</p>
794 <li><b>^</b> - stop parsing options after encountering this option, everything else goes into optargs.</li>
795 <li><b>|</b> - this option is required. If more than one marked, only one is required.</li>
796 <li><b>+X</b> enabling this option also enables option X (switch bit on).</li>
797 <li><b>~X</b> enabling this option disables option X (switch bit off).</li>
798 <li><b>!X</b> this option cannot be used in combination with X (die with error).</li>
799 <li><b>[yz]</b> this option requires at least one of y or z to also be enabled.</li>
802 <p>The following may be appended to a float or double:</p>
805 <li><b><123</b> - error if argument is less than this</li>
806 <li><b>>123</b> - error if argument is greater than this</li>
807 <li><b>=123</b> - default value if argument not supplied</li>
810 <p>Option parsing only understands <>= after . when CFG_TOYBOX_FLOAT
811 is enabled. (Otherwise the code to determine where floating point constants
812 end drops out. When disabled, it can reserve a global data slot for the
813 argument so offsets won't change, but will never fill it out.). You can handle
814 this by using the USE_BLAH() macros with C string concatenation, ala:</p>
816 <blockquote>"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"</blockquote>
818 <p><b>--longopts</b></p>
820 <p>The optflags string can contain long options, which are enclosed in
821 parentheses. They may be appended to an existing option character, in
822 which case the --longopt is a synonym for that option, ala "a:(--fred)"
823 which understands "-a blah" or "--fred blah" as synonyms.</p>
825 <p>Longopts may also appear before any other options in the optflags string,
826 in which case they have no corresponding short argument, but instead set
827 their own bit based on position. So for "(walrus)#(blah)xy:z" "command
828 --walrus 42" would set toys.optflags = 16 (-z = 1, -y = 2, -x = 4, --blah = 8)
829 and would assign this[1] = 42;</p>
831 <p>A short option may have multiple longopt synonyms, "a(one)(two)", but
832 each "bare longopt" (ala "(one)(two)abc" before any option characters)
833 always sets its own bit (although you can group them with +X).</p>
835 <a name="lib_dirtree"><h3>lib/dirtree.c</h3>
837 <p>The directory tree traversal code should be sufficiently generic
838 that commands never need to use readdir(), scandir(), or the fts.h family
841 <p>These functions do not call chdir() or rely on PATH_MAX. Instead they
842 use openat() and friends, using one filehandle per directory level to
843 recurseinto subdirectories. (I.E. they can descend 1000 directories deep
844 if setrlimit(RLIMIT_NOFILE) allows enough open filehandles, and the default
845 in /proc/self/limits is generally 1024.)</p>
847 <p>The basic dirtree functions are:</p>
850 <li><p><b>dirtree_read(char *path, int (*callback)(struct dirtree node))</b> -
851 recursively read directories, either applying callback() or returning
852 a tree of struct dirtree if callback is NULL.</p></li>
854 <li><p><b>dirtree_path(struct dirtree *node, int *plen)</b> - malloc() a
855 string containing the path from the root of this tree to this node. If
856 plen isn't NULL then *plen is how many extra bytes to malloc at the end
859 <li><p><b>dirtree_parentfd(struct dirtree *node)</b> - return fd of
860 containing directory, for use with openat() and such.</p></li>
863 <p>The <b>dirtree_read()</b> function takes two arguments, a starting path for
864 the root of the tree, and a callback function. The callback takes a
865 <b>struct dirtree *</b> (from lib/lib.h) as its argument. If the callback is
866 NULL, the traversal uses a default callback (dirtree_notdotdot()) which
867 recursively assembles a tree of struct dirtree nodes for all files under
868 this directory and subdirectories (filtering out "." and ".." entries),
869 after which dirtree_read() returns the pointer to the root node of this
872 <p>Otherwise the callback() is called on each entry in the directory,
873 with struct dirtree * as its argument. This includes the initial
874 node created by dirtree_read() at the top of the tree.</p>
876 <p><b>struct dirtree</b></p>
878 <p>Each struct dirtree node contains <b>char name[]</b> and <b>struct stat
879 st</b> entries describing a file, plus a <b>char *symlink</b>
880 which is NULL for non-symlinks.</p>
882 <p>During a callback function, the <b>int data</b> field of directory nodes
883 contains a dirfd (for use with the openat() family of functions). This is
884 generally used by calling dirtree_parentfd() on the callback's node argument.
885 For symlinks, data contains the length of the symlink string. On the second
886 callback from DIRTREE_COMEAGAIN (depth-first traversal) data = -1 for
887 all nodes (that's how you can tell it's the second callback).</p>
889 <p>Users of this code may put anything they like into the <b>long extra</b>
890 field. For example, "cp" and "mv" use this to store a dirfd for the destination
891 directory (and use DIRTREE_COMEAGAIN to get the second callback so they can
892 close(node->extra) to avoid running out of filehandles).
893 This field is not directly used by the dirtree code, and
894 thanks to LP64 it's large enough to store a typecast pointer to an
895 arbitrary struct.</p>
897 <p>The return value of the callback combines flags (with boolean or) to tell
898 the traversal infrastructure how to behave:</p>
901 <li><p><b>DIRTREE_SAVE</b> - Save this node, assembling a tree. (Without
902 this the struct dirtree is freed after the callback returns. Filtering out
903 siblings is fine, but discarding a parent while keeping its child leaks
905 <li><p><b>DIRTREE_ABORT</b> - Do not examine any more entries in this
906 directory. (Does not propagate up tree: to abort entire traversal,
907 return DIRTREE_ABORT from parent callbacks too.)</p></li>
908 <li><p><b>DIRTREE_RECURSE</b> - Examine directory contents. Ignored for
909 non-directory entries. The remaining flags only take effect when
910 recursing into the children of a directory.</p></li>
911 <li><p><b>DIRTREE_COMEAGAIN</b> - Call the callback a second time after
912 examining all directory contents, allowing depth-first traversal.
913 On the second call, dirtree->data = -1.</p></li>
914 <li><p><b>DIRTREE_SYMFOLLOW</b> - follow symlinks when populating children's
915 <b>struct stat st</b> (by feeding a nonzero value to the symfollow argument of
916 dirtree_add_node()), which means DIRTREE_RECURSE treats symlinks to
917 directories as directories. (Avoiding infinite recursion is the callback's
918 problem: the non-NULL dirtree->symlink can still distinguish between
922 <p>Each struct dirtree contains three pointers (next, parent, and child)
923 to other struct dirtree.</p>
925 <p>The <b>parent</b> pointer indicates the directory
926 containing this entry; even when not assembling a persistent tree of
927 nodes the parent entries remain live up to the root of the tree while
928 child nodes are active. At the top of the tree the parent pointer is
929 NULL, meaning the node's name[] is either an absolute path or relative
930 to cwd. The function dirtree_parentfd() gets the directory file descriptor
931 for use with openat() and friends, returning AT_FDCWD at the top of tree.</p>
933 <p>The <b>child</b> pointer points to the first node of the list of contents of
934 this directory. If the directory contains no files, or the entry isn't
935 a directory, child is NULL.</p>
937 <p>The <b>next</b> pointer indicates sibling nodes in the same directory as this
938 node, and since it's the first entry in the struct the llist.c traversal
939 mechanisms work to iterate over sibling nodes. Each dirtree node is a
940 single malloc() (even char *symlink points to memory at the end of the node),
941 so llist_free() works but its callback must descend into child nodes (freeing
942 a tree, not just a linked list), plus whatever the user stored in extra.</p>
944 <p>The <b>dirtree_read</b>() function is a simple wrapper, calling <b>dirtree_add_node</b>()
945 to create a root node relative to the current directory, then calling
946 <b>handle_callback</b>() on that node (which recurses as instructed by the callback
947 return flags). Some commands (such as chgrp) bypass this wrapper, for example
948 to control whether or not to follow symlinks to the root node; symlinks
949 listed on the command line are often treated differently than symlinks
950 encountered during recursive directory traversal).
952 <p>The ls command not only bypasses the wrapper, but never returns
953 <b>DIRTREE_RECURSE</b> from the callback, instead calling <b>dirtree_recurse</b>() manually
954 from elsewhere in the program. This gives ls -lR manual control
955 of traversal order, which is neither depth first nor breadth first but
956 instead a sort of FIFO order requried by the ls standard.</p>
959 <h2>Directory toys/</h2>
961 <p>This directory contains command implementations. Each command is a single
962 self-contained file. Adding a new command involves adding a single
963 file, and removing a command involves removing that file. Commands use
964 shared infrastructure from the lib/ and generated/ directories.</p>
966 <p>Currently there are three subdirectories under "toys/" containing commands
967 described in POSIX-2008, the Linux Standard Base 4.1, or "other". The only
968 difference this makes is which menu the command shows up in during "make
969 menuconfig", the directories are otherwise identical. Note that they commands
970 exist within a single namespace at runtime, so you can't have the same
971 command in multiple subdirectories.</p>
973 <p>(There are actually four sub-menus in "make menuconfig", the fourth
974 contains global configuration options for toybox, and lives in Config.in at
977 <p>See <a href="#adding">adding a new command</a> for details on the
978 layout of a command file.</p>
980 <h2>Directory scripts/</h2>
982 <p>Build infrastructure. The makefile calls scripts/make.sh for "make"
983 and scripts/install.sh for "make install".</p>
985 <p>There's also a test suite, "make test" calls make/test.sh, which runs all
986 the tests in make/test/*. You can run individual tests via
987 "scripts/test.sh command", or "TEST_HOST=1 scripts/test.sh command" to run
988 that test against the host implementation instead of the toybox one.</p>
990 <h3>scripts/cfg2files.sh</h3>
992 <p>Run .config through this filter to get a list of enabled commands, which
993 is turned into a list of files in toys via a sed invocation in the top level
997 <h2>Directory kconfig/</h2>
999 <p>Menuconfig infrastructure copied from the Linux kernel. See the
1000 Linux kernel's Documentation/kbuild/kconfig-language.txt</p>
1002 <a name="generated">
1003 <h2>Directory generated/</h2>
1005 <p>All the files in this directory except the README are generated by the
1006 build. (See scripts/make.sh)</p>
1009 <li><p><b>config.h</b> - CFG_COMMAND and USE_COMMAND() macros set by menuconfig via .config.</p></li>
1011 <li><p><b>Config.in</b> - Kconfig entries for each command. Included by top level Config.in. The help text in here is used to generated help.h</p></li>
1013 <li><p><b>help.h</b> - Help text strings for use by "help" command. Building
1014 this file requires python on the host system, so the prebuilt file is shipped
1015 in the build tarball to avoid requiring python to build toybox.</p></li>
1017 <li><p><b>newtoys.h</b> - List of NEWTOY() or OLDTOY() macros for all available
1018 commands. Associates command_main() functions with command names, provides
1019 option string for command line parsing (<a href="#lib_args">see lib/args.c</a>),
1020 specifies where to install each command and whether toysh should fork before
1021 calling it.</p></li>
1024 <p>Everything in this directory is a derivative file produced from something
1025 else. The entire directory is deleted by "make distclean".</p>
1026 <!--#include file="footer.html" -->