7 This tool aims to test U-Boot by executing U-Boot shell commands using the
8 console interface. A single top-level script exists to execute or attach to the
9 U-Boot console, run the entire script of tests against it, and summarize the
10 results. Advantages of this approach are:
12 - Testing is performed in the same way a user or script would interact with
13 U-Boot; there can be no disconnect.
14 - There is no need to write or embed test-related code into U-Boot itself.
15 It is asserted that writing test-related code in Python is simpler and more
16 flexible than writing it all in C. But see :doc:`tests_writing` for caveats
17 and more discussion / analysis.
18 - It is reasonably simple to interact with U-Boot in this way.
23 The test suite is implemented using pytest. Interaction with the U-Boot console
24 involves executing some binary and interacting with its stdin/stdout. You will
25 need to implement various "hook" scripts that are called by the test suite at
28 In order to run the test suite at a minimum we require that both Python 3 and
29 pip for Python 3 are installed. All of the required python modules are
30 described in the requirements.txt file in the /test/py/ directory and can be
31 installed via the command
35 pip install -r requirements.txt
37 In order to execute certain tests on their supported platforms other tools
38 will be required. The following is an incomplete list:
55 Please use the appropriate commands for your distribution to match these tools
56 up with the package that provides them.
58 The test script supports either:
60 - Executing a sandbox port of U-Boot on the local machine as a sub-process,
61 and interacting with it over stdin/stdout.
62 - Executing an external "hook" scripts to flash a U-Boot binary onto a
63 physical board, attach to the board's console stream, and reset the board.
64 Further details are described later.
66 Using `virtualenv` to provide requirements
67 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
69 The recommended way to run the test suite, in order to ensure reproducibility
70 is to use `virtualenv` to set up the necessary environment. This can be done
71 via the following commands:
74 .. code-block:: console
77 $ sudo apt-get install python3 python3-virtualenv
78 $ virtualenv -p /usr/bin/python3 venv
79 $ . ./venv/bin/activate
80 $ pip install -r test/py/requirements.txt
85 To run the test suite on the sandbox port (U-Boot built as a native user-space
86 application), simply execute:
90 ./test/py/test.py --bd sandbox --build
92 The `--bd` option tells the test suite which board type is being tested. This
93 lets the test suite know which features the board has, and hence exactly what
96 The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
97 omit this option and build U-Boot yourself, in whatever way you choose, before
98 running the test script.
100 The test script will attach to U-Boot, execute all valid tests for the board,
101 then print a summary of the test process. A complete log of the test session
102 will be written to `${build_dir}/test-log.html`. This is best viewed in a web
103 browser, but may be read directly as plain text, perhaps with the aid of the
106 If sandbox crashes (e.g. with a segfault) you will see message like this::
109 test/py/u_boot_spawn.py:171: in expect
110 c = os.read(self.fd, 1024).decode(errors='replace')
111 E ValueError: U-Boot exited with signal 11 (Signals.SIGSEGV)
117 By default a short backtrace is reported. If you would like a longer one,
118 pass ``--tb=long`` when running the test. See the pytest documentation for
121 Running tests in parallel
122 ~~~~~~~~~~~~~~~~~~~~~~~~~
124 Note: This does not fully work yet and is documented only so you can try to
127 First install support for parallel tests::
129 pip3 install pytest-xdist
131 Then build sandbox in a suitable build directory. It is not possible to use
132 the --build flag with xdist.
134 Finally, run the tests in parallel using the -n flag::
136 # build sandbox first, in a suitable build directory. It is not possible
137 # to use the --build flag with -n
138 test/py/test.py -B sandbox --build-dir /tmp/b/sandbox -q -k 'not slow' -n32
140 At least the following non-slow tests are known to fail:
143 - test_bind_unbind_with_uclass
145 - test_gpt_rename_partition
146 - test_gpt_swap_partitions
151 Testing under a debugger
152 ~~~~~~~~~~~~~~~~~~~~~~~~
154 If you need to run sandbox under a debugger, you may pass the command-line
155 option `--gdbserver COMM`. This causes two things to happens:
157 - Instead of running U-Boot directly, it will be run under gdbserver, with
158 debug communication via the channel `COMM`. You can attach a debugger to the
159 sandbox process in order to debug it. See `man gdbserver` and the example
160 below for details of valid values for `COMM`.
161 - All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
162 time to execute commands. This is useful if U-Boot is stopped at a breakpoint
171 ./test/py/test.py --bd sandbox --gdbserver localhost:1234
177 gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
179 Alternatively, you could leave off the `-ex` option and type the command
180 manually into gdb once it starts.
182 You can use any debugger you wish, as long as it speaks the gdb remote
183 protocol, or any graphical wrapper around gdb.
185 Some tests deliberately cause the sandbox process to exit, e.g. to test the
186 reset command, or sandbox's CTRL-C handling. When this happens, you will need
187 to attach the debugger to the new sandbox instance. If these tests are not
188 relevant to your debugging session, you can skip them using pytest's -k
189 command-line option; see the next section.
194 --board-type, --bd, -B
195 set the type of the board to be tested. For example, `sandbox` or `seaboard`.
197 --board-identity`, --id
198 sets the identity of the board to be tested. This allows differentiation
199 between multiple instances of the same type of physical board that are
200 attached to the same host machine. This parameter is not interpreted by th
201 test script in any way, but rather is simply passed to the hook scripts
202 described below, and may be used in any site-specific way deemed necessary.
205 indicates that the test script should compile U-Boot itself before running
206 the tests. If using this option, make sure that any environment variables
207 required by the build process are already set, such as `$CROSS_COMPILE`.
210 indicates that `--build` should use buildman to build U-Boot. There is no need
211 to set $CROSS_COMPILE` in this case since buildman handles it.
214 sets the directory containing the compiled U-Boot binaries. If omitted, this
215 is `${source_dir}/build-${board_type}`.
218 sets the directory to write results, such as log files, into.
219 If omitted, the build directory is used.
221 --persistent-data-dir
222 sets the directory used to store persistent test data. This is test data that
223 may be re-used across test runs, such as file-system images.
225 `pytest` also implements a number of its own command-line options. Commonly used
226 options are mentioned below. Please see `pytest` documentation for complete
227 details. Execute `py.test --version` for a brief summary. Note that U-Boot's
228 test.py script passes all command-line arguments directly to `pytest` for
232 selects which tests to run. The default is to run all known tests. This
233 option takes a single argument which is used to filter test names. Simple
234 logical operators are supported. For example:
236 - `'-k ums'` runs only tests with "ums" in their name.
237 - `'-k ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
238 case, "ut_dm" is a parameter to a test rather than the test name. The full
239 test name is e.g. "test_ut[ut_dm_leak]".
240 - `'-k not reset'` runs everything except tests with "reset" in their name.
241 - `'-k ut or hush'` runs only tests with "ut" or "hush" in their name.
242 - `'-k not (ut or hush)'` runs everything except tests with "ut" or "hush" in
246 prevents pytest from hiding a test's stdout. This allows you to see
247 U-Boot's console log in real time on pytest's stdout.
249 Testing real hardware
250 ---------------------
252 The tools and techniques used to interact with real hardware will vary
253 radically between different host and target systems, and the whims of the user.
254 For this reason, the test suite does not attempt to directly interact with real
255 hardware in any way. Rather, it executes a standardized set of "hook" scripts
256 via `$PATH`. These scripts implement certain actions on behalf of the test
257 suite. This keeps the test suite simple and isolated from system variances
258 unrelated to U-Boot features.
263 Environment variables
264 '''''''''''''''''''''
266 The following environment variables are set when running hook scripts:
268 - `UBOOT_BOARD_TYPE` the board type being tested.
269 - `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
271 - `UBOOT_SOURCE_DIR` the U-Boot source directory.
272 - `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
273 - `UBOOT_BUILD_DIR` the U-Boot build directory.
274 - `UBOOT_RESULT_DIR` the test result directory.
275 - `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory.
280 This script provides access to the U-Boot console. The script's stdin/stdout
281 should be connected to the board's console. This process should continue to run
282 indefinitely, until killed. The test suite will run this script in parallel
283 with all other hooks.
285 This script may be implemented e.g. by executing `cu`, `kermit`, `conmux`, etc.
288 If you are able to run U-Boot under a hardware simulator such as QEMU, then
289 you would likely spawn that simulator from this script. However, note that
290 `u-boot-test-reset` may be called multiple times per test script run, and must
291 cause U-Boot to start execution from scratch each time. Hopefully your
292 simulator includes a virtual reset button! If not, you can launch the
293 simulator from `u-boot-test-reset` instead, while arranging for this console
294 process to always communicate with the current simulator instance.
299 Prior to running the test suite against a board, some arrangement must be made
300 so that the board executes the particular U-Boot binary to be tested. Often
301 this involves writing the U-Boot binary to the board's flash ROM. The test
302 suite calls this hook script for that purpose.
304 This script should perform the entire flashing process synchronously; the
305 script should only exit once flashing is complete, and a board reset will
306 cause the newly flashed U-Boot binary to be executed.
308 It is conceivable that this script will do nothing. This might be useful in
311 - Some other process has already written the desired U-Boot binary into the
312 board's flash prior to running the test suite.
313 - The board allows U-Boot to be downloaded directly into RAM, and executed
314 from there. Use of this feature will reduce wear on the board's flash, so
315 may be preferable if available, and if cold boot testing of U-Boot is not
316 required. If this feature is used, the `u-boot-test-reset` script should
317 perform this download, since the board could conceivably be reset multiple
318 times in a single test run.
320 It is up to the user to determine if those situations exist, and to code this
321 hook script appropriately.
323 This script will typically be implemented by calling out to some SoC- or
324 board-specific vendor flashing utility.
329 Whenever the test suite needs to reset the target board, this script is
330 executed. This is guaranteed to happen at least once, prior to executing the
331 first test function. If any test fails, the test infra-structure will execute
332 this script again to restore U-Boot to an operational state before running the
335 This script will likely be implemented by communicating with some form of
336 relay or electronic switch attached to the board's reset signal.
338 The semantics of this script require that when it is executed, U-Boot will
339 start running from scratch. If the U-Boot binary to be tested has been written
340 to flash, pulsing the board's reset signal is likely all this script needs to
341 do. However, in some scenarios, this script may perform other actions. For
342 example, it may call out to some SoC- or board-specific vendor utility in order
343 to download the U-Boot binary directly into RAM and execute it. This would
344 avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
345 saving wear on the flash chip(s).
350 https://source.denx.de/u-boot/u-boot-test-hooks contains some working example hook
351 scripts, and may be useful as a reference when implementing hook scripts for
352 your platform. These scripts are not considered part of U-Boot itself.
354 Board-type-specific configuration
355 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
357 Each board has a different configuration and behaviour. Many of these
358 differences can be automatically detected by parsing the `.config` file in the
359 build directory. However, some differences can't yet be handled automatically.
361 For each board, an optional Python module `u_boot_board_${board_type}` may exist
362 to provide board-specific information to the test script. Any global value
363 defined in these modules is available for use by any test function. The data
364 contained in these scripts must be purely derived from U-Boot source code.
365 Hence, these configuration files are part of the U-Boot source tree too.
367 Execution environment configuration
368 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
370 Each user's hardware setup may enable testing different subsets of the features
371 implemented by a particular board's configuration of U-Boot. For example, a
372 U-Boot configuration may support USB device mode and USB Mass Storage, but this
373 can only be tested if a USB cable is connected between the board and the host
374 machine running the test script.
376 For each board, optional Python modules `u_boot_boardenv_${board_type}` and
377 `u_boot_boardenv_${board_type}_${board_identity}` may exist to provide
378 board-specific and board-identity-specific information to the test script. Any
379 global value defined in these modules is available for use by any test
380 function. The data contained in these is specific to a particular user's
381 hardware configuration. Hence, these configuration files are not part of the
382 U-Boot source tree, and should be installed outside of the source tree. Users
383 should set `$PYTHONPATH` prior to running the test script to allow these
384 modules to be loaded.
386 Board module parameter usage
387 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
389 The test scripts rely on the following variables being defined by the board
394 U-Boot `.config` feature usage
395 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
397 The test scripts rely on various U-Boot `.config` features, either directly in
398 order to test those features, or indirectly in order to query information from
399 the running U-Boot instance in order to test other features.
401 One example is that testing of the `md` command requires knowledge of a RAM
402 address to use for the test. This data is parsed from the output of the
403 `bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled.
405 For a complete list of dependencies, please search the test scripts for
408 - `buildconfig.get(...`
409 - `@pytest.mark.buildconfigspec(...`
410 - `@pytest.mark.notbuildconfigspec(...`
412 Complete invocation example
413 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
415 Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
416 any required environment configuration Python modules into $HOME/ubtest/py,
417 then you would likely invoke the test script as follows:
419 If U-Boot has already been built:
423 PATH=$HOME/ubtest/bin:$PATH \
424 PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
425 ./test/py/test.py --bd seaboard
427 If you want the test script to compile U-Boot for you too, then you likely
428 need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
433 CROSS_COMPILE=arm-none-eabi- \
434 PATH=$HOME/ubtest/bin:$PATH \
435 PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
436 ./test/py/test.py --bd seaboard --build
438 or, using buildman to handle it:
442 PATH=$HOME/ubtest/bin:$PATH \
443 PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
444 ./test/py/test.py --bd seaboard --build --buildman
449 Please refer to the pytest documentation for details of writing pytest tests.
450 Details specific to the U-Boot test suite are described below.
452 A test fixture named `u_boot_console` should be used by each test function. This
453 provides the means to interact with the U-Boot console, and retrieve board and
454 environment configuration information.
456 The function `u_boot_console.run_command()` executes a shell command on the
457 U-Boot console, and returns all output from that command. This allows
458 validation or interpretation of the command output. This function validates
459 that certain strings are not seen on the U-Boot console. These include shell
460 error messages and the U-Boot sign-on message (in order to detect unexpected
461 board resets). See the source of `u_boot_console_base.py` for a complete list of
462 "bad" strings. Some test scenarios are expected to trigger these strings. Use
463 `u_boot_console.disable_check()` to temporarily disable checking for specific
464 strings. See `test_unknown_cmd.py` for an example.
466 Board- and board-environment configuration values may be accessed as sub-fields
467 of the `u_boot_console.config` object, for example
468 `u_boot_console.config.ram_base`.
470 Build configuration values (from `.config`) may be accessed via the dictionary
471 `u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable