1 README for security-tests project
3 ==WHAT IS======================================================================
5 security-tests is repository for testing packages from domain Security.
7 ==WHAT FOR=====================================================================
9 The security-tests repository is designed for testing packages mentioned below
10 with binaries provided for testing them:
13 security-manager-tests
19 There are also inner-tests for testing complex security-tests framework
20 mechanisms with binary:
21 security-tests-inner-test
23 ==HOW TO RUN===================================================================
25 Each test suite may be run with options:
26 --output=<output type> --output=<output type> ...
29 test-binary --output=text --output=xml --file=output.xml
30 --only-from-xml=<xml file> Run only testcases specified in XML file
31 --regexp='regexp' Only selected tests which names match regexp run
32 --start=<test id> Start from concrete test id
33 --group=<group name> Run tests only from one group
34 --runignored Run also ignored tests
35 --list Show a list of Test IDs
36 --listgroups Show a list of Test Group names
37 --only-from-xml=<xml file> Run only testcases specified in XML file
38 XML name is taken from attribute id="part1_part2" as whole.
39 If part1 is not found (no _) then it is implicitily set according to
40 suite part1 from binary tests
41 --listingroup=<group name> Show a list of Test IDS in one group
42 --allowchildlogs Allow to print logs from child process on screen.
43 When active child process will be able to print logs on stdout and
44 stderr. Both descriptors will be closed after test.
47 They can be run also by scripts:
51 Each test can end with one of three possible statuses:
56 ==HOW TO WRITE=================================================================
58 security-tests is based on extended dpl framework providing different macros.
59 Below are categories with macros listed as below:
65 --Test group registering macro-------------------------------------------------
69 RUNNER_TEST_GROUP_INIT
70 Registers group of tests. Test are registered under this group until
71 another group registering macro is called.
73 --Test registering macros------------------------------------------------------
74 Adding/removing those macro calls will add/remove test cases they provide. To
75 change tests, change body of those macro calls. Registered tests are run within
77 Those macros allow additional arguments which are classes with mandatory
80 Called while registering test.
81 Should not throw any exceptions
82 * init(const std::string &testName)
83 Called before test case function in order of classes passed to macro.
85 testName argument is name of the test (first macro argument).
87 called after test case function in reversed order of classes passed to
90 Created instances of those classes may be accessed from within test case body
91 as argument of test case funtion is
92 std::tuple<ClassesPassed> &optionalArgsTuple
97 Function registered by this macro will be run in the same process as
98 framework. No forking allowed unless forked process does not throw
102 Function registered by this macro will be run in child process. No
103 forking allowed unless forked process does not throw any exception.
104 test_runner_multiprocess.h
105 RUNNER_MULTIPROCESS_TEST
106 Function registered by this macro will be run in the same process as
107 framework. Forking allowed. Exception of every process will be
112 Same as RUNNER_TEST but run only with smack enabled.
114 Same as RUNNER_TEST but run only with smack disabled.
115 RUNNER_CHILD_TEST_SMACK
116 Same as RUNNER_TEST_CHILD but run only with smack enabled.
117 RUNNER_CHILD_TEST_NOSMACK
118 Same as RUNNER_TEST_CHILD but run only with smack disabled.
119 RUNNER_MULTIPROCESS_TEST_SMACK
120 Same as RUNNER_TEST_MULTIPROCESS but run only with smack enabled.
122 --Assert macros----------------------------------------------------------------
123 Used within test registering macros.
125 First failed assertion throws test failed exception. If another assertions
126 fail, information about fail conditions and backtrace is cumulated and
127 presented together with already thrown exception message.
132 Assertion with message with backtrace information appended.
133 RUNNER_ASSERT_ERRNO_MSG
134 Assertion with message, error string and backtrace information
137 Assertion with error string and backtrace information appended.
139 Fail with message and backtrace information appended.
141 Assertion with backtrace information appended.
143 Assertion with message classified as ignored.
145 --Performence macros-----------------------------------------------------------
146 Used to do the time measurement.
150 RUNNER_PERF_TEST_BEGIN
151 Start time measurement.
153 End time measurement.
155 --Message macros---------------------------------------------------------------
156 Used to print error messages during test run time.
161 Print error message using red color.
163 --Defer macros-----------------------------------------------------------------
164 Used to defer throwing TestException exceptions (TestFailed, TestIgnored)
165 by catching them and rethrowing later. This mechanism can help in breaking
166 test and passing test result from places where throwing exceptions
171 RUNNER_DEFER_TRYCATCH
172 Catches thrown TestException exceptions and stores them in TestRunner
173 structures for later use. This macro works only inside deffered scope
174 defined by RUNNER_DEFER_SCOPE, otherwise it won't catch exceptions
176 Defines deferred scope. All RUNNER_DEFER_TRYCATCH macros used inside
177 the scope catch and save TestException exceptions. After scope is left
178 all saved exceptions take part in setting result of test. If there
179 is no any uncaught exception then additionally first of saved
180 exceptions is thrown.
182 --Collectors-------------------------------------------------------------------
183 Collectors are classes which collect test results. Each class does it differently.
184 Collectors can be registered by --output parameter (see HOW TO RUN section) but
185 there is also another collector created to write summary.
188 test_results_collector_summary.h
190 Collector writing tests summary. Call SummaryCollector::Register() to
193 --Usage example----------------------------------------------------------------
195 #include <test_runner.h>
196 #include <tests_common.h>
197 #include <summary_collector.h>
199 #include <sys/stat.h>
203 RUNNER_TEST_GROUP_INIT(foo_module)
205 RUNNER_TEST_SMACK(bar_allways_fails)
207 RUNNER_ASSERT(false);
210 RUNNER_TEST(bar_allways_passses)
215 RUNNER_TEST(bar_file1)
217 cosnt char *file = "bar_file1";
218 int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
219 RUNNER_ASSERT_ERRNO_MSG(fd != -1, "Cannot open " << file << " file");
223 RUNNER_CHILD_TEST_NOSMACK(bar_file2_dropped_root)
225 RUNNER_ASSERT_ERRNO(setgid(5000) == 0);
226 RUNNER_ASSERT_ERRNO(setuid(5000) == 0);
228 cosnt char *file = "bar_file2";
229 int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
232 RUNNER_FAIL_MSG("file " << file << "should not be opened");
234 RUNNER_ASSERT_ERRNO_MSG(errno == EACCESS,
235 "Wrong errno on opening " << " file");
242 void init(const std::string &testName) { ... }
243 void finish() { ... }
251 void init(const std::string &testName) { ... }
252 void finish() { ... }
253 void doRestore() { ... }
256 RUNNER_TEST(bar_optional_args, Env, Restore)
258 std::get<0>(optionalArgsTuple).doEnv();
259 std::get<1>(optionalArgsTuple).doRestore();
262 int main(int argc, char *argv[])
264 SummaryCollector::Register();
265 return DPL::Test::TestRunnerSingleton::Instance().ExecTestRunner(argc, argv);
268 --Notes------------------------------------------------------------------------
270 While changing body of test cases, be sure to remove functions and global
271 variables if not used by any other tests.
272 Use const variables instead of #define's.
273 Use mechanisms already provided in common library.