1 README for security-tests project
3 ==WHAT IS======================================================================
5 security-tests is repository for testing packages from domain Security.
7 ==WHAT FOR=====================================================================
9 The security-tests repository is designed for testing packages mentioned below
10 with binaries provided for testing them:
15 security-manager-tests
19 There are also inner-tests for testing complex security-tests framework
20 mechanisms with binary:
21 security-tests-inner-test
23 ==HOW TO RUN===================================================================
25 Each test suite may be run with options:
26 --output=<output type> --output=<output type> ...
29 test-binary --output=text --output=xml --file=output.xml
30 --only-from-xml=<xml file> Run only testcases specified in XML file
31 --regexp='regexp' Only selected tests which names match regexp run
32 --start=<test id> Start from concrete test id
33 --group=<group name> Run tests only from one group
34 --runignored Run also ignored tests
35 --list Show a list of Test IDs
36 --listgroups Show a list of Test Group names
37 --only-from-xml=<xml file> Run only testcases specified in XML file
38 XML name is taken from attribute id="part1_part2" as whole.
39 If part1 is not found (no _) then it is implicitily set according to
40 suite part1 from binary tests
41 --listingroup=<group name> Show a list of Test IDS in one group
42 --allowchildlogs Allow to print logs from child process on screen.
43 When active child process will be able to print logs on stdout and
44 stderr. Both descriptors will be closed after test.
47 They can be run also by scripts:
51 Each test can end with one of three possible statuses:
56 ==HOW TO WRITE=================================================================
58 security-tests is based on extended dpl framework providing different macros.
59 Below are categories with macros listed as below:
65 --Test group registering macro-------------------------------------------------
69 RUNNER_TEST_GROUP_INIT
70 Registers group of tests. Test are registered under this group until
71 another group registering macro is called.
73 --Test registering macros------------------------------------------------------
74 Adding/removing those macro calls will add/remove test cases they provide. To
75 change tests, change body of those macro calls. Registered tests are run within
77 Those macros allow additional arguments which are classes with mandatory
80 Called while registering test.
81 Should not throw any exceptions
82 * init(const std::string &testName)
83 Called before test case function in order of classes passed to macro.
85 testName argument is name of the test (first macro argument).
87 called after test case function in reversed order of classes passed to
90 Created instances of those classes may be accessed from within test case body
91 as argument of test case funtion is
92 std::tuple<ClassesPassed> &optionalArgsTuple
97 Function registered by this macro will be run in the same process as
98 framework. No forking allowed unless forked process does not throw
102 Function registered by this macro will be run in child process. No
103 forking allowed unless forked process does not throw any exception.
104 test_runner_multiprocess.h
105 RUNNER_MULTIPROCESS_TEST
106 Function registered by this macro will be run in the same process as
107 framework. Forking allowed. Exception of every process will be
112 Same as RUNNER_TEST but run only with smack enabled.
114 Same as RUNNER_TEST but run only with smack disabled.
115 RUNNER_CHILD_TEST_SMACK
116 Same as RUNNER_TEST_CHILD but run only with smack enabled.
117 RUNNER_CHILD_TEST_NOSMACK
118 Same as RUNNER_TEST_CHILD but run only with smack disabled.
119 RUNNER_MULTIPROCESS_TEST_SMACK
120 Same as RUNNER_TEST_MULTIPROCESS but run only with smack enabled.
121 RUNNER_MULTIPROCESS_TEST_NOSMACK
122 Same as RUNNER_TEST_MULTIPROCESS but run only with smack disabled.
124 --Assert macros----------------------------------------------------------------
125 Used within test registering macros.
127 First failed assertion throws test failed exception. If another assertions
128 fail, information about fail conditions and backtrace is cumulated and
129 presented together with already thrown exception message.
134 Assertion with message with backtrace information appended.
135 RUNNER_ASSERT_ERRNO_MSG
136 Assertion with message, error string and backtrace information
139 Assertion with error string and backtrace information appended.
141 Fail with message and backtrace information appended.
143 Assertion with backtrace information appended.
145 Assertion with message classified as ignored.
147 --Performence macros-----------------------------------------------------------
148 Used to do the time measurement.
152 RUNNER_PERF_TEST_BEGIN
153 Start time measurement.
155 End time measurement.
157 --Message macros---------------------------------------------------------------
158 Used to print error messages during test run time.
163 Print error message using red color.
165 --Defer macros-----------------------------------------------------------------
166 Used to defer throwing TestException exceptions (TestFailed, TestIgnored)
167 by catching them and rethrowing later. This mechanism can help in breaking
168 test and passing test result from places where throwing exceptions
173 RUNNER_DEFER_TRYCATCH
174 Catches thrown TestException exceptions and stores them in TestRunner
175 structures for later use. This macro works only inside deffered scope
176 defined by RUNNER_DEFER_SCOPE, otherwise it won't catch exceptions
178 Defines deferred scope. All RUNNER_DEFER_TRYCATCH macros used inside
179 the scope catch and save TestException exceptions. After scope is left
180 all saved exceptions take part in setting result of test. If there
181 is no any uncaught exception then additionally first of saved
182 exceptions is thrown.
184 --Collectors-------------------------------------------------------------------
185 Collectors are classes which collect test results. Each class does it differently.
186 Collectors can be registered by --output parameter (see HOW TO RUN section) but
187 there is also another collector created to write summary.
190 test_results_collector_summary.h
192 Collector writing tests summary. Call SummaryCollector::Register() to
195 --Usage example----------------------------------------------------------------
197 #include <test_runner.h>
198 #include <tests_common.h>
199 #include <summary_collector.h>
201 #include <sys/stat.h>
205 RUNNER_TEST_GROUP_INIT(foo_module)
207 RUNNER_TEST_SMACK(bar_allways_fails)
209 RUNNER_ASSERT(false);
212 RUNNER_TEST(bar_allways_passses)
217 RUNNER_TEST(bar_file1)
219 cosnt char *file = "bar_file1";
220 int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
221 RUNNER_ASSERT_ERRNO_MSG(fd != -1, "Cannot open " << file << " file");
225 RUNNER_CHILD_TEST_NOSMACK(bar_file2_dropped_root)
227 RUNNER_ASSERT_ERRNO(setgid(5000) == 0);
228 RUNNER_ASSERT_ERRNO(setuid(5000) == 0);
230 cosnt char *file = "bar_file2";
231 int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
234 RUNNER_FAIL_MSG("file " << file << "should not be opened");
236 RUNNER_ASSERT_ERRNO_MSG(errno == EACCESS,
237 "Wrong errno on opening " << " file");
244 void init(const std::string &testName) { ... }
245 void finish() { ... }
253 void init(const std::string &testName) { ... }
254 void finish() { ... }
255 void doRestore() { ... }
258 RUNNER_TEST(bar_optional_args, Env, Restore)
260 std::get<0>(optionalArgsTuple).doEnv();
261 std::get<1>(optionalArgsTuple).doRestore();
264 int main(int argc, char *argv[])
266 SummaryCollector::Register();
267 return DPL::Test::TestRunnerSingleton::Instance().ExecTestRunner(argc, argv);
270 --Notes------------------------------------------------------------------------
272 While changing body of test cases, be sure to remove functions and global
273 variables if not used by any other tests.
274 Use const variables instead of #define's.
275 Use mechanisms already provided in common library.