1 README for security-tests project
3 ==WHAT IS======================================================================
5 security-tests is repository for testing packages from domain Security.
7 ==WHAT FOR=====================================================================
9 The security-tests repository is designed for testing packages mentioned below
10 with binaries provided for testing them:
15 libprivilege-control-test
17 security-server-tests-client-smack
18 security-server-tests-stress
19 security-server-tests-server
20 security-server-tests-api-speed
21 security-server-tests-password
22 security-server-tests-privilege
23 security-server-tests-dbus
25 security-manager-tests
29 There are also inner-tests for testing complex security-tests framework
30 mechanisms with binary:
31 security-tests-inner-test
33 ==HOW TO RUN===================================================================
35 Each test suite may be run with options:
36 --output=<output type> --output=<output type> ...
39 test-binary --output=text --output=xml --file=output.xml
40 --only-from-xml=<xml file> Run only testcases specified in XML file
41 --regexp='regexp' Only selected tests which names match regexp run
42 --start=<test id> Start from concrete test id
43 --group=<group name> Run tests only from one group
44 --runignored Run also ignored tests
45 --list Show a list of Test IDs
46 --listgroups Show a list of Test Group names
47 --only-from-xml=<xml file> Run only testcases specified in XML file
48 XML name is taken from attribute id="part1_part2" as whole.
49 If part1 is not found (no _) then it is implicitily set according to
50 suite part1 from binary tests
51 --listingroup=<group name> Show a list of Test IDS in one group
52 --allowchildlogs Allow to print logs from child process on screen.
53 When active child process will be able to print logs on stdout and
54 stderr. Both descriptors will be closed after test.
57 They can be run also by scripts:
61 Each test can end with one of three possible statuses:
66 ==HOW TO WRITE=================================================================
68 security-tests is based on extended dpl framework providing different macros.
69 Below are categories with macros listed as below:
75 --Test group registering macro-------------------------------------------------
79 RUNNER_TEST_GROUP_INIT
80 Registers group of tests. Test are registered under this group until
81 another group registering macro is called.
83 --Test registering macros------------------------------------------------------
84 Adding/removing those macro calls will add/remove test cases they provide. To
85 change tests, change body of those macro calls. Registered tests are run within
87 Those macros allow additional arguments which are classes with mandatory
90 Called while registering test.
91 Should not throw any exceptions
92 * init(const std::string &testName)
93 Called before test case function in order of classes passed to macro.
95 testName argument is name of the test (first macro argument).
97 called after test case function in reversed order of classes passed to
100 Created instances of those classes may be accessed from within test case body
101 as argument of test case funtion is
102 std::tuple<ClassesPassed> &optionalArgsTuple
107 Function registered by this macro will be run in the same process as
108 framework. No forking allowed unless forked process does not throw
112 Function registered by this macro will be run in child process. No
113 forking allowed unless forked process does not throw any exception.
114 test_runner_multiprocess.h
115 RUNNER_MULTIPROCESS_TEST
116 Function registered by this macro will be run in the same process as
117 framework. Forking allowed. Exception of every process will be
122 Same as RUNNER_TEST but run only with smack enabled.
124 Same as RUNNER_TEST but run only with smack disabled.
125 RUNNER_CHILD_TEST_SMACK
126 Same as RUNNER_TEST_CHILD but run only with smack enabled.
127 RUNNER_CHILD_TEST_NOSMACK
128 Same as RUNNER_TEST_CHILD but run only with smack disabled.
129 RUNNER_MULTIPROCESS_TEST_SMACK
130 Same as RUNNER_TEST_MULTIPROCESS but run only with smack enabled.
131 RUNNER_MULTIPROCESS_TEST_NOSMACK
132 Same as RUNNER_TEST_MULTIPROCESS but run only with smack disabled.
134 --Assert macros----------------------------------------------------------------
135 Used within test registering macros.
137 First failed assertion throws test failed exception. If another assertions
138 fail, information about fail conditions and backtrace is cumulated and
139 presented together with already thrown exception message.
144 Assertion with message with backtrace information appended.
145 RUNNER_ASSERT_ERRNO_MSG
146 Assertion with message, error string and backtrace information
149 Assertion with error string and backtrace information appended.
151 Fail with message and backtrace information appended.
153 Assertion with backtrace information appended.
155 Assertion with message classified as ignored.
157 --Performence macros-----------------------------------------------------------
158 Used to do the time measurement.
162 RUNNER_PERF_TEST_BEGIN
163 Start time measurement.
165 End time measurement.
167 --Message macros---------------------------------------------------------------
168 Used to print error messages during test run time.
173 Print error message using red color.
175 --Defer macros-----------------------------------------------------------------
176 Used to defer throwing TestException exceptions (TestFailed, TestIgnored)
177 by catching them and rethrowing later. This mechanism can help in breaking
178 test and passing test result from places where throwing exceptions
183 RUNNER_DEFER_TRYCATCH
184 Catches thrown TestException exceptions and stores them in TestRunner
185 structures for later use. This macro works only inside deffered scope
186 defined by RUNNER_DEFER_SCOPE, otherwise it won't catch exceptions
188 Defines deferred scope. All RUNNER_DEFER_TRYCATCH macros used inside
189 the scope catch and save TestException exceptions. After scope is left
190 all saved exceptions take part in setting result of test. If there
191 is no any uncaught exception then additionally first of saved
192 exceptions is thrown.
194 --Collectors-------------------------------------------------------------------
195 Collectors are classes which collect test results. Each class does it differently.
196 Collectors can be registered by --output parameter (see HOW TO RUN section) but
197 there is also another collector created to write summary.
200 test_results_collector_summary.h
202 Collector writing tests summary. Call SummaryCollector::Register() to
205 --Usage example----------------------------------------------------------------
207 #include <test_runner.h>
208 #include <tests_common.h>
209 #include <summary_collector.h>
211 #include <sys/stat.h>
215 RUNNER_TEST_GROUP_INIT(foo_module)
217 RUNNER_TEST_SMACK(bar_allways_fails)
219 RUNNER_ASSERT(false);
222 RUNNER_TEST(bar_allways_passses)
227 RUNNER_TEST(bar_file1)
229 cosnt char *file = "bar_file1";
230 int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
231 RUNNER_ASSERT_ERRNO_MSG(fd != -1, "Cannot open " << file << " file");
235 RUNNER_CHILD_TEST_NOSMACK(bar_file2_dropped_root)
237 RUNNER_ASSERT_ERRNO(setgid(5000) == 0);
238 RUNNER_ASSERT_ERRNO(setuid(5000) == 0);
240 cosnt char *file = "bar_file2";
241 int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
244 RUNNER_FAIL_MSG("file " << file << "should not be opened");
246 RUNNER_ASSERT_ERRNO_MSG(errno == EACCESS,
247 "Wrong errno on opening " << " file");
254 void init(const std::string &testName) { ... }
255 void finish() { ... }
263 void init(const std::string &testName) { ... }
264 void finish() { ... }
265 void doRestore() { ... }
268 RUNNER_TEST(bar_optional_args, Env, Restore)
270 std::get<0>(optionalArgsTuple).doEnv();
271 std::get<1>(optionalArgsTuple).doRestore();
274 int main(int argc, char *argv[])
276 SummaryCollector::Register();
277 return DPL::Test::TestRunnerSingleton::Instance().ExecTestRunner(argc, argv);
280 --Notes------------------------------------------------------------------------
282 While changing body of test cases, be sure to remove functions and global
283 variables if not used by any other tests.
284 Use const variables instead of #define's.
285 Use mechanisms already provided in common library.