README for security-tests project ==WHAT IS====================================================================== security-tests is repository for testing packages from domain Security. ==WHAT FOR===================================================================== The security-tests repository is designed for testing packages mentioned below with binaries provided for testing them: libsmack libsmack-test security-manager security-manager-tests cynara cynara-test There are also inner-tests for testing complex security-tests framework mechanisms with binary: security-tests-inner-test ==HOW TO RUN=================================================================== Each test suite may be run with options: --output= --output= ... --output=xml example: test-binary --output=text --output=xml --file=output.xml --only-from-xml= Run only testcases specified in XML file --regexp='regexp' Only selected tests which names match regexp run --start= Start from concrete test id --group= Run tests only from one group --runignored Run also ignored tests --list Show a list of Test IDs --listgroups Show a list of Test Group names --only-from-xml= Run only testcases specified in XML file XML name is taken from attribute id="part1_part2" as whole. If part1 is not found (no _) then it is implicitily set according to suite part1 from binary tests --listingroup= Show a list of Test IDS in one group --allowchildlogs Allow to print logs from child process on screen. When active child process will be able to print logs on stdout and stderr. Both descriptors will be closed after test. --help Print help They can be run also by scripts: security-tests.sh security-tests-all.sh Each test can end with one of three possible statuses: FAILED OK IGNORED ==HOW TO WRITE================================================================= security-tests is based on extended dpl framework providing different macros. Below are categories with macros listed as below: library include macro description --Test group registering macro------------------------------------------------- dpl-test-framework test_runner.h RUNNER_TEST_GROUP_INIT Registers group of tests. Test are registered under this group until another group registering macro is called. --Test registering macros------------------------------------------------------ Adding/removing those macro calls will add/remove test cases they provide. To change tests, change body of those macro calls. Registered tests are run within group alphabetically. Those macros allow additional arguments which are classes with mandatory methods: * (constructor) () Called while registering test. Should not throw any exceptions * init(const std::string &testName) Called before test case function in order of classes passed to macro. Should not be forked. testName argument is name of the test (first macro argument). * finish(void) called after test case function in reversed order of classes passed to macro. Should not be forked. Created instances of those classes may be accessed from within test case body as argument of test case funtion is std::tuple &optionalArgsTuple dpl-test-framework test_runner.h RUNNER_TEST Function registered by this macro will be run in the same process as framework. No forking allowed unless forked process does not throw any exception. test_runner_child.h RUNNER_CHILD_TEST Function registered by this macro will be run in child process. No forking allowed unless forked process does not throw any exception. test_runner_multiprocess.h RUNNER_MULTIPROCESS_TEST Function registered by this macro will be run in the same process as framework. Forking allowed. Exception of every process will be registered. tests-common tests_common.h RUNNER_TEST_SMACK Same as RUNNER_TEST but run only with smack enabled. RUNNER_TEST_NOSMACK Same as RUNNER_TEST but run only with smack disabled. RUNNER_CHILD_TEST_SMACK Same as RUNNER_TEST_CHILD but run only with smack enabled. RUNNER_CHILD_TEST_NOSMACK Same as RUNNER_TEST_CHILD but run only with smack disabled. RUNNER_MULTIPROCESS_TEST_SMACK Same as RUNNER_TEST_MULTIPROCESS but run only with smack enabled. RUNNER_MULTIPROCESS_TEST_NOSMACK Same as RUNNER_TEST_MULTIPROCESS but run only with smack disabled. --Assert macros---------------------------------------------------------------- Used within test registering macros. First failed assertion throws test failed exception. If another assertions fail, information about fail conditions and backtrace is cumulated and presented together with already thrown exception message. dpl-test-framework test_runner.h RUNNER_ASSERT_MSG Assertion with message with backtrace information appended. RUNNER_ASSERT_ERRNO_MSG Assertion with message, error string and backtrace information appended. RUNNER_ASSERT_ERRNO Assertion with error string and backtrace information appended. RUNNER_FAIL_MSG Fail with message and backtrace information appended. RUNNER_ASSERT Assertion with backtrace information appended. RUNNER_IGNORED_MSG Assertion with message classified as ignored. --Performence macros----------------------------------------------------------- Used to do the time measurement. dpl-test-framework test_runner.h RUNNER_PERF_TEST_BEGIN Start time measurement. RUNNER_PERF_TEST_END End time measurement. --Message macros--------------------------------------------------------------- Used to print error messages during test run time. dpl-test-framework test_runner.h RUNNER_ERROR_MSG Print error message using red color. --Defer macros----------------------------------------------------------------- Used to defer throwing TestException exceptions (TestFailed, TestIgnored) by catching them and rethrowing later. This mechanism can help in breaking test and passing test result from places where throwing exceptions is not allowed dpl-test-framework test_runner.h RUNNER_DEFER_TRYCATCH Catches thrown TestException exceptions and stores them in TestRunner structures for later use. This macro works only inside deffered scope defined by RUNNER_DEFER_SCOPE, otherwise it won't catch exceptions RUNNER_DEFER_SCOPE Defines deferred scope. All RUNNER_DEFER_TRYCATCH macros used inside the scope catch and save TestException exceptions. After scope is left all saved exceptions take part in setting result of test. If there is no any uncaught exception then additionally first of saved exceptions is thrown. --Collectors------------------------------------------------------------------- Collectors are classes which collect test results. Each class does it differently. Collectors can be registered by --output parameter (see HOW TO RUN section) but there is also another collector created to write summary. dpl-test-framework test_results_collector_summary.h SummaryCollector Collector writing tests summary. Call SummaryCollector::Register() to register it --Usage example---------------------------------------------------------------- #include #include #include #include #include #include RUNNER_TEST_GROUP_INIT(foo_module) RUNNER_TEST_SMACK(bar_allways_fails) { RUNNER_ASSERT(false); } RUNNER_TEST(bar_allways_passses) { RUNNER_ASSERT(true); } RUNNER_TEST(bar_file1) { cosnt char *file = "bar_file1"; int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY)); RUNNER_ASSERT_ERRNO_MSG(fd != -1, "Cannot open " << file << " file"); close(fd); } RUNNER_CHILD_TEST_NOSMACK(bar_file2_dropped_root) { RUNNER_ASSERT_ERRNO(setgid(5000) == 0); RUNNER_ASSERT_ERRNO(setuid(5000) == 0); cosnt char *file = "bar_file2"; int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY)); if(fd != -1) { close(fd); RUNNER_FAIL_MSG("file " << file << "should not be opened"); } RUNNER_ASSERT_ERRNO_MSG(errno == EACCESS, "Wrong errno on opening " << " file"); } class Env { public: Env() { ... } void init(const std::string &testName) { ... } void finish() { ... } void doEnv() { ... } }; class Restore { public: Restore() { ... } void init(const std::string &testName) { ... } void finish() { ... } void doRestore() { ... } }; RUNNER_TEST(bar_optional_args, Env, Restore) { std::get<0>(optionalArgsTuple).doEnv(); std::get<1>(optionalArgsTuple).doRestore(); } int main(int argc, char *argv[]) { SummaryCollector::Register(); return DPL::Test::TestRunnerSingleton::Instance().ExecTestRunner(argc, argv); } --Notes------------------------------------------------------------------------ While changing body of test cases, be sure to remove functions and global variables if not used by any other tests. Use const variables instead of #define's. Use mechanisms already provided in common library.