X-Git-Url: http://review.tizen.org/git/?a=blobdiff_plain;f=README;h=c946fe76002223e1a767b0fd6f0e7a3516f9ea1f;hb=c96e454948003646973b678be5e2ea9057fe0892;hp=62fc8c12b8f7e8773cfa02e1ecc8e4dd8a243dc3;hpb=2b2a1fb44c37fc2084358f7d8f6b97473fd66f32;p=platform%2Fcore%2Ftest%2Fsecurity-tests.git diff --git a/README b/README index 62fc8c1..c946fe7 100644 --- a/README +++ b/README @@ -1,65 +1,245 @@ README for security-tests project -The project security-tests is designed for testing other packages: - --libsmack --libprivilege-control --security-server - -The test runner framework used by this project is dpl/test. For more -information about dpl see project wrt-commons. - -The tests designed for each project are grouped in separate folders: -1, libprivilege-control - Sorces in folder tests/libprivilege-control-tests - Test cases defined in tests/libprivilege-control-tests/test.cases.cpp -2, libsmack - Sources in folder tests/libsmack-tests - Test cases defined in tests/libsmack-tests/test.cases.cpp -3, security server - Sources in folders tests/security-server-tests - Test cases for different aspects of package divided into several - security_server_tests_*.cpp files inside that folder. - -Tests are run on the tested device by executing security-tests-all.sh or each -test suite separately: -- libsmack-test, -- libprivilege-control-test, -- security-server-tests-client-smack, -- security-server-tests-label, -- security-server-tests-pid-reuser, -- security-server-tests-server, -- security-server-tests-api-speed, -- security-server-tests-password, -- security-server-tests-dbus. +==WHAT IS====================================================================== + +security-tests is repository for testing packages from domain Security. + +==WHAT FOR===================================================================== + +The security-tests repository is designed for testing packages mentioned below +with binaries provided for testing them: + +libsmack + libsmack-test +libprivilege-control + libprivilege-control-test +security-server + security-server-tests-client-smack + security-server-tests-stress + security-server-tests-server + security-server-tests-api-speed + security-server-tests-password + security-server-tests-privilege + security-server-tests-dbus +security-manager + security-manager-tests +cynara + cynara-test + +There are also inner-tests for testing complex security-tests framework +mechanisms with binary: + security-tests-inner-test + +==HOW TO RUN=================================================================== Each test suite may be run with options: --output= --output= ... - possible output types: - --output=csv - --output=html - --output=tap - --output=text --output=xml example: test-binary --output=text --output=xml --file=output.xml - --regexp='regexp' Only selected tests which names match regexp run - --start= Start from concrete test id --group= - Run tests only from one group - --runignored Run also ignored tests - --list Show a list of Test IDs - --listgroups Show a list of Test Group names - --only-from-xml= Run only testcases specified in XML file - XML name is taken from attribute id="part1_part2" as whole. - If part1 is not found (no _) then it is implicitily set according to - suite part1 from binary tests - --listingroup= Show a list of Test IDS in one group - --allowchildlogs Allow to print logs from child process on screen. - When active child process will be able to print logs on stdout and - stderr. Both descriptors will be closed after test. - --help print help - -Apart from unit tests, the package also provides scripts designed to check: -- CPU usage by SMACK related processes. (test-performance-check.sh, which - internally uses the perf binary, and gives output to stdout.) -- Time of last booting of the system. (boot-time-measure.bash) + --only-from-xml= Run only testcases specified in XML file + --regexp='regexp' Only selected tests which names match regexp run + --start= Start from concrete test id + --group= Run tests only from one group + --runignored Run also ignored tests + --list Show a list of Test IDs + --listgroups Show a list of Test Group names + --only-from-xml= Run only testcases specified in XML file + XML name is taken from attribute id="part1_part2" as whole. + If part1 is not found (no _) then it is implicitily set according to + suite part1 from binary tests + --listingroup= Show a list of Test IDS in one group + --allowchildlogs Allow to print logs from child process on screen. + When active child process will be able to print logs on stdout and + stderr. Both descriptors will be closed after test. + --help Print help + +They can be run also by scripts: + security-tests.sh + security-tests-all.sh + +Each test can end with one of three possible statuses: + FAILED + OK + IGNORED + +==HOW TO WRITE================================================================= + +security-tests is based on extended dpl framework providing different macros. +Below are categories with macros listed as below: +library + include + macro + description + +--Test group registering macro------------------------------------------------- + +dpl-test-framework + test_runner.h + RUNNER_TEST_GROUP_INIT + Registers group of tests. Test are registered under this group until + another group registering macro is called. + +--Test registering macros------------------------------------------------------ +Adding/removing those macro calls will add/remove test cases they provide. To +change tests, change body of those macro calls. Registered tests are run within +group alphabetically. + +dpl-test-framework + test_runner.h + RUNNER_TEST + Function registered by this macro will be run in the same process as + framework. No forking allowed unless forked process does not throw + any exception. + test_runner_child.h + RUNNER_CHILD_TEST + Function registered by this macro will be run in child process. No + forking allowed unless forked process does not throw any exception. + test_runner_multiprocess.h + RUNNER_MULTIPROCESS_TEST + Function registered by this macro will be run in the same process as + framework. Forking allowed. Exception of every process will be + registered. +tests-common + tests_common.h + RUNNER_TEST_SMACK + Same as RUNNER_TEST but run only with smack enabled. + RUNNER_TEST_NOSMACK + Same as RUNNER_TEST but run only with smack disabled. + RUNNER_CHILD_TEST_SMACK + Same as RUNNER_TEST_CHILD but run only with smack enabled. + RUNNER_CHILD_TEST_NOSMACK + Same as RUNNER_TEST_CHILD but run only with smack disabled. + RUNNER_MULTIPROCESS_TEST_SMACK + Same as RUNNER_TEST_MULTIPROCESS but run only with smack enabled. + RUNNER_MULTIPROCESS_TEST_NOSMACK + Same as RUNNER_TEST_MULTIPROCESS but run only with smack disabled. + +--Assert macros---------------------------------------------------------------- +Used within test registering macros. + +First failed assertion throws test failed exception. If another assertions +fail, information about fail conditions and backtrace is cumulated and +presented together with already thrown exception message. + +dpl-test-framework + test_runner.h + RUNNER_ASSERT_MSG + Assertion with message with backtrace information appended. + RUNNER_ASSERT_ERRNO_MSG + Assertion with message, error string and backtrace information + appended. + RUNNER_ASSERT_ERRNO + Assertion with error string and backtrace information appended. + RUNNER_FAIL_MSG + Fail with message and backtrace information appended. + RUNNER_ASSERT + Assertion with backtrace information appended. + RUNNER_IGNORED_MSG + Assertion with message classified as ignored. + +--Performence macros----------------------------------------------------------- +Used to do the time measurement. + +dpl-test-framework + test_runner.h + RUNNER_PERF_TEST_BEGIN + Start time measurement. + RUNNER_PERF_TEST_END + End time measurement. + +--Message macros--------------------------------------------------------------- +Used to print error messages during test run time. + +dpl-test-framework + test_runner.h + RUNNER_ERROR_MSG + Print error message using red color. + +--Defer macros----------------------------------------------------------------- +Used to defer throwing TestException exceptions (TestFailed, TestIgnored) +by catching them and rethrowing later. This mechanism can help in breaking +test and passing test result from places where throwing exceptions +is not allowed + +dpl-test-framework + test_runner.h + RUNNER_DEFER_TRYCATCH + Catches thrown TestException exceptions and stores them in TestRunner + structures for later use. This macro works only inside deffered scope + defined by RUNNER_DEFER_SCOPE, otherwise it won't catch exceptions + RUNNER_DEFER_SCOPE + Defines deferred scope. All RUNNER_DEFER_TRYCATCH macros used inside + the scope catch and save TestException exceptions. After scope is left + all saved exceptions take part in setting result of test. If there + is no any uncaught exception then additionally first of saved + exceptions is thrown. + +--Collectors------------------------------------------------------------------- +Collectors are classes which collect test results. Each class does it differently. +Collectors can be registered by --output parameter (see HOW TO RUN section) but +there is also another collector created to write summary. + +dpl-test-framework + test_results_collector_summary.h + SummaryCollector + Collector writing tests summary. Call SummaryCollector::Register() to + register it + +--Usage example---------------------------------------------------------------- + +#include +#include +#include + +#include +#include +#include + +RUNNER_TEST_GROUP_INIT(foo_module) + +RUNNER_TEST_SMACK(bar_allways_fails) +{ + RUNNER_ASSERT(false); +} + +RUNNER_TEST(bar_allways_passses) +{ + RUNNER_ASSERT(true); +} + +RUNNER_TEST(bar_file1) +{ + cosnt char *file = "bar_file1"; + int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY)); + RUNNER_ASSERT_ERRNO_MSG(fd != -1, "Cannot open " << file << " file"); + close(fd); +} + +RUNNER_CHILD_TEST_NOSMACK(bar_file2_dropped_root) +{ + RUNNER_ASSERT_ERRNO(setgid(5000) == 0); + RUNNER_ASSERT_ERRNO(setuid(5000) == 0); + + cosnt char *file = "bar_file2"; + int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY)); + if(fd != -1) { + close(fd); + RUNNER_FAIL_MSG("file " << file << "should not be opened"); + } + RUNNER_ASSERT_ERRNO_MSG(errno == EACCESS, + "Wrong errno on opening " << " file"); +} + +int main(int argc, char *argv[]) +{ + SummaryCollector::Register(); + return DPL::Test::TestRunnerSingleton::Instance().ExecTestRunner(argc, argv); +} + +--Notes------------------------------------------------------------------------ + + While changing body of test cases, be sure to remove functions and global +variables if not used by any other tests. + Use const variables instead of #define's. + Use mechanisms already provided in common library.