README for security-tests project

==WHAT IS======================================================================

security-tests is repository for testing packages from domain Security.

==WHAT FOR=====================================================================

The security-tests repository is designed for testing packages mentioned below
with binaries provided for testing them:

libsmack
  libsmack-test
libprivilege-control
  libprivilege-control-test
security-server
  security-server-tests-client-smack
  security-server-tests-stress
  security-server-tests-server
  security-server-tests-api-speed
  security-server-tests-password
  security-server-tests-privilege
  security-server-tests-dbus
security-manager
  security-manager-tests
cynara
  cynara-test

There are also inner-tests for testing complex security-tests framework
mechanisms with binary:
  security-tests-inner-test

==HOW TO RUN===================================================================

Each test suite may be run with options:
  --output=<output type> --output=<output type> ...
        --output=xml
      example:
        test-binary --output=text --output=xml --file=output.xml
  --only-from-xml=<xml file>    Run only testcases specified in XML file
  --regexp='regexp'             Only selected tests which names match regexp run
  --start=<test id>             Start from concrete test id
  --group=<group name>          Run tests only from one group
  --runignored                  Run also ignored tests
  --list                        Show a list of Test IDs
  --listgroups                  Show a list of Test Group names
  --only-from-xml=<xml file>    Run only testcases specified in XML file
        XML name is taken from attribute id="part1_part2" as whole.
        If part1 is not found (no _) then it is implicitily set according to
        suite part1 from binary tests
  --listingroup=<group name>    Show a list of Test IDS in one group
  --allowchildlogs              Allow to print logs from child process on screen.
        When active child process will be able to print logs on stdout and
        stderr. Both descriptors will be closed after test.
  --help                        Print help

They can be run also by scripts:
  security-tests.sh
  security-tests-all.sh

Each test can end with one of three possible statuses:
  FAILED
  OK
  IGNORED

==HOW TO WRITE=================================================================

security-tests is based on extended dpl framework providing different macros.
Below are categories with macros listed as below:
library
  include
    macro
          description

--Test group registering macro-------------------------------------------------

dpl-test-framework
  test_runner.h
    RUNNER_TEST_GROUP_INIT
          Registers group of tests. Test are registered under this group until
          another group registering macro is called.

--Test registering macros------------------------------------------------------
Adding/removing those macro calls will add/remove test cases they provide. To
change tests, change body of those macro calls. Registered tests are run within
group alphabetically.

dpl-test-framework
  test_runner.h
    RUNNER_TEST
          Function registered by this macro will be run in the same process as
          framework. No forking allowed unless forked process does not throw
          any exception.
  test_runner_child.h
    RUNNER_CHILD_TEST
          Function registered by this macro will be run in child process. No
          forking allowed unless forked process does not throw any exception.
  test_runner_multiprocess.h
    RUNNER_MULTIPROCESS_TEST
          Function registered by this macro will be run in the same process as
          framework. Forking allowed. Exception of every process will be
          registered.
tests-common
  tests_common.h
    RUNNER_TEST_SMACK
          Same as RUNNER_TEST but run only with smack enabled.
    RUNNER_TEST_NOSMACK
          Same as RUNNER_TEST but run only with smack disabled.
    RUNNER_CHILD_TEST_SMACK
          Same as RUNNER_TEST_CHILD but run only with smack enabled.
    RUNNER_CHILD_TEST_NOSMACK
          Same as RUNNER_TEST_CHILD but run only with smack disabled.
    RUNNER_MULTIPROCESS_TEST_SMACK
          Same as RUNNER_TEST_MULTIPROCESS but run only with smack enabled.
    RUNNER_MULTIPROCESS_TEST_NOSMACK
          Same as RUNNER_TEST_MULTIPROCESS but run only with smack disabled.

--Assert macros----------------------------------------------------------------
Used within test registering macros.

First failed assertion throws test failed exception. If another assertions
fail, information about fail conditions and backtrace is cumulated and
presented together with already thrown exception message.

dpl-test-framework
  test_runner.h
    RUNNER_ASSERT_MSG
          Assertion with message with backtrace information appended.
    RUNNER_ASSERT_ERRNO_MSG
          Assertion with message, error string and backtrace information
          appended.
    RUNNER_ASSERT_ERRNO
          Assertion with error string and backtrace information appended.
    RUNNER_FAIL_MSG
          Fail with message and backtrace information appended.
    RUNNER_ASSERT
          Assertion with backtrace information appended.
    RUNNER_IGNORED_MSG
          Assertion with message classified as ignored.

--Performence macros-----------------------------------------------------------
Used to do the time measurement.

dpl-test-framework
  test_runner.h
    RUNNER_PERF_TEST_BEGIN
          Start time measurement.
    RUNNER_PERF_TEST_END
          End time measurement.

--Message macros---------------------------------------------------------------
Used to print error messages during test run time.

dpl-test-framework
  test_runner.h
    RUNNER_ERROR_MSG
          Print error message using red color.

--Collectors-------------------------------------------------------------------
Collectors are classes which collect test results. Each class does it differently.
Collectors can be registered by --output parameter (see HOW TO RUN section) but
there is also another collector created to write summary.

dpl-test-framework
  test_results_collector_summary.h
    SummaryCollector
          Collector writing tests summary. Call SummaryCollector::Register() to
          register it

--Usage example----------------------------------------------------------------

#include <test_runner.h>
#include <tests_common.h>
#include <summary_collector.h>

#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>

RUNNER_TEST_GROUP_INIT(foo_module)

RUNNER_TEST_SMACK(bar_allways_fails)
{
    RUNNER_ASSERT(false);
}

RUNNER_TEST(bar_allways_passses)
{
    RUNNER_ASSERT(true);
}

RUNNER_TEST(bar_file1)
{
    cosnt char *file = "bar_file1";
    int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
    RUNNER_ASSERT_ERRNO_MSG(fd != -1, "Cannot open " << file << " file");
    close(fd);
}

RUNNER_CHILD_TEST_NOSMACK(bar_file2_dropped_root)
{
    RUNNER_ASSERT_ERRNO(setgid(5000) == 0);
    RUNNER_ASSERT_ERRNO(setuid(5000) == 0);

    cosnt char *file = "bar_file2";
    int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
    if(fd != -1) {
        close(fd);
        RUNNER_FAIL_MSG("file " << file << "should not be opened");
    }
    RUNNER_ASSERT_ERRNO_MSG(errno == EACCESS,
                            "Wrong errno on opening " << " file");
}

int main(int argc, char *argv[])
{
    SummaryCollector::Register();
    return DPL::Test::TestRunnerSingleton::Instance().ExecTestRunner(argc, argv);
}

--Notes------------------------------------------------------------------------

  While changing body of test cases, be sure to remove functions and global
variables if not used by any other tests.
  Use const variables instead of #define's.
  Use mechanisms already provided in common library.