README for security-tests project

==WHAT IS======================================================================

security-tests is repository for testing packages from domain Security.

==WHAT FOR=====================================================================

The security-tests repository is designed for testing packages mentioned below
with binaries provided for testing them:

security-manager
  security-manager-tests
cynara
  cynara-test
ode
  ode-tests

There are also inner-tests for testing complex security-tests framework
mechanisms with binary:
  security-tests-inner-test

==HOW TO RUN===================================================================

Each test suite may be run with options:
  --output=<output type> --output=<output type> ...
        --output=xml
      example:
        test-binary --output=text --output=xml --file=output.xml
  --only-from-xml=<xml file>    Run only testcases specified in XML file
  --regexp='regexp'             Only selected tests which names match regexp run
  --start=<test id>             Start from concrete test id
  --group=<group name>          Run tests only from one group
  --runignored                  Run also ignored tests
  --list                        Show a list of Test IDs
  --listgroups                  Show a list of Test Group names
  --only-from-xml=<xml file>    Run only testcases specified in XML file
        XML name is taken from attribute id="part1_part2" as whole.
        If part1 is not found (no _) then it is implicitily set according to
        suite part1 from binary tests
  --listingroup=<group name>    Show a list of Test IDS in one group
  --allowchildlogs              Allow to print logs from child process on screen.
        When active child process will be able to print logs on stdout and
        stderr. Both descriptors will be closed after test.
  --help                        Print help

They can be run also by scripts:
  security-tests.sh
  security-tests-all.sh

Each test can end with one of three possible statuses:
  FAILED
  OK
  IGNORED

==HOW TO WRITE=================================================================

security-tests is based on extended dpl framework providing different macros.
Below are categories with macros listed as below:
library
  include
    macro
          description

--Test group registering macro-------------------------------------------------

dpl-test-framework
  test_runner.h
    RUNNER_TEST_GROUP_INIT
          Registers group of tests. Test are registered under this group until
          another group registering macro is called.

--Test registering macros------------------------------------------------------
Adding/removing those macro calls will add/remove test cases they provide. To
change tests, change body of those macro calls. Registered tests are run within
group alphabetically.
Those macros allow additional arguments which are classes with mandatory
methods:
* (constructor) ()
    Called while registering test.
    Should not throw any exceptions
* init(const std::string &testName)
    Called before test case function in order of classes passed to macro.
    Should not be forked.
    testName argument is name of the test (first macro argument).
* finish(void)
    called after test case function in reversed order of classes passed to
    macro.
    Should not be forked.
Created instances of those classes may be accessed from within test case body
as argument of test case funtion is
  std::tuple<ClassesPassed> &optionalArgsTuple

dpl-test-framework
  test_runner.h
    RUNNER_TEST
          Function registered by this macro will be run in the same process as
          framework. No forking allowed unless forked process does not throw
          any exception.
  test_runner_child.h
    RUNNER_CHILD_TEST
          Function registered by this macro will be run in child process. No
          forking allowed unless forked process does not throw any exception.
  test_runner_multiprocess.h
    RUNNER_MULTIPROCESS_TEST
          Function registered by this macro will be run in the same process as
          framework. Forking allowed. Exception of every process will be
          registered.
tests-common
  tests_common.h
    RUNNER_TEST_SMACK
          Same as RUNNER_TEST but run only with smack enabled.
    RUNNER_TEST_NOSMACK
          Same as RUNNER_TEST but run only with smack disabled.
    RUNNER_CHILD_TEST_SMACK
          Same as RUNNER_TEST_CHILD but run only with smack enabled.
    RUNNER_CHILD_TEST_NOSMACK
          Same as RUNNER_TEST_CHILD but run only with smack disabled.
    RUNNER_MULTIPROCESS_TEST_SMACK
          Same as RUNNER_TEST_MULTIPROCESS but run only with smack enabled.

--Assert macros----------------------------------------------------------------
Used within test registering macros.

First failed assertion throws test failed exception. If another assertions
fail, information about fail conditions and backtrace is cumulated and
presented together with already thrown exception message.

dpl-test-framework
  test_runner.h
    RUNNER_ASSERT_MSG
          Assertion with message with backtrace information appended.
    RUNNER_ASSERT_ERRNO_MSG
          Assertion with message, error string and backtrace information
          appended.
    RUNNER_ASSERT_ERRNO
          Assertion with error string and backtrace information appended.
    RUNNER_FAIL_MSG
          Fail with message and backtrace information appended.
    RUNNER_ASSERT
          Assertion with backtrace information appended.
    RUNNER_IGNORED_MSG
          Assertion with message classified as ignored.

--Performence macros-----------------------------------------------------------
Used to do the time measurement.

dpl-test-framework
  test_runner.h
    RUNNER_PERF_TEST_BEGIN
          Start time measurement.
    RUNNER_PERF_TEST_END
          End time measurement.

--Message macros---------------------------------------------------------------
Used to print error messages during test run time.

dpl-test-framework
  test_runner.h
    RUNNER_ERROR_MSG
          Print error message using red color.

--Defer macros-----------------------------------------------------------------
Used to defer throwing TestException exceptions (TestFailed, TestIgnored)
by catching them and rethrowing later. This mechanism can help in breaking
test and passing test result from places where throwing exceptions
is not allowed

dpl-test-framework
  test_runner.h
    RUNNER_DEFER_TRYCATCH
          Catches thrown TestException exceptions and stores them in TestRunner
          structures for later use. This macro works only inside deffered scope
          defined by RUNNER_DEFER_SCOPE, otherwise it won't catch exceptions
    RUNNER_DEFER_SCOPE
          Defines deferred scope. All RUNNER_DEFER_TRYCATCH macros used inside
          the scope catch and save TestException exceptions. After scope is left
          all saved exceptions take part in setting result of test. If there
          is no any uncaught exception then additionally first of saved
          exceptions is thrown.

--Collectors-------------------------------------------------------------------
Collectors are classes which collect test results. Each class does it differently.
Collectors can be registered by --output parameter (see HOW TO RUN section) but
there is also another collector created to write summary.

dpl-test-framework
  test_results_collector_summary.h
    SummaryCollector
          Collector writing tests summary. Call SummaryCollector::Register() to
          register it

--Usage example----------------------------------------------------------------

#include <test_runner.h>
#include <tests_common.h>
#include <summary_collector.h>

#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>

RUNNER_TEST_GROUP_INIT(foo_module)

RUNNER_TEST_SMACK(bar_allways_fails)
{
    RUNNER_ASSERT(false);
}

RUNNER_TEST(bar_allways_passses)
{
    RUNNER_ASSERT(true);
}

RUNNER_TEST(bar_file1)
{
    cosnt char *file = "bar_file1";
    int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
    RUNNER_ASSERT_ERRNO_MSG(fd != -1, "Cannot open " << file << " file");
    close(fd);
}

RUNNER_CHILD_TEST_NOSMACK(bar_file2_dropped_root)
{
    RUNNER_ASSERT_ERRNO(setgid(5000) == 0);
    RUNNER_ASSERT_ERRNO(setuid(5000) == 0);

    cosnt char *file = "bar_file2";
    int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
    if(fd != -1) {
        close(fd);
        RUNNER_FAIL_MSG("file " << file << "should not be opened");
    }
    RUNNER_ASSERT_ERRNO_MSG(errno == EACCESS,
                            "Wrong errno on opening " << " file");
}

class Env
{
public:
    Env() { ... }
    void init(const std::string &testName) { ... }
    void finish() { ... }
    void doEnv() { ... }
};

class Restore
{
public:
    Restore() { ... }
    void init(const std::string &testName) { ... }
    void finish() { ... }
    void doRestore() { ... }
};

RUNNER_TEST(bar_optional_args, Env, Restore)
{
    std::get<0>(optionalArgsTuple).doEnv();
    std::get<1>(optionalArgsTuple).doRestore();
}

int main(int argc, char *argv[])
{
    SummaryCollector::Register();
    return DPL::Test::TestRunnerSingleton::Instance().ExecTestRunner(argc, argv);
}

--Notes------------------------------------------------------------------------

  While changing body of test cases, be sure to remove functions and global
variables if not used by any other tests.
  Use const variables instead of #define's.
  Use mechanisms already provided in common library.