Compute Library
18.05
|
Main framework class. More...
#include <Framework.h>
Public Member Functions | |
std::set< InstrumentsDescription > | available_instruments () const |
Supported instrument types for benchmarking. More... | |
void | init (const std::vector< framework::InstrumentsDescription > &instruments, int num_iterations, DatasetMode mode, const std::string &name_filter, const std::string &id_filter, LogLevel log_level) |
Init the framework. More... | |
void | push_suite (std::string name) |
Add a new test suite. More... | |
void | pop_suite () |
Remove innermost test suite. More... | |
template<typename T > | |
void | add_test_case (std::string test_name, DatasetMode mode, TestCaseFactory::Status status) |
Add a test case to the framework. More... | |
template<typename T , typename D > | |
void | add_data_test_case (std::string test_name, DatasetMode mode, TestCaseFactory::Status status, std::string description, D &&data) |
Add a data test case to the framework. More... | |
void | add_test_info (std::string info) |
Add info string for the next expectation/assertion. More... | |
void | clear_test_info () |
Clear the collected test info. More... | |
bool | has_test_info () const |
Check if any info has been registered. More... | |
void | print_test_info (std::ostream &os) const |
Print test info. More... | |
void | log_test_start (const TestInfo &info) |
Tell the framework that execution of a test starts. More... | |
void | log_test_skipped (const TestInfo &info) |
Tell the framework that a test case is skipped. More... | |
void | log_test_end (const TestInfo &info) |
Tell the framework that a test case finished. More... | |
void | log_failed_expectation (const TestError &error) |
Tell the framework that the currently running test case failed a non-fatal expectation. More... | |
void | log_info (const std::string &info) |
Print the debug information that has already been logged. More... | |
int | num_iterations () const |
Number of iterations per test case. More... | |
void | set_num_iterations (int num_iterations) |
Set number of iterations per test case. More... | |
bool | throw_errors () const |
Should errors be caught or thrown by the framework. More... | |
void | set_throw_errors (bool throw_errors) |
Set whether errors are caught or thrown by the framework. More... | |
bool | stop_on_error () const |
Indicates if test execution is stopped after the first failed test. More... | |
void | set_stop_on_error (bool stop_on_error) |
Set whether to abort execution after the first failed test. More... | |
bool | error_on_missing_assets () const |
Indicates if a test should be marked as failed when its assets are missing. More... | |
void | set_error_on_missing_assets (bool error_on_missing_assets) |
Set whether a test should be considered as failed if its assets cannot be found. More... | |
bool | run () |
Run all enabled test cases. More... | |
void | set_test_result (TestInfo info, TestResult result) |
Set the result for an executed test case. More... | |
void | print_test_results (Printer &printer) const |
Use the specified printer to output test results from the last run. More... | |
Profiler | get_profiler () const |
Factory method to obtain a configured profiler. More... | |
void | add_printer (Printer *printer) |
Set the printer used for the output of test results. More... | |
std::vector< TestInfo > | test_infos () const |
List of TestInfo's. More... | |
LogLevel | log_level () const |
Get the current logging level. More... | |
Static Public Member Functions | |
static Framework & | get () |
Access to the singleton. More... | |
Main framework class.
Keeps track of the global state, owns all test cases and collects results.
Definition at line 83 of file Framework.h.
|
inline |
Add a data test case to the framework.
[in] | test_name | Name of the new test case. |
[in] | mode | Mode in which to include the test. |
[in] | status | Status of the test case. |
[in] | description | Description of data . |
[in] | data | Data that will be used as input to the test. |
Definition at line 348 of file Framework.h.
References TestInfo::mode, and TestInfo::status.
Referenced by TestCaseRegistrar< T >::TestCaseRegistrar().
void add_printer | ( | Printer * | printer | ) |
Set the printer used for the output of test results.
[in] | printer | Pointer to a printer. |
Definition at line 629 of file Framework.cpp.
Referenced by main().
|
inline |
Add a test case to the framework.
[in] | test_name | Name of the new test case. |
[in] | mode | Mode in which to include the test. |
[in] | status | Status of the test case. |
Definition at line 342 of file Framework.h.
References arm_compute::support::cpp14::make_unique().
Referenced by TestCaseRegistrar< T >::TestCaseRegistrar().
void add_test_info | ( | std::string | info | ) |
Add info string for the next expectation/assertion.
[in] | info | Info string. |
Definition at line 128 of file Framework.cpp.
std::set< InstrumentsDescription > available_instruments | ( | ) | const |
Supported instrument types for benchmarking.
Definition at line 73 of file Framework.cpp.
Referenced by CommonOptions::CommonOptions().
void clear_test_info | ( | ) |
Clear the collected test info.
Definition at line 133 of file Framework.cpp.
Referenced by arm_compute::test::framework::ARM_COMPUTE_PRINT_INFO().
bool error_on_missing_assets | ( | ) | const |
Indicates if a test should be marked as failed when its assets are missing.
Definition at line 261 of file Framework.cpp.
References TestResult::CRASHED, arm_compute::test::framework::DEBUG, TestResult::DISABLED, TestCaseFactory::DISABLED, arm_compute::error(), arm_compute::test::framework::ERRORS, TestResult::EXPECTED_FAILURE, TestCaseFactory::EXPECTED_FAILURE, TestResult::FAILED, Framework::get_profiler(), arm_compute::test::validation::info(), TestError::level(), Framework::log_test_end(), Framework::log_test_skipped(), Framework::log_test_start(), TestCaseFactory::make(), TestResult::measurements, Profiler::measurements(), TestResult::NOT_RUN, Printer::print_error(), Printer::print_errors_footer(), Printer::print_errors_header(), Printer::print_info(), Framework::set_test_result(), Profiler::start(), TestInfo::status, TestResult::status, TestCaseFactory::status(), Profiler::stop(), TestResult::SUCCESS, Profiler::test_start(), and Profiler::test_stop().
Referenced by Framework::set_error_on_missing_assets().
|
static |
Access to the singleton.
Definition at line 97 of file Framework.cpp.
Referenced by arm_compute::test::framework::ARM_COMPUTE_PRINT_INFO(), CommonOptions::CommonOptions(), main(), TestCaseRegistrar< T >::TestCaseRegistrar(), and TestSuiteRegistrar::TestSuiteRegistrar().
Profiler get_profiler | ( | ) | const |
Factory method to obtain a configured profiler.
The profiler enables all instruments that have been passed to the init method.
Definition at line 600 of file Framework.cpp.
References Profiler::add(), and arm_compute::test::framework::ALL.
Referenced by Framework::error_on_missing_assets(), and main().
bool has_test_info | ( | ) | const |
Check if any info has been registered.
Definition at line 138 of file Framework.cpp.
void init | ( | const std::vector< framework::InstrumentsDescription > & | instruments, |
int | num_iterations, | ||
DatasetMode | mode, | ||
const std::string & | name_filter, | ||
const std::string & | id_filter, | ||
LogLevel | log_level | ||
) |
Init the framework.
[in] | instruments | Instrument types that will be used for benchmarking. |
[in] | num_iterations | Number of iterations per test. |
[in] | mode | Dataset mode. |
[in] | name_filter | Regular expression to filter tests by name. Only matching tests will be executed. |
[in] | id_filter | String to match selected test ids. Only matching tests will be executed. |
[in] | log_level | Verbosity of the output. |
Definition at line 103 of file Framework.cpp.
References arm_compute::test::join(), Framework::log_level(), and Framework::num_iterations().
Referenced by main().
void log_failed_expectation | ( | const TestError & | error | ) |
Tell the framework that the currently running test case failed a non-fatal expectation.
[in] | error | Description of the error. |
Definition at line 197 of file Framework.cpp.
References ARM_COMPUTE_ERROR_ON, TestCaseFactory::EXPECTED_FAILURE, TestResult::FAILED, TestError::level(), Printer::print_error(), TestInfo::status, and TestResult::status.
void log_info | ( | const std::string & | info | ) |
Print the debug information that has already been logged.
[in] | info | Description of the log info. |
Definition at line 215 of file Framework.cpp.
References arm_compute::test::framework::DEBUG, and Printer::print_info().
Referenced by arm_compute::test::framework::ARM_COMPUTE_PRINT_INFO().
LogLevel log_level | ( | ) | const |
Get the current logging level.
Definition at line 655 of file Framework.cpp.
Referenced by Framework::init().
void log_test_end | ( | const TestInfo & | info | ) |
Tell the framework that a test case finished.
[in] | info | Test info. |
Definition at line 178 of file Framework.cpp.
References arm_compute::test::framework::MEASUREMENTS, Printer::print_measurements(), Printer::print_test_footer(), and arm_compute::test::framework::TESTS.
Referenced by Framework::error_on_missing_assets().
void log_test_skipped | ( | const TestInfo & | info | ) |
Tell the framework that a test case is skipped.
[in] | info | Test info. |
Definition at line 173 of file Framework.cpp.
References arm_compute::test::validation::info().
Referenced by Framework::error_on_missing_assets().
void log_test_start | ( | const TestInfo & | info | ) |
Tell the framework that execution of a test starts.
[in] | info | Test info. |
Definition at line 162 of file Framework.cpp.
References Printer::print_test_header(), and arm_compute::test::framework::TESTS.
Referenced by Framework::error_on_missing_assets().
int num_iterations | ( | ) | const |
Number of iterations per test case.
Definition at line 226 of file Framework.cpp.
Referenced by Framework::init(), and Framework::set_num_iterations().
void pop_suite | ( | ) |
Remove innermost test suite.
Definition at line 123 of file Framework.cpp.
Referenced by TestSuiteRegistrar::TestSuiteRegistrar().
void print_test_info | ( | std::ostream & | os | ) | const |
Print test info.
[out] | os | Output stream. |
Definition at line 143 of file Framework.cpp.
References arm_compute::utility::for_each().
Referenced by arm_compute::test::framework::ARM_COMPUTE_PRINT_INFO().
void print_test_results | ( | Printer & | printer | ) | const |
Use the specified printer to output test results from the last run.
This method can be used if the test results need to be obtained using a different printer than the one managed by the framework.
[in] | printer | Printer used to output results. |
Definition at line 586 of file Framework.cpp.
References Printer::print_measurements(), Printer::print_run_footer(), Printer::print_run_header(), Printer::print_test_footer(), and Printer::print_test_header().
void push_suite | ( | std::string | name | ) |
Add a new test suite.
[in] | name | Name of the added test suite. |
Definition at line 118 of file Framework.cpp.
Referenced by TestSuiteRegistrar::TestSuiteRegistrar().
bool run | ( | ) |
Run all enabled test cases.
Definition at line 508 of file Framework.cpp.
References CLKernelLibrary::clear_programs_cache(), TestResult::CRASHED, TestResult::DISABLED, TestResult::EXPECTED_FAILURE, TestResult::FAILED, CLScheduler::get(), CLKernelLibrary::get(), TestFilter::is_selected(), TestCaseFactory::mode(), TestCaseFactory::name(), arm_compute::test::framework::NONE, Printer::print_run_footer(), Printer::print_run_header(), CLScheduler::set_context(), TestCaseFactory::status(), TestResult::SUCCESS, and arm_compute::test::framework::TESTS.
Referenced by main().
void set_error_on_missing_assets | ( | bool | error_on_missing_assets | ) |
Set whether a test should be considered as failed if its assets cannot be found.
[in] | error_on_missing_assets | True if a test should be marked as failed when its assets are missing. |
Definition at line 256 of file Framework.cpp.
References Framework::error_on_missing_assets().
Referenced by main().
void set_num_iterations | ( | int | num_iterations | ) |
Set number of iterations per test case.
[in] | num_iterations | Number of iterations per test case. |
Definition at line 231 of file Framework.cpp.
References Framework::num_iterations().
void set_stop_on_error | ( | bool | stop_on_error | ) |
Set whether to abort execution after the first failed test.
[in] | stop_on_error | True if execution is going to be aborted after first failed test. |
Definition at line 246 of file Framework.cpp.
References Framework::stop_on_error().
Referenced by main().
void set_test_result | ( | TestInfo | info, |
TestResult | result | ||
) |
Set the result for an executed test case.
[in] | info | Test info. |
[in] | result | Execution result. |
Definition at line 581 of file Framework.cpp.
Referenced by Framework::error_on_missing_assets().
void set_throw_errors | ( | bool | throw_errors | ) |
Set whether errors are caught or thrown by the framework.
[in] | throw_errors | True if errors should be thrown. |
Definition at line 236 of file Framework.cpp.
References Framework::throw_errors().
Referenced by main().
bool stop_on_error | ( | ) | const |
Indicates if test execution is stopped after the first failed test.
Definition at line 251 of file Framework.cpp.
Referenced by Framework::set_stop_on_error().
std::vector< TestInfo > test_infos | ( | ) | const |
List of TestInfo's.
Definition at line 634 of file Framework.cpp.
References TestFilter::is_selected(), and TestInfo::name.
Referenced by main().
bool throw_errors | ( | ) | const |
Should errors be caught or thrown by the framework.
Definition at line 241 of file Framework.cpp.
Referenced by Framework::set_throw_errors().