- _ _ ____ _
- ___| | | | _ \| |
- / __| | | | |_) | |
- | (__| |_| | _ <| |___
+ _ _ ____ _
+ ___| | | | _ \| |
+ / __| | | | |_) | |
+ | (__| |_| | _ <| |___
\___|\___/|_| \_\_____|
The cURL Test Suite
-Requires:
- perl, sh
+ 1. Running
+ 1.1 Requires to run
+ 1.2 Port numbers used by test servers
+ 1.3 Test servers
+ 1.4 Run
+ 1.5 Shell startup scripts
+ 1.6 Memory test
+ 1.7 Debug
+ 1.8 Logs
+ 1.9 Test input files
+ 1.10 Code coverage
+ 1.11 Remote testing
-Run:
- 'make test'. This invokes the 'runtests.sh' shell script. Edit the top
- variables of that script in case you have some specific needs.
+ 2. Numbering
+ 2.1 Test case numbering
- The script breaks on the first test that doesn't do OK. Run the script
- with -v for more verbose output.
+ 3. Write tests
+ 3.1 test data
+ 3.2 curl tests
+ 3.3 libcurl tests
+ 3.4 unit tests
-Logs:
- All logs are generated in the logs/ subdirctory (it is emtpied first
- in the runtests.sh script)
+ 4. TODO
+ 4.1 More protocols
+ 4.2 SOCKS auth
-Data:
- All test-data are put in the data/ subdirctory.
+==============================================================================
- For each tests there exist four files. Replace N with the test number:
+1. Running
- nameN.txt: test description as displayed when run
+ 1.1 Requires to run
- commandN.txt: command line options for this test
+ perl (and a unix-style shell)
+ python (and a unix-style shell)
+ diff (when a test fails, a diff is shown)
+ stunnel (for HTTPS and FTPS tests)
+ OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
- httpN.txt: the full dump of the HTTP protocol communication that curl is
- expected to use when performing this test
+ 1.2 Port numbers used by test servers
- replyN.txt: the full dump the server should reply to curl for this test.
- If the final result that curl should've got is not in this
- file, you can instead name the file replyN0001.txt. This enables
- you to fiddle more. ;-)
+ - TCP/8990 for HTTP
+ - TCP/8991 for HTTPS
+ - TCP/8992 for FTP
+ - TCP/8993 for FTPS
+ - TCP/8994 for HTTP IPv6
+ - TCP/8995 for FTP (2)
+ - TCP/8996 for FTP IPv6
+ - UDP/8997 for TFTP
+ - UDP/8998 for TFTP IPv6
+ - TCP/8999 for SCP/SFTP
+ - TCP/9000 for SOCKS
+ - TCP/9001 for POP3
+ - TCP/9002 for IMAP
+ - TCP/9003 for SMTP
+ - TCP/9004 for SMTP IPv6
+ - TCP/9005 for RTSP
+ - TCP/9006 for RTSP IPv6
+ - TCP/9007 for GOPHER
+ - TCP/9008 for GOPHER IPv6
+ - TCP/9008 for HTTPS server with TLS-SRP support
-FIX:
+ 1.3 Test servers
- * Make httpserver.pl work when we PUT without Content-Length:
+ The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone
+ servers on the ports listed above to which it makes requests. For SSL tests,
+ it runs stunnel to handle encryption to the regular servers. For SSH, it
+ runs a standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform
+ the SOCKS functionality and requires a SSH client and server.
+ The base port number (8990), which all the individual port numbers are
+ indexed from, can be set explicitly using runtests.pl' -b option to allow
+ running more than one instance of the test suite simultaneously on one
+ machine, or just move the servers in case you have local services on any of
+ those ports.
+
+ The HTTP server supports listening on a Unix domain socket, the default
+ location is 'http.sock'.
+
+ 1.4 Run
+
+ 'make test'. This builds the test suite support code and invokes the
+ 'runtests.pl' perl script to run all the tests. Edit the top variables
+ of that script in case you have some specific needs, or run the script
+ manually (after the support code has been built).
+
+ The script breaks on the first test that doesn't do OK. Use -a to prevent
+ the script from aborting on the first error. Run the script with -v for more
+ verbose output. Use -d to run the test servers with debug output enabled as
+ well. Specifying -k keeps all the log files generated by the test intact.
+
+ Use -s for shorter output, or pass test numbers to run specific tests only
+ (like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case
+ ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from
+ 3 to 9. Any test numbers starting with ! are disabled, as are any test
+ numbers found in the file data/DISABLED (one per line).
+
+ When -s is not present, each successful test will display on one line the
+ test number and description and on the next line a set of flags, the test
+ result, current test sequence, total number of tests to be run and an
+ estimated amount of time to complete the test run. The flags consist of
+ these letters describing what is checked in this test:
+
+ s stdout
+ d data
+ u upload
+ p protocol
+ o output
+ e exit code
+ m memory
+ v valgrind
+
+ 1.5 Shell startup scripts
+
+ Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
+ influenced by the output of system wide or user specific shell startup
+ scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which
+ output text messages or escape sequences on user login. When these shell
+ startup messages or escape sequences are output they might corrupt the
+ expected stream of data which flows to the sftp-server or from the ssh
+ client which can result in bad test behaviour or even prevent the test
+ server from running.
+
+ If the test suite ssh or sftp server fails to start up and logs the message
+ 'Received message too long' then you are certainly suffering the unwanted
+ output of a shell startup script. Locate, cleanup or adjust the shell
+ script.
+
+ 1.6 Memory test
+
+ The test script will check that all allocated memory is freed properly IF
+ curl has been built with the CURLDEBUG define set. The script will
+ automatically detect if that is the case, and it will use the
+ 'memanalyze.pl' script to analyze the memory debugging output.
+
+ Also, if you run tests on a machine where valgrind is found, the script will
+ use valgrind to run the test with (unless you use -n) to further verify
+ correctness.
+
+ runtests.pl's -t option will enable torture testing mode, which runs each
+ test many times and makes each different memory allocation fail on each
+ successive run. This tests the out of memory error handling code to ensure
+ that memory leaks do not occur even in those situations. It can help to
+ compile curl with CPPFLAGS=-DMEMDEBUG_LOG_SYNC when using this option, to
+ ensure that the memory log file is properly written even if curl crashes.
+
+ 1.7 Debug
+
+ If a test case fails, you can conveniently get the script to invoke the
+ debugger (gdb) for you with the server running and the exact same command
+ line parameters that failed. Just invoke 'runtests.pl <test number> -g' and
+ then just type 'run' in the debugger to perform the command through the
+ debugger.
+
+ 1.8 Logs
+
+ All logs are generated in the logs/ subdirectory (it is emptied first in the
+ runtests.pl script). Use runtests.pl -k to force it to keep the temporary
+ files after the test run since successful runs will clean it up otherwise.
+
+ 1.9 Test input files
+
+ All test cases are put in the data/ subdirectory. Each test is stored in the
+ file named according to the test number.
+
+ See FILEFORMAT for the description of the test case files.
+
+ 1.10 Code coverage
+
+ gcc provides a tool that can determine the code coverage figures for
+ the test suite. To use it, configure curl with
+ CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal
+ and torture tests to get more full coverage, i.e. do:
+
+ make test
+ make test-torture
+
+ The graphical tool ggcov can be used to browse the source and create
+ coverage reports on *NIX hosts:
+
+ ggcov -r lib src
+
+ The text mode tool gcov may also be used, but it doesn't handle object files
+ in more than one directory very well.
+
+ 1.11 Remote testing
+
+ The runtests.pl script provides some hooks to allow curl to be tested on a
+ machine where perl can not be run. The test framework in this case runs on
+ a workstation where perl is available, while curl itself is run on a remote
+ system using ssh or some other remote execution method. See the comments at
+ the beginning of runtests.pl for details.
+
+2. Numbering
+
+ 2.1 Test case numbering
+
+ 1 - 99 HTTP
+ 100 - 199 FTP*
+ 200 - 299 FILE*
+ 300 - 399 HTTPS
+ 400 - 499 FTPS
+ 500 - 599 libcurl source code tests, not using the curl command tool
+ 600 - 699 SCP/SFTP
+ 700 - 799 SOCKS4 (even numbers) and SOCK5 (odd numbers)
+ 800 - 899 POP3, IMAP, SMTP
+ 1000 - 1299 miscellaneous*
+ 1300 - 1399 unit tests*
+ 1400 - 1499 miscellaneous*
+ 1500 - 1599 libcurl source code tests, not using the curl command tool
+ (same as 5xx)
+ 2000 - x multiple sequential protocols per test case*
+
+ Since 30-apr-2003, there's nothing in the system that requires us to keep
+ within these number series, and those sections marked with * actually
+ contain tests for a variety of protocols. Each test case now specifies its
+ own server requirements, independent of test number.
+
+3. Write tests
+
+ Here's a quick description on writing test cases. We basically have three
+ kinds of tests: the ones that test the curl tool, the ones that build small
+ applications and test libcurl directly and the unit tests that test
+ individual (possibly internal) functions.
+
+ 3.1 test data
+
+ Each test has a master file that controls all the test data. What to read,
+ what the protocol exchange should look like, what exit code to expect and
+ what command line arguments to use etc.
+
+ These files are tests/data/test[num] where [num] is described in section 2
+ of this document, and the XML-like file format of them is described in the
+ separate tests/FILEFORMAT document.
+
+ 3.2 curl tests
+
+ A test case that runs the curl tool and verifies that it gets the correct
+ data, it sends the correct data, it uses the correct protocol primitives
+ etc.
+
+ 3.3 libcurl tests
+
+ The libcurl tests are identical to the curl ones, except that they use a
+ specific and dedicated custom-built program to run instead of "curl". This
+ tool is built from source code placed in tests/libtest and if you want to
+ make a new libcurl test that is where you add your code.
+
+ 3.4 unit tests
+
+ Unit tests are tests in the 13xx sequence and they are placed in tests/unit.
+ There's a tests/unit/README describing the specific set of checks and macros
+ that may be used when writing tests that verify behaviors of specific
+ individual functions.
+
+ The unit tests depend on curl being built with debug enabled.
+
+4. TODO
+
+ 4.1 More protocols
+
+ Add tests for TELNET, LDAP, DICT...
+
+ 4.2 SOCKS auth
+
+ SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the
+ test mechanism) doesn't support them
projects / platform / upstream / curl.git / blobdiff